### SQL Example: Get sqlite-lines Version Source: https://github.com/asg017/sqlite-lines/blob/main/docs.md This SQL query demonstrates how to retrieve the version string of the currently loaded sqlite-lines extension. This is useful for verifying the installed version and ensuring compatibility. ```sql select lines_version(); ``` -------------------------------- ### SQL Example: Get sqlite-lines Debug Information Source: https://github.com/asg017/sqlite-lines/blob/main/docs.md This SQL query retrieves detailed debugging information about the sqlite-lines build. It includes the version, commit hash, and build date, which can be helpful for troubleshooting. ```sql select lines_debug(); ``` -------------------------------- ### Install sqlite-lines for Deno Source: https://github.com/asg017/sqlite-lines/blob/main/README.md Provides the import path for using sqlite-lines in Deno. Deno manages dependencies via URLs. ```bash deno.land/x/sqlite_lines ``` -------------------------------- ### Build WASM sql.js with sqlite-lines Source: https://github.com/asg017/sqlite-lines/blob/main/docs.md This command builds the WebAssembly version of sql.js, incorporating sqlite-lines functionalities. It requires the Emscripten SDK to be installed. The `make test-sqljs` command starts a local server for manual testing. ```bash make sqljs # will start a local server and open tests/test-sqljs.html for manual testing make test-sqljs ``` -------------------------------- ### SQL Example: Use `lines_read` Table Function Source: https://github.com/asg017/sqlite-lines/blob/main/docs.md This SQL query shows how to use the `lines_read` table function to read lines from a file specified by its path. It also includes an example of how to parse JSON data from each line using the `->` operator. ```sql select * from lines_read("my-file.txt"); select line -> '$.id' as id, line -> '$.name' as name from lines_read("my-file.ndjson"); ``` -------------------------------- ### Install sqlite-lines NPM Package Source: https://github.com/asg017/sqlite-lines/blob/main/npm/sqlite-lines/README.md Installs the sqlite-lines package using npm. This command is used by Node.js developers to add the package to their project. ```bash npm install sqlite-lines ``` -------------------------------- ### Install Datasette plugin for sqlite-lines Source: https://github.com/asg017/sqlite-lines/blob/main/README.md Installs the datasette-sqlite-lines plugin for Datasette. This allows Datasette to utilize the sqlite-lines functionality. ```bash datasette install datasette-sqlite-lines ``` -------------------------------- ### Install sqlite-lines for Ruby Source: https://github.com/asg017/sqlite-lines/blob/main/README.md Installs the sqlite-lines gem for Ruby. This command uses the RubyGems package manager. ```bash gem install sqlite-lines ``` -------------------------------- ### Test sqlite-lines Loadable Extension Source: https://github.com/asg017/sqlite-lines/blob/main/docs.md This command runs tests for the loadable extension of sqlite-lines. It requires Python 3 to be installed on the system. ```bash make test-loadable ``` -------------------------------- ### sqlite-lines CLI: Basic Transformations Source: https://context7.com/asg017/sqlite-lines/llms.txt Illustrates basic usage of the sqlite-lines CLI tool for transforming text files. Examples include uppercasing text and adding line numbers. ```bash # Transform each line: uppercase text with line numbers cat names.txt | sqlite-lines 'rowid || " " || upper(d)' # 1 ALICE # 2 BOB # 3 CAROL ``` -------------------------------- ### Load sqlite-lines as a loadable extension in Datasette Source: https://github.com/asg017/sqlite-lines/blob/main/README.md Demonstrates how to load the sqlite-lines extension with Datasette using the `--load-extension` flag. This example uses a version that limits filesystem access for security. ```bash datasette data.db --load-extension ./lines_nofs0 ``` -------------------------------- ### Install sqlite-lines Python Package Source: https://github.com/asg017/sqlite-lines/blob/main/python/sqlite_lines/README.md Installs the sqlite-lines Python package using pip. This command is used to add the package to your Python environment. ```bash pip install sqlite-lines ``` -------------------------------- ### Load sqlite-lines Extension in Python Source: https://github.com/asg017/sqlite-lines/blob/main/python/sqlite_lines/README.md Demonstrates how to use the sqlite-lines Python package to load the sqlite-lines extension into a sqlite3 connection. It shows how to get the extension path and then load it. ```python import sqlite_lines # Get the path to the loadable extension print(sqlite_lines.loadable_lines()) # Load the extension into a connection import sqlite3 conn = sqlite3.connect(':memory:') sqlite_lines.load(conn) # Verify the extension is loaded by calling a function from it print(conn.execute('select lines_version()').fetchone()) ``` -------------------------------- ### Get sqlite-lines Extension Path in Python Source: https://github.com/asg017/sqlite-lines/blob/main/python/sqlite_lines/README.md Retrieves the full path to the locally installed sqlite-lines extension using the `loadable_lines()` function. This path can be used with `sqlite3.Connection.load_extension()`. ```python import sqlite_lines print(sqlite_lines.loadable_lines()) # '/.../venv/lib/python3.9/site-packages/sqlite_lines/lines0' ``` -------------------------------- ### sqlite-lines CLI: Creating New JSON Objects Source: https://context7.com/asg017/sqlite-lines/llms.txt Demonstrates creating new JSON objects from existing data using the sqlite-lines CLI. This example constructs JSON objects with 'id' and 'score' fields. ```bash # Create new JSON objects cat data.ndjson | sqlite-lines 'json_object("id", d ->> "$.id", "score", d ->> "$.stats.score")' # {"id":1,"score":95} # {"id":2,"score":87} ``` -------------------------------- ### SQL: Get sqlite-lines extension version Source: https://context7.com/asg017/sqlite-lines/llms.txt The `lines_version()` scalar function returns the semantic version string of the currently loaded `sqlite-lines` extension. This is useful for verifying the installed version and ensuring compatibility. ```sql SELECT lines_version(); -- 'v0.2.2' ``` -------------------------------- ### SQL Example: Use `lines` Table Function Source: https://github.com/asg017/sqlite-lines/blob/main/docs.md This SQL query demonstrates the usage of the `lines` table function, which splits a given text document into individual lines. It shows how to select the line content and its corresponding line number (rowid). ```sql select rowid, line from lines('a b c'); ``` -------------------------------- ### Load sqlite-lines Extension in Deno Source: https://context7.com/asg017/sqlite-lines/llms.txt Shows how to load the sqlite-lines extension into a Deno x/sqlite3 Database instance. It includes examples for querying the version and processing log files using `lines_read()`. ```typescript import { Database } from "https://deno.land/x/sqlite3@0.8.0/mod.ts"; import * as sqlite_lines from "https://deno.land/x/sqlite_lines@v0.2.2/mod.ts"; const db = new Database(":memory:"); db.enableLoadExtension = true; sqlite_lines.load(db); // Query version const [version] = db .prepare("SELECT lines_version()") .value<[string]>()!; console.log(version); // "v0.2.2" // Process log file const stmt = db.prepare(` SELECT rowid, line FROM lines_read(?) WHERE line LIKE '%ERROR%' `); for (const [lineNum, content] of stmt.values<[number, string]>("app.log")) { console.log(`Line ${lineNum}: ${content}`); } ``` -------------------------------- ### Get sqlite-lines Loadable Path for better-sqlite3 Source: https://github.com/asg017/sqlite-lines/blob/main/npm/sqlite-lines/README.md Retrieves the file system path to the compiled sqlite-lines extension. This path is intended for use with the .loadExtension() method in better-sqlite3. ```javascript import Database from "better-sqlite3"; import * as sqlite_lines from "sqlite-lines"; const db = new Database(":memory:"); db.loadExtension(sqlite_lines.getLoadablePath()); ``` -------------------------------- ### Get sqlite-lines Loadable Path for node-sqlite3 Source: https://github.com/asg017/sqlite-lines/blob/main/npm/sqlite-lines/README.md Obtains the file system path for the sqlite-lines extension, suitable for use with the .loadExtension() method in node-sqlite3. This ensures the correct extension is loaded for the database. ```javascript import sqlite3 from "sqlite3"; import * as sqlite_lines from "sqlite-lines"; const db = new sqlite3.Database(":memory:"); db.loadExtension(sqlite_lines.getLoadablePath()); ``` -------------------------------- ### Load sqlite-lines Extension in Node.js Source: https://context7.com/asg017/sqlite-lines/llms.txt Demonstrates how to load the sqlite-lines extension for both better-sqlite3 and node-sqlite3 in Node.js. It shows how to get the loadable path and use the `lines_read()` function to query data from files. ```javascript // Using better-sqlite3 import Database from "better-sqlite3"; import * as sqlite_lines from "sqlite-lines"; const db = new Database(":memory:"); db.loadExtension(sqlite_lines.getLoadablePath()); // Verify installation const version = db.prepare("SELECT lines_version()").pluck().get(); console.log(version); // "v0.2.2" // Process NDJSON file const stmt = db.prepare(` SELECT line ->> '$.name' as name, line ->> '$.email' as email FROM lines_read(?) WHERE line ->> '$.active' = true `); for (const row of stmt.iterate('users.ndjson')) { console.log(`${row.name}: ${row.email}`); } // Using node-sqlite3 import sqlite3 from "sqlite3"; import * as sqlite_lines from "sqlite-lines"; const db = new sqlite3.Database(":memory:"); db.loadExtension(sqlite_lines.getLoadablePath()); db.each( "SELECT line ->> '$.id' as id FROM lines_read('data.ndjson')", (err, row) => { if (err) throw err; console.log(row.id); } ); ``` -------------------------------- ### sqlite-lines CLI: Filtering Data Source: https://context7.com/asg017/sqlite-lines/llms.txt Shows how to filter data from NDJSON files using the sqlite-lines CLI. This example extracts names of users older than 30. ```bash # Filter: names of users older than 30 cat users.ndjson | sqlite-lines 'd ->> "$.name"' 'd ->> "$.age" > 30' # Carol # David ``` -------------------------------- ### Load sqlite-lines Extension in Ruby Source: https://context7.com/asg017/sqlite-lines/llms.txt Demonstrates loading the sqlite-lines extension into a Ruby SQLite3 database connection. It includes examples for checking the version and processing NDJSON files using `lines_read()`. ```ruby require 'sqlite3' require 'sqlite_lines' db = SQLite3::Database.new(':memory:') db.enable_load_extension(true) SqliteLines.load(db) db.enable_load_extension(false) # Check version version = db.get_first_value('SELECT lines_version()') puts version # "v0.2.2" # Process NDJSON db.execute(<<-SQL, ['events.ndjson']) do |row| SELECT line ->> '$.event_type' as event_type, COUNT(*) as count FROM lines_read(?) GROUP BY 1 ORDER BY 2 DESC SQL puts "#{row[0]}: #{row[1]}" end # Get loadable path for manual loading puts SqliteLines.lines_loadable_path # '/path/to/gems/sqlite_lines/lines0' ``` -------------------------------- ### Filter Words using LIKE Operator Source: https://github.com/asg017/sqlite-lines/blob/main/benchmarks/README.md Compares the speed of sqlite-lines with SQLite's LIKE operator against the traditional `cat | grep` command for filtering words starting with 'sh' from a dictionary file. ```bash # cat | grep approach # cat /usr/share/dict/words | grep '^sh' # sqlite-lines + SQLite LIKE approach # cat /usr/share/dict/words | sqlite-lines -c "SELECT line FROM lines WHERE line LIKE 'sh%'" ``` -------------------------------- ### Filter lines with SQL using sqlite-lines CLI Source: https://github.com/asg017/sqlite-lines/blob/main/README.md Shows how to filter lines from standard input using the sqlite-lines CLI with a SQL WHERE clause. This example retrieves names of people older than 40 from a JSON Lines file. ```bash # get the names of all people older than 40 cat data.ndjson | sqlite-lines 'd ->> "$.name"' 'd ->> "$.age" > 40' ``` -------------------------------- ### Process NDJSON Data with sqlite-lines and SQLite JSON Functions Source: https://github.com/asg017/sqlite-lines/blob/main/README.md Shows how to use `sqlite-lines` with ndjson files and SQLite's built-in JSON functions to extract and aggregate data. This example calculates the top 5 country codes from Google's Quick, Draw! dataset. ```sql select line ->> '$.countrycode' as countrycode, count(*) from lines_read('./calendar.ndjson') group by 1 order by 2 desc limit 5; /* ┌─────────────┬──────────┐ │ countrycode │ count(*) │ ├─────────────┼──────────┤ │ US │ 141001 │ │ GB │ 22560 │ │ CA │ 11759 │ │ RU │ 9250 │ │ DE │ 8748 │ └─────────────┴──────────┘ */ ``` -------------------------------- ### SQL: Get sqlite-lines extension debug information Source: https://context7.com/asg017/sqlite-lines/llms.txt The `lines_debug()` scalar function provides detailed debugging information about the `sqlite-lines` extension, including its version, build date, commit hash, and build configuration (e.g., filesystem access enabled/disabled). ```sql SELECT lines_debug(); /* Version: v0.2.2 Date: 2022-05-15T16:57:23Z-0700 Commit: c87a67c6e76 */ -- No-filesystem build will show: /* Version: v0.2.2 Date: 2022-05-15T16:57:23Z-0700 Commit: c87a67c6e76 NO FILESYSTEM */ ``` -------------------------------- ### SQL: Parse inline JSON array into rows using lines() Source: https://context7.com/asg017/sqlite-lines/llms.txt This example demonstrates using the `lines()` function in conjunction with SQLite's JSON operators (`->>`) to parse newline-delimited JSON (NDJSON) strings directly within SQL queries. Each line is treated as a separate JSON object. ```sql SELECT line ->> '$.id' as id, line ->> '$.name' as name FROM lines('{"id":1,"name":"Alice"} {"id":2,"name":"Bob"} {"id":3,"name":"Carol"}'); /* ┌────┬───────┐ │ id │ name │ ├────┼───────┤ │ 1 │ Alice │ │ 2 │ Bob │ │ 3 │ Carol │ └────┴───────┘ */ ``` -------------------------------- ### Build sqlite-lines Loadable Extension Source: https://github.com/asg017/sqlite-lines/blob/main/docs.md This command builds the runtime-loadable extension for sqlite-lines. The output is a shared library (`.dylib` for MacOS, `.so` for Linux) that can be dynamically loaded into SQLite. ```bash make loadable ``` -------------------------------- ### Clone and Navigate sqlite-lines Repository Source: https://github.com/asg017/sqlite-lines/blob/main/docs.md This snippet shows the basic commands to clone the sqlite-lines repository from GitHub and navigate into the project directory. It's the initial step for building or using the library. ```bash git clone git@github.com:asg017/sqlite-lines.git cd sqlite-lines ``` -------------------------------- ### Load sqlite-lines with better-sqlite3 Source: https://github.com/asg017/sqlite-lines/blob/main/npm/sqlite-lines/README.md Demonstrates how to load the sqlite-lines extension into a better-sqlite3 database instance. It imports necessary modules, creates an in-memory database, loads the extension using getLoadablePath(), and then queries the version of the extension. ```javascript import Database from "better-sqlite3"; import * as sqlite_lines from "sqlite-lines"; const db = new Database(":memory:"); db.loadExtension(sqlite_lines.getLoadablePath()); const version = db.prepare("select lines_version()").pluck().get(); console.log(version); // "v0.2.0" ``` -------------------------------- ### Show Tool Versions Source: https://github.com/asg017/sqlite-lines/blob/main/benchmarks/README.md Displays the versions of various tools used in the benchmarks, including ndjson-map, Python, DuckDB, dsq, sqlite-lines, pandas, and zq. This is achieved by running the --version flag or equivalent commands for each tool. ```bash + ndjson-map --version 0.3.1 + python3 --version Python 3.8.10 + duckdb --version v0.4.0 da9ee490d + dsq --version dsq 0.17.0 + sqlite3 :memory: '.load ../dist/lines0' 'select lines_version()' v0.0.1 + python3 -c 'import pandas as pd; print(pd.__version__)' 1.4.2 + zq --version Version: v1.0.0 ``` -------------------------------- ### Load sqlite-lines with node-sqlite3 Source: https://github.com/asg017/sqlite-lines/blob/main/npm/sqlite-lines/README.md Shows how to load the sqlite-lines extension into a node-sqlite3 database instance. It imports the required libraries, initializes a database, loads the extension, and then retrieves the extension's version. ```javascript import sqlite3 from "sqlite3"; import * as sqlite_lines from "sqlite-lines"; const db = new sqlite3.Database(":memory:"); db.loadExtension(sqlite_lines.getLoadablePath()); db.get("select lines_version()", (err, row) => { console.log(row); // {json_schema_version(): "v0.2.0"} }); ``` -------------------------------- ### Initialize and Query SQL.js Database (JavaScript) Source: https://github.com/asg017/sqlite-lines/blob/main/tests/test-sqljs.html Initializes an SQL.js database with a custom file locator and executes several predefined queries. The results are then formatted and displayed in an HTML table. This snippet requires the SQL.js library and htl for DOM manipulation. ```javascript import { html } from "https://cdn.skypack.dev/htl"; const config = { locateFile: (filename, prefix) => { return `../dist/${filename}`; }, }; initSqlJs(config).then(function (SQL) { var db = new SQL.Database(); function printResults(query) { let stmt = db.prepare(query); const columns = stmt.getColumnNames(); const rows = []; while (stmt.step()) { rows.push(stmt.get()); } document.body.appendChild(html`
${query}
${columns.map((column) => html``)} ${rows.map( (values) => html` ${values.map((d) => html``)} ` )}
${column}
${d}
`); } try { printResults(`select name from pragma_function_list where name like 'lines%'`); printResults(`select name from pragma_module_list where name like 'lines%'`); printResults("select lines_version(), lines_debug()"); printResults(`select rowid, line from lines('a|b|c|d|e', '|')`); document.querySelector("#results").innerText = "✅ tests passed"; } catch { document.querySelector("#results").innerText = "❌ test(s) failed, check console"; } }); ``` -------------------------------- ### Build sqlite-lines CLI Source: https://github.com/asg017/sqlite-lines/blob/main/README.md Instructions to build the command-line interface for sqlite-lines. This involves using the 'make' command to compile the CLI, which can then be executed from the terminal. ```bash make cli ./dist/sqlite-lines ``` -------------------------------- ### Build sqlite-lines CLI Source: https://github.com/asg017/sqlite-lines/blob/main/docs.md This command builds a standalone command-line interface for sqlite-lines. This CLI can be used to interact with sqlite-lines functionalities directly from the terminal. ```bash make cli make test-cli ``` -------------------------------- ### Load sqlite-lines as a loadable extension in SQLite CLI Source: https://github.com/asg017/sqlite-lines/blob/main/README.md Demonstrates how to load the sqlite-lines extension using the SQLite command-line interface. This allows direct use of its functions within SQL queries. ```sql .load ./lines0 select lines_version(); -- v0.0.1 ``` -------------------------------- ### Check Supported Platform Source: https://github.com/asg017/sqlite-lines/blob/main/npm/sqlite-lines/README.md Determines the operating system and CPU architecture of the current machine using Node.js process properties. This is useful for verifying compatibility with the sqlite-lines NPM package. ```bash $ node -e 'console.log([process.platform, process.arch])' [ 'darwin', 'x64' ] ``` -------------------------------- ### Build SQLite CLI with sqlite-lines Included Source: https://github.com/asg017/sqlite-lines/blob/main/docs.md These commands build a custom SQLite command-line interface (CLI) that includes the sqlite-lines functionalities statically linked. This allows using sqlite-lines functions directly within the SQLite CLI. ```bash make sqlite3 make test-sqlite3 ``` -------------------------------- ### Transform lines with SQL using sqlite-lines CLI Source: https://github.com/asg017/sqlite-lines/blob/main/README.md Demonstrates how to use the sqlite-lines CLI to transform lines of text from standard input using SQL expressions. It shows how to process each line, including uppercasing text and extracting JSON values. ```bash $ cat names.txt | sqlite-lines 'rowid || upper(d)' 1ALEX 2BRIAN 3CRAIG ``` ```bash $ cat data.ndjson | sqlite-lines 'd ->> "$.id"' ``` ```bash $ cat data.ndjson | sqlite-lines 'json_object("name", d ->> "$.name", "age": d ->> "$.stats.age")' ``` -------------------------------- ### sqlite-lines CLI: Aggregating Data Source: https://context7.com/asg017/sqlite-lines/llms.txt Illustrates how to aggregate data using the sqlite-lines CLI, specifically counting events by type from an NDJSON file. ```bash # Aggregate: count events by type cat events.ndjson | sqlite-lines 'd ->> "$.type" || ": " || count(*)' '1=1' 'd ->> "$.type"' # click: 1523 # view: 8934 # purchase: 342 ``` -------------------------------- ### Load sqlite-lines Extension in Deno Source: https://github.com/asg017/sqlite-lines/blob/main/deno/README.md This snippet demonstrates how to load the sqlite-lines extension into a Deno SQLite database using the x/sqlite3 and x/sqlite_lines modules. It initializes an in-memory database, enables extension loading, loads the sqlite_lines extension, and then queries the extension's version. ```javascript import { Database } from "https://deno.land/x/sqlite3@0.8.0/mod.ts"; import * as sqlite_lines from "https://deno.land/x/sqlite_lines@v0.2.2/mod.ts"; const db = new Database(":memory:"); db.enableLoadExtension = true; sqlite_lines.load(db); const [version] = db .prepare("select lines_version()") .value<[string]>()!; console.log(version); ``` -------------------------------- ### Deno Permissions for x/sqlite_lines Source: https://github.com/asg017/sqlite-lines/blob/main/deno/README.md This command shows the necessary Deno permissions to run applications using the x/sqlite_lines module. It requires network and filesystem access to download and cache the SQLite extension, along with unstable features and Foreign Function Interface (FFI) support. ```bash deno run -A --unstable ``` -------------------------------- ### Load and Read Lines from a File in SQL Source: https://github.com/asg017/sqlite-lines/blob/main/README.md Demonstrates how to load the sqlite-lines extension and read lines from a specified file using the `lines_read()` function in SQL. This is the basic usage for accessing file content line by line within SQLite. ```sql .load ./lines0 select line from lines_read('logs.txt'); ``` -------------------------------- ### Aggregate lines with SQL using sqlite-lines CLI Source: https://github.com/asg017/sqlite-lines/blob/main/README.md Illustrates how to aggregate lines from standard input using the sqlite-lines CLI with a SQL GROUP BY clause. This functionality allows for summarizing data based on specified criteria. ```bash # Example for aggregation (specific SQL expression would be needed) # cat data.ndjson | sqlite-lines 'column_to_group_by' 'aggregation_expression' 'd ->> "$.column_to_group_by"' ``` -------------------------------- ### Parse and Calculate Sum from NDJSON File Source: https://github.com/asg017/sqlite-lines/blob/main/benchmarks/README.md Compares the performance of different tools in parsing a 180MB NDJSON file and calculating the sum of elements in a specific property. Includes Python's standard file iteration, ndjson-cli, dsq, sqlite-utils, zq, Pandas, sqlite-lines, and DuckDB. ```bash # Example of a Python script for parsing NDJSON # (Actual script content not provided in the text, this is illustrative) # py.sh # import json # total = 0 # with open('large.ndjson', 'r') as f: # for line in f: # data = json.loads(line) # total += data['some_property'] # print(total) # Example of ndjson-cli usage # ndjson-cli '.[your_property]' < large.ndjson | node -e "let sum=0; process.stdin.on('data', chunk => sum += parseInt(chunk)); process.stdin.on('end', () => console.log(sum))" # Example of dsq usage # dsq --input-format ndjson --output-format json 'SELECT SUM(your_property) FROM file' # Example of sqlite-lines usage # cat large.ndjson | sqlite-lines -c "SELECT SUM(json_extract(line, '$.your_property')) FROM lines" ``` -------------------------------- ### Compile sqlite-lines Object File Source: https://github.com/asg017/sqlite-lines/blob/main/docs.md This command compiles the sqlite-lines library into an object file (`.o`). This object file can then be linked into your own C/C++ applications, allowing you to statically include sqlite-lines functionality. ```bash make dist/lines0.o ``` -------------------------------- ### Load sqlite-lines Extension Manually in Python Source: https://github.com/asg017/sqlite-lines/blob/main/python/sqlite_lines/README.md Demonstrates manually loading the sqlite-lines extension into a sqlite3 connection object. This involves enabling extension loading, calling `sqlite_lines.load()`, and then disabling extension loading. ```python import sqlite_lines import sqlite3 conn = sqlite3.connect(':memory:') # Enable extension loading conn.enable_load_extension(True) # Load the sqlite-lines extension sqlite_lines.load(conn) # Disable extension loading for security conn.enable_load_extension(False) # Verify the extension is loaded print(conn.execute('select lines_version()').fetchone()) # ('v0.1.0') ``` -------------------------------- ### Load sqlite-lines as a loadable extension in Node.js Source: https://github.com/asg017/sqlite-lines/blob/main/README.md Illustrates loading the sqlite-lines extension in Node.js using the `better-sqlite3` library. This enables the use of SQLite extensions within Node.js applications. ```javascript const Database = require("better-sqlite3"); const db = new Database(":memory:"); db.loadExtension("./lines0"); console.log(db.prepare("select lines_version()").get()); // { 'lines_version()': 'v0.0.1' } ``` -------------------------------- ### Read Lines from All Files in a Directory using fsdir and lines_read Source: https://github.com/asg017/sqlite-lines/blob/main/README.md Illustrates how to combine SQLite's `fsdir()` function with `lines_read()` to iterate through all files in a directory and read lines from those matching a specific pattern (e.g., '.txt'). This is useful for batch processing log files or other text-based data. ```sql select name as file, lines.rowid as line_number, line from fsdir('logs') join lines_read(name) as lines where name like '%.txt'; /* ┌─────────────────────┬──────┐ │ file │ line_number | line │ ├───────┼─────────────┤──────┤ | a.txt | 1 | x | | a.txt | 2 | y | | a.txt | 3 | z | | b.txt | 1 | xx | | b.txt | 2 | yy | | c.txt | 1 | xxx | └───────┴─────────────┴──────┘ */ ``` -------------------------------- ### Python: Load sqlite-lines extension into connection Source: https://context7.com/asg017/sqlite-lines/llms.txt This Python code snippet shows how to load the `sqlite-lines` extension into a `sqlite3` connection using the `sqlite_lines.load()` function. It also demonstrates how to verify the extension is loaded by querying its version and processing an NDJSON file. ```python import sqlite3 import sqlite_lines # Connect and load extension conn = sqlite3.connect(':memory:') conn.enable_load_extension(True) sqlite_lines.load(conn) conn.enable_load_extension(False) # Verify installation version = conn.execute('SELECT lines_version()').fetchone()[0] print(version) # 'v0.2.2' # Process NDJSON file results = conn.execute(''' SELECT line ->> '$.user_id' as user_id, line ->> '$.event' as event, line ->> '$.timestamp' as ts FROM lines_read('events.ndjson') WHERE line ->> '$.event' = 'purchase' LIMIT 10 ''').fetchall() for row in results: print(f"User {row[0]}: {row[1]} at {row[2]}") # Get loadable path for manual loading print(sqlite_lines.loadable_path()) # '/path/to/site-packages/sqlite_lines/lines0' ``` -------------------------------- ### Python Integration: sqlite_lines.load() Source: https://context7.com/asg017/sqlite-lines/llms.txt Loads the sqlite-lines extension into a Python sqlite3 connection. Provides `loadable_path()` for manual loading. ```APIDOC ## sqlite_lines.load(connection) ### Description Loads the sqlite-lines extension into a Python sqlite3 connection. Also available: `loadable_path()` returns the path to the extension for manual loading. ### Method Python Function ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body - **connection** (sqlite3.Connection) - The active SQLite connection object. ### Request Example ```python import sqlite3 import sqlite_lines # Connect and load extension conn = sqlite3.connect(':memory:') conn.enable_load_extension(True) sqlite_lines.load(conn) conn.enable_load_extension(False) # Verify installation version = conn.execute('SELECT lines_version()').fetchone()[0] print(version) # 'v0.2.2' # Process NDJSON file results = conn.execute(''' SELECT line ->> '$.user_id' as user_id, line ->> '$.event' as event, line ->> '$.timestamp' as ts FROM lines_read('events.ndjson') WHERE line ->> '$.event' = 'purchase' LIMIT 10 ''').fetchall() for row in results: print(f"User {row[0]}: {row[1]} at {row[2]}") # Get loadable path for manual loading print(sqlite_lines.loadable_path()) # '/path/to/site-packages/sqlite_lines/lines0' ``` ### Response #### Success Response (200) This function does not return a value directly but modifies the connection to load the extension. #### Response Example None ``` -------------------------------- ### Load sqlite-lines as a loadable extension in Python Source: https://github.com/asg017/sqlite-lines/blob/main/README.md Shows how to load the sqlite-lines extension within a Python script using the `sqlite3` module. It enables loading external extensions into an in-memory SQLite database. ```python import sqlite3 con = sqlite3.connect(":memory:") con.enable_load_extension(True) con.load_extension("./lines0") print(con.execute("select lines_version()").fetchone()) # ('v0.0.1',) ``` -------------------------------- ### sqlite-lines CLI: Extracting JSON Fields Source: https://context7.com/asg017/sqlite-lines/llms.txt Demonstrates using the sqlite-lines CLI to extract specific JSON fields from NDJSON files. It shows how to target fields using the `->>` operator. ```bash # Extract JSON fields from NDJSON cat users.ndjson | sqlite-lines 'd ->> "$.name"' # Alice # Bob # Carol ``` -------------------------------- ### SQL Scalar Function: lines_debug() Source: https://context7.com/asg017/sqlite-lines/llms.txt Returns debugging information including version, build date, commit hash, and build configuration. ```APIDOC ## lines_debug() ### Description Returns debugging information including version, build date, commit hash, and build configuration (e.g., whether filesystem access is disabled). ### Method SQL Function ### Parameters None ### Request Example ```sql SELECT lines_debug(); ``` ### Response #### Success Response (200) - **debug_info** (text) - A multi-line string containing debugging information. #### Response Example ```text Version: v0.2.2 Date: 2022-05-15T16:57:23Z-0700 Commit: c87a67c6e76 NO FILESYSTEM ``` ``` -------------------------------- ### Insert NDJSON Objects into SQLite Table Source: https://github.com/asg017/sqlite-lines/blob/main/benchmarks/README.md Demonstrates the ability of sqlite-lines to transform NDJSON data and insert it into a flat SQLite table, contrasting its performance with sqlite-utils and DuckDB's approach. ```bash # sqlite-lines for insertion # cat data.ndjson | sqlite-lines --table my_table --create --pk id # sqlite-utils for insertion (illustrative, as it's slower) # sqlite-utils insert my_database.db my_table data.ndjson --detect-types # DuckDB for insertion (different format, not directly comparable for SQLite) # duckdb -c "CREATE TABLE my_table AS SELECT * FROM read_json_objects('data.ndjson');" ``` -------------------------------- ### Table Functions Source: https://github.com/asg017/sqlite-lines/blob/main/docs.md Provides information about table functions available in sqlite-lines. ```APIDOC ## Table Functions ### `lines(document, [delimeter])` **Description**: A table function that reads a given document (TEXT or BLOB) into memory and generates a row for each line. **Method**: SQL **Endpoint**: N/A ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body - **document** (TEXT or BLOB) - Required - The document to read. - **delimeter** (char) - Optional - The single character delimiter to split lines on. Defaults to newline (`\n`). ### Request Example ```sql SELECT rowid, line FROM lines('a\nb\nc'); ``` ### Response #### Success Response (200) - **line** (text) - The content of the line. - **rowid** (integer) - The line number, starting at 1. #### Response Example ```json { "rowid": 1, "line": "a" } { "rowid": 2, "line": "b" } { "rowid": 3, "line": "c" } ``` ### `lines_read(path, [delimeter])` **Description**: A table function that reads a file from the given path and generates a row for each line. **Method**: SQL **Endpoint**: N/A ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body - **path** (text) - Required - The path to the file to read. - **delimeter** (char) - Optional - The single character delimiter to split lines on. Defaults to newline (`\n`). ### Request Example ```sql SELECT * FROM lines_read("my-file.txt"); SELECT line -> '$.id' as id, line -> '$.name' as name FROM lines_read("my-file.ndjson"); ``` ### Response #### Success Response (200) - **line** (text) - The content of the line. - **path** (text) - The path of the file being read. #### Response Example ```json { "line": "This is the first line.", "path": "my-file.txt" } { "line": "This is the second line.", "path": "my-file.txt" } ``` ``` -------------------------------- ### SQL Scalar Function: lines_version() Source: https://context7.com/asg017/sqlite-lines/llms.txt Returns the semantic version string of the loaded sqlite-lines extension. ```APIDOC ## lines_version() ### Description Returns the semantic version string of the loaded sqlite-lines extension. ### Method SQL Function ### Parameters None ### Request Example ```sql SELECT lines_version(); ``` ### Response #### Success Response (200) - **version** (text) - The semantic version string of the extension. #### Response Example ```json "v0.2.2" ``` ``` -------------------------------- ### Handle Large NDJSON File Parsing Source: https://github.com/asg017/sqlite-lines/blob/main/benchmarks/README.md Evaluates the capability of sqlite-lines to handle extremely large NDJSON datasets (5GB+) without running into memory limitations, specifically addressing the SQLite maximum blob size. Other tools that are too slow or memory-intensive are omitted. ```bash # sqlite-lines is designed to handle large files by not loading the entire file into memory. # The benchmark likely involves piping a large NDJSON file to sqlite-lines. # Example (conceptual): # cat large_dataset.ndjson | sqlite-lines -c "SELECT COUNT(*) FROM lines" ``` -------------------------------- ### SQL Table Function: lines_read() Source: https://context7.com/asg017/sqlite-lines/llms.txt Reads a file from the filesystem and generates a row for each line. Streams the file without loading it entirely into memory. ```APIDOC ## lines_read(path, [delimiter]) ### Description Reads a file from the filesystem and generates a row for each line. Unlike `lines()`, this streams the file without loading it entirely into memory, allowing processing of files larger than SQLite's 1GB limit. Same return columns and delimiter options as `lines()`. ### Method SQL Function ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```sql -- Read all lines from a text file SELECT rowid as line_number, line FROM lines_read('data.txt'); -- Parse NDJSON file and extract fields SELECT line ->> '$.id' as id, line ->> '$.name' as name, line ->> '$.stats.score' as score FROM lines_read('users.ndjson'); -- Aggregate NDJSON data: top 5 countries in a dataset SELECT line ->> '$.countrycode' as countrycode, count(*) as cnt FROM lines_read('./calendar.ndjson') GROUP BY 1 ORDER BY 2 DESC LIMIT 5; -- Read lines from multiple files using fsdir() SELECT name as file, lines.rowid as line_number, line FROM fsdir('logs') JOIN lines_read(name) as lines WHERE name LIKE '%.txt'; -- Filter specific line by rowid (optimized single-row lookup) SELECT line FROM lines_read('config.txt') WHERE rowid = 5; -- Use custom delimiter for pipe-separated files SELECT line FROM lines_read('data.psv', '|'); ``` ### Response #### Success Response (200) - **rowid** (integer) - Line number starting at 1. - **line** (text) - The content of the line. #### Response Example ```json { "rowid": 1, "line": "This is the first line." } ``` ``` -------------------------------- ### Scalar Functions Source: https://github.com/asg017/sqlite-lines/blob/main/docs.md Provides information about scalar functions available in sqlite-lines. ```APIDOC ## Scalar Functions ### `lines_version()` **Description**: Returns the version string of the `sqlite-lines` library. **Method**: SQL **Endpoint**: N/A ### Parameters None ### Request Example ```sql SELECT lines_version(); ``` ### Response #### Success Response (200) - **version** (string) - The version string of `sqlite-lines`. #### Response Example ```json { "version": "v0.0.0" } ``` ### `lines_debug()` **Description**: Returns a string with debugging information for `sqlite-lines`, including version, commit hash, and build date. **Method**: SQL **Endpoint**: N/A ### Parameters None ### Request Example ```sql SELECT lines_debug(); ``` ### Response #### Success Response (200) - **debug_info** (string) - A string containing debugging information. #### Response Example ```json { "debug_info": "Version: v0.0.0\nDate: 2022-05-15T16:57:23Z-0700\nCommit: c87a67c6e76\nNO FILESYSTEM" } ``` ``` -------------------------------- ### SQL Table Function: lines() Source: https://context7.com/asg017/sqlite-lines/llms.txt Reads a TEXT or BLOB value into memory and generates a row for each line. Supports custom delimiters. ```APIDOC ## lines(document, [delimiter]) ### Description Reads a TEXT or BLOB value into memory and generates a row for each line in the document. Returns `line` (text content) and `rowid` (line number starting at 1). The default delimiter is `\n` with automatic `\r` stripping for CRLF files. Custom single-character delimiters are supported. ### Method SQL Function ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```sql -- Basic usage: split a string into lines SELECT rowid, line FROM lines('apple\nbanana\ncherry'); -- Custom delimiter: split by comma SELECT rowid, line FROM lines('red,green,blue', ','); -- Parse inline JSON array into rows SELECT line ->> '$.id' as id, line ->> '$.name' as name FROM lines('{"id":1,"name":"Alice"}\n{"id":2,"name":"Bob"}\n{"id":3,"name":"Carol"}'); ``` ### Response #### Success Response (200) - **rowid** (integer) - Line number starting at 1. - **line** (text) - The content of the line. #### Response Example ```json { "rowid": 1, "line": "apple" } ``` ``` -------------------------------- ### SQL: Read lines from a file using lines_read() Source: https://context7.com/asg017/sqlite-lines/llms.txt The `lines_read()` SQL function reads a file from the filesystem line by line. It streams the file content, avoiding memory limitations for large files. It returns the line content and its row number, similar to the `lines()` function. ```sql -- Read all lines from a text file SELECT rowid as line_number, line FROM lines_read('data.txt'); -- Parse NDJSON file and extract fields SELECT line ->> '$.id' as id, line ->> '$.name' as name, line ->> '$.stats.score' as score FROM lines_read('users.ndjson'); -- Aggregate NDJSON data: top 5 countries in a dataset SELECT line ->> '$.countrycode' as countrycode, count(*) as cnt FROM lines_read('./calendar.ndjson') GROUP BY 1 ORDER BY 2 DESC LIMIT 5; /* ┌─────────────┬────────┐ │ countrycode │ cnt │ ├─────────────┼────────┤ │ US │ 141001 │ │ GB │ 22560 │ │ CA │ 11759 │ │ RU │ 9250 │ │ DE │ 8748 │ └─────────────┴────────┘ */ -- Read lines from multiple files using fsdir() SELECT name as file, lines.rowid as line_number, line FROM fsdir('logs') JOIN lines_read(name) as lines WHERE name LIKE '%.txt'; -- Filter specific line by rowid (optimized single-row lookup) SELECT line FROM lines_read('config.txt') WHERE rowid = 5; -- Use custom delimiter for pipe-separated files SELECT line FROM lines_read('data.psv', '|'); ``` -------------------------------- ### SQL: Read lines from text or blob with custom delimiter Source: https://context7.com/asg017/sqlite-lines/llms.txt The `lines()` SQL function supports custom single-character delimiters for splitting text or BLOB data. This allows processing data formats other than standard line-based files, such as CSV or pipe-separated values. ```sql SELECT rowid, line FROM lines('red,green,blue', ','); /* ┌───────┬───────┐ │ rowid │ line │ ├───────┼───────┤ │ 1 │ red │ │ 2 │ green │ │ 3 │ blue │ └───────┴───────┘ */ ``` -------------------------------- ### SQL: Read lines from text or blob with default delimiter Source: https://context7.com/asg017/sqlite-lines/llms.txt The `lines()` SQL function reads a TEXT or BLOB value and splits it into rows based on newline characters. It returns the line content and its row number. This is useful for processing in-memory strings or small files. ```sql SELECT rowid, line FROM lines('apple banana cherry'); /* ┌───────┬────────┐ │ rowid │ line │ ├───────┼────────┤ │ 1 │ apple │ │ 2 │ banana │ │ 3 │ cherry │ └───────┴────────┘ */ ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.