### Example Chunked File Mode Configuration Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/configuration.md An example demonstrating how to set up the configuration for chunked file mode. This setup assumes 1MB chunks for a 100MB database, using 4-digit suffixes for chunk numbering. ```typescript { from: "inline", config: { serverMode: "chunked", urlPrefix: "https://cdn.example.com/db/chunk", serverChunkSize: 1048576, // 1 MB chunks databaseLengthBytes: 104857600, // 100 MB total suffixLength: 4, // 4-digit suffix (0000, 0001, etc) requestChunkSize: 4096 } } ``` -------------------------------- ### Install Dependencies and Initialize Project Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/README.md Installs necessary development and project dependencies, and initializes TypeScript configuration. ```sh mkdir example cd example echo '{}' > package.json npm install --save-dev webpack webpack-cli typescript ts-loader http-server npm install --save sql.js-httpvfs npx tsc --init ``` -------------------------------- ### Build and Serve Project Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/README.md Builds the JavaScript bundle using webpack and starts a local HTTP server. ```sh ./node_modules/.bin/webpack --mode=development ./node_modules/.bin/http-server ``` -------------------------------- ### Install sql.js-httpvfs Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/_DOCUMENTATION_SUMMARY.txt Install the sql.js-httpvfs package using npm. ```bash npm install sql.js-httpvfs ``` -------------------------------- ### Loading sql.js (v1.x - WebAssembly) Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/sql.js/README.md Example of asynchronously loading sql.js version 1.x using WebAssembly, requiring an initialization function. ```html ``` ```javascript var initSqlJs = require('sql-wasm.js'); initSqlJs().then(function(SQL){ var db = new SQL.Database(); //... }); ``` -------------------------------- ### Inline Full File Mode Example with Cache Busting Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/configuration.md An example of inline configuration for full file mode, including the cacheBust option. Remember to change the cacheBust value when the database is updated. ```typescript { from: "inline", config: { serverMode: "full", url: "https://cdn.example.com/databases/products.sqlite3", requestChunkSize: 4096, cacheBust: "v2024-01-15" // Change when database updates } } ``` -------------------------------- ### Get Stats for Multiple Databases with Worker Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/api-reference/worker-utilities.md This example shows how to initialize a worker with multiple virtual databases and retrieve statistics for each one individually using their virtual filenames. ```typescript const worker = await createDbWorker([ { from: "inline", virtualFilename: "main", config: {...} }, { from: "inline", virtualFilename: "archive", config: {...} } ], workerUrl, wasmUrl); // Get stats for each database const mainStats = await worker.worker.getStats("main"); const archiveStats = await worker.worker.getStats("archive"); console.log("Main DB:", mainStats); console.log("Archive DB:", archiveStats); ``` -------------------------------- ### Comprehensive Error Handling for Database Initialization Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/errors.md This TypeScript example demonstrates how to initialize a database worker with sql.js-httpvfs and includes detailed error handling for network, configuration, and setup issues. It logs specific error messages to the console and returns null on failure. ```typescript import { createDbWorker } from "sql.js-httpvfs"; // Comprehensive error handling const initializeDatabase = async () => { try { const worker = await createDbWorker( [config], workerUrl.toString(), wasmUrl.toString(), 10 * 1024 * 1024 // 10 MB limit ); return worker; } catch (e) { const error = e as Error; if (error.message.includes("Couldn't load")) { console.error("❌ Network error: Cannot reach server", error.message); return null; } if (error.message.includes("Length of the file")) { console.error("❌ Config error: Unknown database size", error.message); return null; } if (error.message.includes("Worker")) { console.error("❌ Setup error: Worker failed to load", error.message); return null; } if (error.message.includes("wasm")) { console.error("❌ Setup error: WebAssembly not supported", error.message); return null; } console.error("❌ Unexpected error:", error); return null; } }; const worker = await initializeDatabase(); if (!worker) { // Show error UI document.body.innerHTML = '

Failed to initialize database. See console.

'; } ``` -------------------------------- ### Query Method Convenience Example Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/api-reference/LazyHttpDatabase.md Demonstrates the convenience of the query() method compared to manually processing exec() results for object conversion. ```typescript // Instead of processing exec() results: const result = await db.exec("SELECT id, name FROM users WHERE active = ?", [1]); const users = result[0].values.map(row => ({ id: row[0], name: row[1] })); // Use query() for convenience: const users = await db.query("SELECT id, name FROM users WHERE active = ?", [1]); // users is already [{ id: 1, name: "Alice" }, ...] ``` -------------------------------- ### Loading sql.js (v0.x) Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/sql.js/README.md Example of synchronously loading sql.js in older versions using an HTML script tag or CommonJS module. ```html ``` ```javascript var SQL = require('sql.js'); var db = new SQL.Database(); //... ``` -------------------------------- ### Setup and Query DOM with SeriesVtab Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/api-reference/SeriesVtab.md Demonstrates how to set up the SeriesVtab worker, query the DOM for elements, and insert new elements. Ensure you have the correct config, workerUrl, and wasmUrl. ```typescript import { createDbWorker } from "sql.js-httpvfs"; const worker = await createDbWorker([config], workerUrl, wasmUrl); // The SeriesVtab is automatically registered in the database // Just start querying the DOM! const buttons = await worker.db.query( `SELECT selector, textContent FROM dom WHERE selector MATCH 'button'` ); console.log("Found buttons:", buttons); // Insert a new element await worker.db.exec( `INSERT INTO dom (parent, tagName, textContent, className) VALUES ('body', 'button', 'Click me!', 'my-button')` ); // The new button now exists in the DOM ``` -------------------------------- ### Creating a Lazy File for a Database Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/api-reference/createLazyFile.md Demonstrates how to initialize and use `createLazyFile` to load a SQLite database file. This example shows setting up the configuration, creating the lazy file instance, and then using it with `sql.CustomDatabase` for querying. It also shows how to access performance metrics after operations. ```typescript import { createLazyFile } from "sql.js-httpvfs"; // Inside a worker const sql = await initSqlJs({ locateFile: (file) => wasmUrl }); // Define how to fetch data for a full database file const config = { rangeMapper: (fromByte, toByte) => ({ url: "https://example.com/database.sqlite3", fromByte, toByte }), requestChunkSize: 4096, logPageReads: true }; // Create the lazy file const lazyFile = createLazyFile( sql.FS, "/", "mydatabase.sqlite3", true, // canRead false, // canWrite (databases are read-only) config ); // Now sqlite can use this file const db = new sql.CustomDatabase("mydatabase.sqlite3"); // Read the database const result = db.exec("SELECT * FROM users LIMIT 10"); // Access performance stats console.log(`Total fetched: ${lazyFile.contents.totalFetchedBytes} bytes`); console.log(`Total requests: ${lazyFile.contents.totalRequests}`); console.log(`Page access log:`, lazyFile.contents.readPages); ``` -------------------------------- ### Create and Use a SQLite Worker Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/api-reference/createDbWorker.md This example demonstrates how to create a web worker for SQLite database access using an inline configuration. It shows how to query the database and retrieve statistics, and how to release the worker when finished. Ensure the worker and WASM URLs are correctly set. ```typescript import { createDbWorker } from "sql.js-httpvfs"; // Inline configuration for a full database file const config = { from: "inline", config: { serverMode: "full", url: "/my-database.sqlite3", requestChunkSize: 4096, } }; const worker = await createDbWorker( [config], new URL("sql.js-httpvfs/dist/sqlite.worker.js", import.meta.url).toString(), new URL("sql.js-httpvfs/dist/sql-wasm.wasm", import.meta.url).toString(), 10 * 1024 * 1024 // 10 MB limit ); // Query the database const result = await worker.db.query("SELECT * FROM users WHERE id = ?", [123]); console.log(result); // Get statistics about bytes fetched const stats = await worker.worker.getStats(); console.log(`Fetched ${stats.totalFetchedBytes} bytes in ${stats.totalRequests} requests`); // Clean up when done worker.release(); ``` -------------------------------- ### Example Input Structure for QueryExecResult Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/api-reference/toObjects.md Demonstrates the expected input structure for the toObjects function, which includes column names and a 2D array of values. ```typescript [ { columns: ["id", "name", "email"], values: [ [1, "Alice", "alice@example.com"], [2, "Bob", "bob@example.com"] ] } ] ``` -------------------------------- ### RequestLimiter for Bandwidth Quotas Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/types.md Example implementation of RequestLimiter to enforce bandwidth quotas. It tracks the total fetched bytes and throws an error if the limit is exceeded. ```typescript let totalFetched = 0; const maxBytes = 10 * 1024 * 1024; // 10 MB limit const limiter = (bytes) => { if (totalFetched + bytes > maxBytes) { totalFetched = 0; throw new Error("Bandwidth quota exceeded"); } totalFetched += bytes; }; ``` -------------------------------- ### Initialize sql.js-httpvfs Worker Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/README.md Example of how to initialize the sql.js-httpvfs worker in a JavaScript/TypeScript environment. This involves specifying worker and WASM URLs, and configuring the database connection. ```typescript import { createDbWorker } from "sql.js-httpvfs" // sadly there's no good way to package workers and wasm directly so you need a way to get these two URLs from your bundler. // This is the webpack5 way to create a asset bundle of the worker and wasm: const workerUrl = new URL( "sql.js-httpvfs/dist/sqlite.worker.js", import.meta.url, ); const wasmUrl = new URL( "sql.js-httpvfs/dist/sql-wasm.wasm", import.meta.url, ); // the legacy webpack4 way is something like `import wasmUrl from "file-loader!sql.js-httpvfs/dist/sql-wasm.wasm"`. // the config is either the url to the create_db script, or a inline configuration: const config = { from: "inline", config: { serverMode: "full", // file is just a plain old full sqlite database requestChunkSize: 4096, // the page size of the sqlite database (by default 4096) url: "/foo/bar/test.sqlite3" // url to the database (relative or full) } }; // or: const config = { from: "jsonconfig", configUrl: "/foo/bar/config.json" } let maxBytesToRead = 10 * 1024 * 1024; const worker = await createDbWorker( [config], workerUrl.toString(), wasmUrl.toString(), maxBytesToRead // optional, defaults to Infinity ); // you can also pass multiple config objects which can then be used as separate database schemas with `ATTACH virtualFilename as schemaname`, where virtualFilename is also set in the config object. // worker.db is a now SQL.js instance except that all functions return Promises. const result = await worker.db.exec(`select * from table where id = ?`, [123]); // worker.worker.bytesRead is a Promise for the number of bytes read by the worker. // if a request would cause it to exceed maxBytesToRead, that request will throw a SQLite disk I/O error. console.log(await worker.worker.bytesRead); // you can reset bytesRead by assigning to it: worker.worker.bytesRead = 0; ``` -------------------------------- ### LazyHttpDatabase Usage Example Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/api-reference/LazyHttpDatabase.md Illustrates how to use the LazyHttpDatabase instance obtained from a worker, including querying data, accessing lazy file information, and executing raw SQL. ```typescript import { createDbWorker } from "sql.js-httpvfs"; const worker = await createDbWorker(/* config */); const db = worker.db; // LazyHttpDatabase instance // Query with automatic object conversion const users = await db.query<{ id: number; name: string }> ("SELECT * FROM users WHERE age > ?", [18] ); // Access lazy file information const lazyFile = db.lazyFiles.get(db.filename); if (lazyFile) { console.log(`Total file size: ${lazyFile.contents.length} bytes`); console.log(`Fetched so far: ${lazyFile.contents.totalFetchedBytes} bytes`); } // Raw exec for complex multi-statement queries const results = await db.exec( ` SELECT COUNT(*) as total FROM users; SELECT COUNT(*) as total FROM orders; ` ); console.log(results); // Array of QueryExecResult objects ``` -------------------------------- ### RangeMapper for Single Full File Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/types.md Example implementation of RangeMapper for fetching a single, complete file. It directly maps the requested byte range to the file's URL. ```typescript const rangeMapper = (from, to) => ({ url: "https://example.com/database.sqlite3", fromByte: from, toByte: to }); ``` -------------------------------- ### Create Lazy File with Bandwidth Limiting Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/api-reference/createLazyFile.md Create a lazy file with a custom request limiter to control bandwidth usage per session. This example enforces a 10 MB limit and resets the counter if exceeded. ```typescript let bytesThisSession = 0; const maxBytesPerSession = 10 * 1024 * 1024; // 10 MB limit const limitedConfig = { rangeMapper: (from, to) => ({ url: "https://example.com/database.sqlite3", fromByte: from, toByte: to }), requestChunkSize: 4096, requestLimiter: (bytes) => { if (bytesThisSession + bytes > maxBytesPerSession) { bytesThisSession = 0; throw new Error("Bandwidth quota exceeded for this session"); } bytesThisSession += bytes; } }; const lazyFile = createLazyFile(sql.FS, "/", "db.sqlite3", true, false, limitedConfig); ``` -------------------------------- ### Configure Multiple Databases Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/configuration.md Define an array of database configurations, specifying the source, virtual filename, and specific settings for each database. This setup is useful when you need to manage and access data from several distinct SQLite files. ```typescript const configs = [ { from: "inline", virtualFilename: "main", config: { serverMode: "full", url: "/main.db.sqlite3", requestChunkSize: 4096, cacheBust: "2024-01-15" } }, { from: "jsonconfig", virtualFilename: "analytics", configUrl: "/config/analytics.json" }, { from: "inline", virtualFilename: "temp", config: { serverMode: "chunked", urlPrefix: "https://cdn.example.com/chunks/temp/chunk", serverChunkSize: 2097152, databaseLengthBytes: 104857600, suffixLength: 3, requestChunkSize: 4096 } } ]; const worker = await createDbWorker(configs, workerUrl, wasmUrl); ``` -------------------------------- ### RangeMapper for Chunked Files Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/types.md Example implementation of RangeMapper for handling chunked files. It calculates the appropriate chunk URL and local byte offsets based on the requested range. ```typescript const rangeMapper = (from, to) => { const chunkSize = 1048576; // 1 MB const chunkId = Math.floor(from / chunkSize); const localFrom = from % chunkSize; const localTo = localFrom + (to - from); return { url: `https://example.com/db/chunk${String(chunkId).padStart(4, '0')}`, fromByte: localFrom, toByte: localTo }; }; ``` -------------------------------- ### Create and Query a Database Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/sql.js/README.md Demonstrates creating a new database, preparing and executing SQL statements with parameter binding, and fetching results. It also shows how to execute multiple SQL statements at once and retrieve all results. ```javascript // Create a database var db = new SQL.Database(); // NOTE: You can also use new SQL.Database(data) where // data is an Uint8Array representing an SQLite database file // Prepare an sql statement var stmt = db.prepare("SELECT * FROM hello WHERE a=:aval AND b=:bval"); // Bind values to the parameters and fetch the results of the query var result = stmt.getAsObject({':aval' : 1, ':bval' : 'world'}); console.log(result); // Will print {a:1, b:'world'} // Bind other values stmt.bind([0, 'hello']); while (stmt.step()) console.log(stmt.get()); // Will print [0, 'hello'] // free the memory used by the statement stmt.free(); // You can not use your statement anymore once it has been freed. // But not freeing your statements causes memory leaks. You don't want that. // Execute a single SQL string that contains multiple statements sqlstr = "CREATE TABLE hello (a int, b char);"; sqlstr += "INSERT INTO hello VALUES (0, 'hello');" sqlstr += "INSERT INTO hello VALUES (1, 'world');" db.run(sqlstr); // Run the query without returning anything var res = db.exec("SELECT * FROM hello"); /* [ {columns:['a','b'], values:[[0,'hello'],[1,'world']]} ] */ ``` -------------------------------- ### Browser: Basic Database Operations Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/sql.js/README.md Demonstrates creating a database, executing SQL statements like CREATE TABLE and INSERT, preparing statements, and fetching data in a browser environment. ```html Output is in Javascript console ``` -------------------------------- ### SQL REPL Initialization and Execution Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/sql.js/examples/repl.html Initializes sql.js, creates a new database, and sets up an event listener to execute SQL queries entered by the user. Results and errors are displayed on the page. ```javascript //Open a blank database var db; initSqlJs({ locateFile: filename => `../dist/${filename}` }).then(function (SQL) { db = new SQL.Database(); }); document.getElementById('submit').onclick = function () { var sql = document.getElementById('input').value; var result = '', error = ''; try { result = db.exec(sql); } catch (e) { error = e; } document.getElementById('result').innerHTML = JSON.stringify(result, null, ' '); document.getElementById('error').innerHTML = error; }; ``` -------------------------------- ### Get Database Statistics Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/INDEX.md Retrieves statistics for a specific virtual database file. Accessible via the worker proxy. ```typescript worker.getStats(virtualFilename?: string): Promise ``` -------------------------------- ### Initialize sql.js and Load/Create Database Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/sql.js/examples/persistent.html Configures sql.js to locate the WASM file and initializes the database. It loads data from localStorage if available, otherwise creates a new database with a 'views' table. ```javascript var baseUrl = '../dist/'; function toBinArray(str) { var l = str.length, arr = new Uint8Array(l); for (var i = 0; i < l; i++) arr[i] = str.charCodeAt(i); return arr; } function toBinString(arr) { var uarr = new Uint8Array(arr); var strings = [], chunksize = 0xffff; for (var i = 0; i * chunksize < uarr.length; i++) { strings.push(String.fromCharCode.apply(null, uarr.subarray(i * chunksize, (i + 1) * chunksize))); } return strings.join(''); } var config = { locateFile: filename => `${baseUrl}/${filename}` } initSqlJs(config).then(function (SQL) { var dbstr = window.localStorage.getItem("viewcount.sqlite"); if (dbstr) { var db = new SQL.Database(toBinArray(dbstr)); } else { var db = new SQL.Database(); db.run("CREATE TABLE views (date INTEGER PRIMARY KEY)"); } db.run("INSERT INTO views(date) VALUES (?)", [Date.now()]); document.getElementById('views').textContent = db.exec("SELECT COUNT(*) FROM views")[0].values[0][0]; var count = 0, dates = document.getElementById("dates"); db.each("SELECT date FROM views ORDER BY date ASC", function callback(row) { var li = document.createElement("li"); li.textContent = new Date(row.date); dates.appendChild(li); }, function done() { var dbstr = toBinString(db.export()); window.localStorage.setItem("viewcount.sqlite", dbstr); }); }); ``` -------------------------------- ### Create and Populate SQLite Database Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/README.md Creates a new SQLite database file and inserts sample data into a table. ```sh sqlite3 example.sqlite3 "create table mytable(foo, bar)" sqlite3 example.sqlite3 "insert into mytable values ('hello', 'world')" ``` -------------------------------- ### Initialize SQL.js and Create Database Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/sql.js/examples/simple.html Initializes SQL.js with a custom file locator and creates a new in-memory database. The `locateFile` function is crucial when the WASM file is not in the same directory as the HTML page. ```javascript config = { locateFile: (filename, prefix) => { console.log(`prefix is : ${prefix}`); return `../dist/${filename}`; } } // The `initSqlJs` function is globally provided by all of the main dist files if loaded in the browser. // We must specify this locateFile function if we are loading a wasm file from anywhere other than the current html page's folder. initSqlJs(config).then(function (SQL) { //Create the database var db = new SQL.Database(); // Run a query without reading the results db.run("CREATE TABLE test (col1, col2);"); // Insert two rows: (1,111) and (2,222) db.run("INSERT INTO test VALUES (?,?), (?,?)", [1, 111, 2, 222]); // Prepare a statement var stmt = db.prepare("SELECT * FROM test WHERE col1 BETWEEN $start AND $end"); stmt.getAsObject({ $start: 1, $end: 1 }); // {col1:1, col2:111} // Bind new values stmt.bind({ $start: 1, $end: 2 }); while (stmt.step()) { // var row = stmt.getAsObject(); console.log('Here is a row: ' + JSON.stringify(row)); } }); ``` -------------------------------- ### Get Database Statistics from Worker Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/types.md Demonstrates how to retrieve database statistics from a worker thread. This is useful for monitoring database access patterns and performance. ```typescript const stats = await worker.worker.getStats(); console.log(`Fetched ${stats.totalFetchedBytes} / ${stats.totalBytes} bytes in ${stats.totalRequests} requests`); ``` -------------------------------- ### Node.js: Read Database from Disk Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/sql.js/README.md Reads an SQLite database file from the disk in a Node.js environment using the `fs` module. Requires `sql-wasm.js` to be installed or available. ```javascript var fs = require('fs'); var initSqlJs = require('sql-wasm.js'); var filebuffer = fs.readFileSync('test.sqlite'); initSqlJs().then(function(SQL){ // Load the db var db = new SQL.Database(filebuffer); }); ``` -------------------------------- ### Initialize and Query with SQL.js-HTTPvfs Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/00-START-HERE.md Initializes a SQL.js-HTTPvfs worker with a database configuration and demonstrates how to perform a query and retrieve statistics. Ensure the worker and WASM files are correctly referenced. ```javascript import { createDbWorker } from "sql.js-httpvfs"; // Initialize const worker = await createDbWorker( [{ from: "inline", config: { serverMode: "full", url: "/database.sqlite3", requestChunkSize: 4096 } }], new URL("sql.js-httpvfs/dist/sqlite.worker.js", import.meta.url).toString(), new URL("sql.js-httpvfs/dist/sql-wasm.wasm", import.meta.url).toString() ); // Query const results = await worker.db.query("SELECT * FROM users"); console.log(results); // Monitor const stats = await worker.worker.getStats(); console.log(`Fetched ${stats.totalFetchedBytes} bytes`); ``` -------------------------------- ### Full File Mode Configuration Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/README.md Configure the server to serve the entire database file as a single unit. Specify the URL to the database file. ```javascript serverMode: "full" url: "/database.sqlite3" ``` -------------------------------- ### Configuration Options Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/MANIFEST.md Various options for configuring file loading and worker initialization. ```APIDOC ## Configuration Options ### Full File Mode - **serverMode** (boolean) - Enables server mode. - **url** (string) - The URL of the database file. - **requestChunkSize** (number) - The size of chunks to request. - **cacheBust** (boolean) - Enables cache busting. ### Chunked File Mode - **serverMode** (boolean) - Enables server mode. - **urlPrefix** (string) - The prefix for URLs of database chunks. - **serverChunkSize** (number) - The size of chunks served by the server. - **databaseLengthBytes** (number) - The total length of the database in bytes. - **suffixLength** (number) - The length of the suffix for chunked files. - **requestChunkSize** (number) - The size of chunks to request. - **cacheBust** (boolean) - Enables cache busting. ### Lazy File Config - **rangeMapper** (function) - A function to map byte ranges. - **fileLength** (number) - The total length of the file in bytes. - **requestChunkSize** (number) - The size of chunks to request. - **maxReadHeads** (number) - Maximum number of concurrent read heads. - **maxReadSpeed** (number) - Maximum read speed in bytes per second. - **logPageReads** (boolean) - Enables logging of page reads. - **requestLimiter** (function) - A callback function to limit requests. ### Worker Initialization - **configs** (Array) - An array of database configurations. - **workerUrl** (string) - The URL of the worker script. - **wasmUrl** (string) - The URL of the WASM module. - **maxBytesToRead** (number) - The maximum number of bytes to read at once. ``` -------------------------------- ### Get Accessed Pages Log Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/INDEX.md Retrieves a log of accessed pages for a specific virtual database file. Useful for performance analysis. Accessible via the worker proxy. ```typescript worker.getResetAccessedPages(virtualFilename?: string): Promise ``` -------------------------------- ### initSqlJs Function Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/sql.js/documentation_index.md The entry point for sql.js. It initializes the library and returns the main SqlJs object. ```APIDOC ## initSqlJs ### Description Initializes the sql.js library and returns the main `SqlJs` object. ### Parameters - **config** (`SqlJsConfig`) - Optional - Configuration object for sql.js initialization. ### Returns - **SqlJs** - The main sql.js object containing database and statement functionalities. ``` -------------------------------- ### Querying Multiple Databases Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/README.md Set up and query multiple databases simultaneously. Use virtual filenames to distinguish between them in queries. ```javascript const worker = await createDbWorker( [ { from: "inline", virtualFilename: "main", config: {...} }, { from: "inline", virtualFilename: "archive", config: {...} } ], workerUrl, wasmUrl ); // Query across databases const results = await worker.db.query(` SELECT m.*, a.data FROM main.table m LEFT JOIN archive.table a ON m.id = a.id `); ``` -------------------------------- ### Get Database File Access Statistics Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/api-reference/worker-utilities.md Retrieve statistics about database file access, including total bytes and requests. This helps in analyzing database efficiency and fetch patterns. ```typescript const worker = await createDbWorker([config], workerUrl, wasmUrl); // Run some queries await worker.db.query("SELECT * FROM large_table LIMIT 100"); // Get statistics const stats = await worker.worker.getStats(); console.log(`Database size: ${(stats.totalBytes / 1024 / 1024).toFixed(2)} MB`); console.log(`Fetched: ${(stats.totalFetchedBytes / 1024 / 1024).toFixed(2)} MB`); console.log(`Efficiency: ${(stats.totalFetchedBytes / stats.totalBytes * 100).toFixed(1)}%`); console.log(`Total requests: ${stats.totalRequests}`); ``` -------------------------------- ### Project File Structure Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/README.md Overview of the directory and file organization for the sql.js-httpvfs project. ```tree sql.js-httpvfs/ ├── src/ │ ├── index.ts # Main exports │ ├── db.ts # createDbWorker() and DOM VTable │ ├── sqlite.worker.ts # Worker module │ ├── lazyFile.ts # LazyUint8Array and HTTP fetching │ ├── vtab.ts # DOM virtual table implementation │ └── util.ts # Utility types ├── dist/ │ ├── index.js # Main bundle │ ├── sqlite.worker.js # Worker script │ └── sql-wasm.wasm # SQLite binary └── sql.js/ # sql.js submodule ``` -------------------------------- ### Handle Server Load Failures Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/errors.md Catch and handle errors when the database worker fails to load from the server. This example shows how to detect the specific error message and implement a retry mechanism. ```typescript try { const worker = await createDbWorker([config], workerUrl, wasmUrl); } catch (e) { if (e.message.includes("Couldn't load")) { console.error("Failed to load database from server:", e); // Retry with exponential backoff setTimeout(() => retryLoad(), 1000); } } ``` -------------------------------- ### Performance Monitoring with sql.js-httpvfs Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/README.md Retrieve performance statistics from the worker, including efficiency and total requests. Debug page access by getting and iterating over accessed pages to see cache hits and misses. ```javascript const stats = await worker.worker.getStats(); console.log(`Efficiency: ${(stats.totalFetchedBytes / stats.totalBytes * 100).toFixed(1)}%`); console.log(`Total requests: ${stats.totalRequests}`); // Debug page access const pages = await worker.worker.getResetAccessedPages(); pages.forEach(p => { console.log(`Page ${p.pageno}: ${p.wasCached ? 'hit' : 'miss'}`); }); ``` -------------------------------- ### Browser: Load Database from Server (fetch) Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/sql.js/README.md Loads an SQLite database file from a server using the Fetch API. Ensure the `locateFile` function points to the correct distribution folder. ```javascript const sqlPromise = initSqlJs({ locateFile: file => `https://path/to/your/dist/folder/dist/${file}` }); const dataPromise = fetch("/path/to/databse.sqlite").then(res => res.arrayBuffer()); const [SQL, buf] = await Promise.all([sqlPromise, dataPromise]) const db = new SQL.Database(new Uint8Array(buf)); ``` -------------------------------- ### API Reference Overview Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/_DOCUMENTATION_SUMMARY.txt The API reference is organized by functionality, with each function and class having its own dedicated page. This includes exact signatures, parameter tables, return type explanations, error conditions, and working code examples. ```APIDOC ## API Reference This section details the available functions and classes within the sql.js-httpvfs library. ### Functions and Classes - **createDbWorker**: Creates a new sql.js worker instance. - **LazyHttpDatabase**: Represents a database that can be loaded over HTTP. - **LazyUint8Array**: A utility for handling large byte arrays fetched over HTTP. - **createLazyFile**: Creates a virtual file that can be loaded lazily. - **toObjects**: Converts query results to an array of objects. - **worker-utilities**: Utilities for monitoring and managing workers. - **SeriesVtab**: A virtual table for handling series data. Each API reference page includes: - Exact signature from source code. - Parameter tables with types and descriptions. - Explanation of return types. - Catalog of error conditions. - Working code examples. - Source file and line number references. ``` -------------------------------- ### Basic Database Query with sql.js-httpvfs Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/README.md Initialize a database worker and execute a simple query. Ensure the worker and WASM files are correctly referenced. ```javascript import { createDbWorker } from "sql.js-httpvfs"; const worker = await createDbWorker( [{ from: "inline", config: { serverMode: "full", url: "/database.sqlite3", requestChunkSize: 4096 } }], new URL("sql.js-httpvfs/dist/sqlite.worker.js", import.meta.url).toString(), new URL("sql.js-httpvfs/dist/sql-wasm.wasm", import.meta.url).toString() ); const results = await worker.db.query("SELECT * FROM users"); console.log(results); ``` -------------------------------- ### Initialize SQL.js Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/sql.js/README.md Initialize the SQL.js library, specifying the location of the WASM binary. This is required before creating a database instance. The `locateFile` option is crucial for loading the WASM file correctly, especially in different environments like Node.js or when using bundlers. ```javascript const initSqlJs = require('sql.js'); // or if you are in a browser: // var initSqlJs = window.initSqlJs; const SQL = await initSqlJs({ // Required to load the wasm binary asynchronously. Of course, you can host it wherever you want // You can omit locateFile completely when running in node locateFile: file => `https://sql.js.org/dist/${file}` }); ``` -------------------------------- ### Handle DOM Virtual Table Errors Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/errors.md Recover from errors when interacting with the DOM virtual table, such as missing parent selectors or invalid tag names during INSERT operations. This example shows how to catch and log specific errors. ```typescript try { await worker.db.exec( `INSERT INTO dom (parent, tagName, textContent) VALUES ('body', 'div', 'Hello')` ); } catch (e) { if (e.message.includes("parent") && e.message.includes("must be set")) { // Parent selector missing - fix it console.error("ERROR: Must specify parent element for INSERT"); } else if (e.message.includes("could not be found")) { // Parent doesn't exist console.error("ERROR: Parent element not found in DOM"); // Inspect DOM first const available = await worker.db.query( `SELECT selector FROM dom WHERE selector MATCH 'body > *' LIMIT 5` ); console.log("Available parents:", available); } } ``` -------------------------------- ### Handle Invalid JSON Config in SQL.js-HTTPvfs Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/errors.md When the configuration file URL returns a non-2xx status or points to a non-existent file, a 'Could not load httpvfs config' error occurs. This example shows how to catch this error and attempt to use a fallback configuration. ```typescript try { const config = { from: "jsonconfig", configUrl: "/config.json" }; const worker = await createDbWorker([config], workerUrl, wasmUrl); } catch (e) { if (e.message.includes("Could not load httpvfs config")) { console.error("Config file unreachable. Checking alternate location..."); const fallback = { from: "inline", config: { /* hardcoded config */ } }; } } ``` -------------------------------- ### SQL Data Definition and Insertion Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/sql.js/examples/GUI/index.html This snippet defines an 'employees' table and inserts sample data into it. It's a common pattern for setting up a database for testing or demonstration. ```sql DROP TABLE IF EXISTS employees; CREATE TABLE employees( id integer, name text, designation text, manager integer, hired_on date, salary integer, commission float, dept integer); INSERT INTO employees VALUES (1,'JOHNSON','ADMIN',6,'1990-12-17',18000,NULL,4); INSERT INTO employees VALUES (2,'HARDING','MANAGER',9,'1998-02-02',52000,300,3); INSERT INTO employees VALUES (3,'TAFT','SALES I',2,'1996-01-02',25000,500,3); INSERT INTO employees VALUES (4,'HOOVER','SALES I',2,'1990-04-02',27000,NULL,3); INSERT INTO employees VALUES (5,'LINCOLN','TECH',6,'1994-06-23',22500,1400,4); INSERT INTO employees VALUES (6,'GARFIELD','MANAGER',9,'1993-05-01',54000,NULL,4); INSERT INTO employees VALUES (7,'POLK','TECH',6,'1997-09-22',25000,NULL,4); INSERT INTO employees VALUES (8,'GRANT','ENGINEER',10,'1997-03-30',32000,NULL,2); INSERT INTO employees VALUES (9,'JACKSON','CEO',NULL,'1990-01-01',75000,NULL,4); INSERT INTO employees VALUES (10,'FILLMORE','MANAGER',9,'1994-08-09',56000,NULL,2); INSERT INTO employees VALUES (11,'ADAMS','ENGINEER',10,'1996-03-15',34000,NULL,2); INSERT INTO employees VALUES (12,'WASHINGTON','ADMIN',6,'1998-04-16',18000,NULL,4); INSERT INTO employees VALUES (13,'MONROE','ENGINEER',10,'2000-12-03',30000,NULL,2); INSERT INTO employees VALUES (14,'ROOSEVELT','CPA',9,'1995-10-12',35000,NULL,1); ``` -------------------------------- ### Query Optimization with Covering Indexes Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/README.md Analyze query access patterns using 'EXPLAIN QUERY PLAN'. Improve performance by creating covering indexes that include all necessary columns, thus avoiding separate table lookups. ```javascript // Get query plan to understand access patterns const plan = await worker.db.query("EXPLAIN QUERY PLAN SELECT ..."); // Use COVERING INDEX to avoid table access // If index contains all needed columns, no separate table lookup needed await worker.db.exec( "CREATE INDEX idx_covering ON users(status, email, name)" ); // Check which pages are accessed const pages = await worker.worker.getResetAccessedPages(); ``` -------------------------------- ### Configure Multiple Databases with Virtual Filenames Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/configuration.md When attaching multiple databases, define each with a unique `virtualFilename`. This allows querying tables from different databases using their virtual names. ```typescript const configs = [ { from: "inline", virtualFilename: "main", config: { serverMode: "full", url: "/main.sqlite3", requestChunkSize: 4096 } }, { from: "inline", virtualFilename: "archive", config: { serverMode: "full", url: "/archive.sqlite3", requestChunkSize: 4096 } } ]; const worker = await createDbWorker(configs, workerUrl, wasmUrl); // Query tables from both databases const result = await worker.db.query(` SELECT m.id, m.name, a.data FROM main.users m LEFT JOIN archive.history a ON m.id = a.user_id `); ``` -------------------------------- ### Module Structure Overview Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/INDEX.md This diagram illustrates the main modules and exported functions of the sql.js-httpvfs library, indicating their source files. ```plaintext Export Root (src/index.ts) ├── createDbWorker() [src/db.ts] ├── toObjects() [src/sqlite.worker.ts] ├── types: │ ├── WorkerHttpvfs [src/db.ts] │ ├── LazyHttpDatabase [src/sqlite.worker.ts] │ ├── SplitFileConfig [src/sqlite.worker.ts] │ ├── SqliteStats [src/sqlite.worker.ts] │ └── PageReadLog [src/lazyFile.ts] ├── LazyUint8Array [src/lazyFile.ts] ├── createLazyFile() [src/lazyFile.ts] └── SeriesVtab [src/vtab.ts] (registered in worker) ``` -------------------------------- ### Create and Use Lazy File Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/api-reference/LazyUint8Array.md Demonstrates how to create a LazyUint8Array instance from a remote database file and access its statistics. Ensure that range requests are supported by the server for optimal performance. ```typescript import { createLazyFile } from "sql.js-httpvfs"; // Create a lazy file from a full database const lazyFile = createLazyFile( emscriptenFS, "/", "mydatabase.sqlite3", true, // canRead false, // canWrite { rangeMapper: (from, to) => ({ url: "https://example.com/database.sqlite3", fromByte: from, toByte: to }), requestChunkSize: 4096, fileLength: 1024 * 1024 * 100, // 100 MB logPageReads: true, maxReadHeads: 3, } ); // Get statistics console.log(`Total fetched: ${lazyFile.contents.totalFetchedBytes} bytes`); console.log(`Total requests: ${lazyFile.contents.totalRequests}`); console.log(`Page reads: ${lazyFile.contents.readPages.length}`); // View page access log lazyFile.contents.readPages.forEach(log => { console.log(`Page ${log.pageno}: ${log.wasCached ? 'cached' : 'fetched'}, prefetch=${log.prefetch}`); }); ``` -------------------------------- ### Load SQL.js with RequireJS Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/sql.js/examples/requireJS.html Configure RequireJS to load SQL.js from a specified base URL and initialize it with custom locateFile options. This is useful when the SQL.js library files are not in the same directory as your HTML. ```javascript var baseUrl = '../dist' require.config({ baseUrl: baseUrl }); // Options: 'sql-wasm', 'sql-asm', 'sql-asm-memory-growth.js', 'sql-wasm-debug', 'sql-asm-debug' require(['sql-wasm'], function success(initSqlJs) { console.log(typeof initSqlJs); if (typeof initSqlJs !== 'function') { document.body.style.backgroundColor = 'red'; console.log('initSqlJs returned: ', initSqlJs); alert("Failed to require sql.js through AMD"); return; } // The `initSqlJs` function is globally provided by all of the main dist files if loaded in the browser. // We must specify this locateFile function if we are loading a wasm file from anywhere other than the current html page's folder. var config = { locateFile: filename => `${baseUrl}/${filename}` } initSqlJs(config).then(function (SQL) { //Create the database var db = new SQL.Database(); // Run a query without reading the results db.run("CREATE TABLE test (col1, col2);"); // Insert two rows: (1,111) and (2,222) db.run("INSERT INTO test VALUES (?,?), (?,?)", [1, 111, 2, 222]); // Prepare a statement var stmt = db.prepare("SELECT * FROM test WHERE col1 BETWEEN $start AND $end"); stmt.getAsObject({ $start: 1, $end: 1 }); // {col1:1, col2:111} // Bind new values stmt.bind({ $start: 1, $end: 2 }); while (stmt.step()) { // var row = stmt.getAsObject(); // [...] do something with the row of result console.log('Here is a row: ', row); } }); }, function error(err) { document.body.style.backgroundColor = 'red'; console.log(err); alert("Module load failed: " + err); } ); // Output is in Javscript console ``` -------------------------------- ### Browser: Load Database from User File Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/sql.js/README.md Loads an SQLite database file selected by the user via an HTML input element. Requires an `dbFileElm` element in the HTML. ```javascript dbFileElm.onchange = () => { var f = dbFileElm.files[0]; var r = new FileReader(); r.onload = function() { var Uints = new Uint8Array(r.result); db = new SQL.Database(Uints); } r.readAsArrayBuffer(f); } ``` -------------------------------- ### Create Database Worker Source: https://github.com/phiresky/sql.js-httpvfs/blob/master/_autodocs/INDEX.md Main entry point for creating a worker for httpvfs. Requires configuration, worker URL, and WASM URL. Optionally set max bytes to read. ```typescript createDbWorker( configs: SplitFileConfig[], workerUrl: string, wasmUrl: string, maxBytesToRead?: number ): Promise ```