### Create plugin directory structure Source: https://github.com/simonw/sqlite-utils/blob/main/docs/plugins.rst To build a plugin, start by creating a directory named after your plugin. This example creates a `hello-world` directory. ```bash mkdir hello-world cd hello-world ``` -------------------------------- ### Install a sqlite-utils plugin Source: https://github.com/simonw/sqlite-utils/blob/main/docs/plugins.rst Use the `sqlite-utils install` command to add a plugin. This example shows installing a plugin named `sqlite-utils-name-of-plugin`. ```bash sqlite-utils install sqlite-utils-name-of-plugin ``` -------------------------------- ### List installed plugins Source: https://github.com/simonw/sqlite-utils/blob/main/docs/plugins.rst View a JSON list of all installed plugins using the `sqlite-utils plugins` command. ```bash sqlite-utils plugins ``` -------------------------------- ### Install sqlean.py with sqlite-utils Source: https://github.com/simonw/sqlite-utils/blob/main/docs/installation.rst Install the sqlean.py package, a drop-in replacement for Python's sqlite3 module with full extension loading support, using the sqlite-utils command. ```bash sqlite-utils install sqlean.py ``` -------------------------------- ### Install Python Packages Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli-reference.rst Install Python packages from PyPI into the same environment as sqlite-utils. Use `-U` to upgrade packages and `-e` for editable installs. ```bash sqlite-utils install [OPTIONS] [PACKAGES]... ``` -------------------------------- ### Run live documentation server with Just Source: https://github.com/simonw/sqlite-utils/blob/main/docs/contributing.rst Start the live documentation server, which includes running Cog first, using the Just command. ```bash just docs ``` -------------------------------- ### Install and Use sqlite-utils with Pyodide Source: https://github.com/simonw/sqlite-utils/blob/main/docs/changelog.rst Demonstrates how to install and use the sqlite-utils package within a Pyodide environment, enabling Python execution in the browser. ```python >>> import micropip >>> await micropip.install("sqlite-utils") >>> import sqlite_utils >>> db = sqlite_utils.Database(memory=True) >>> list(db.query("select 3 * 5")) [{'3 * 5': 15}] ``` -------------------------------- ### Query a View Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst After creating a view, you can query it like a regular table. This example selects all data from the 'version' view. ```bash sqlite-utils mydb.db "select * from version" ``` -------------------------------- ### Build documentation using uv Source: https://github.com/simonw/sqlite-utils/blob/main/docs/contributing.rst Build the project documentation locally using 'uv run' and 'make livehtml'. This also starts a live-reloading server. ```bash uv run make livehtml --directory docs ``` -------------------------------- ### Install sqlite-dump with sqlite-utils Source: https://github.com/simonw/sqlite-utils/blob/main/docs/installation.rst Install the sqlite-dump package, required for the .iterdump() method and the sqlite-utils dump command, using the sqlite-utils command. ```bash sqlite-utils install sqlite-dump ``` -------------------------------- ### Install Python Package Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst Install a Python package from PyPI into the sqlite-utils environment. This command is a wrapper around 'pip install'. Use -U to upgrade. ```bash sqlite-utils install beautifulsoup4 ``` -------------------------------- ### Install sqlite-utils with pip Source: https://github.com/simonw/sqlite-utils/blob/main/docs/installation.rst Install the sqlite-utils package, which includes both the Python library and the command-line tool, using pip. ```bash pip install sqlite-utils ``` -------------------------------- ### Install plugin in editable mode Source: https://github.com/simonw/sqlite-utils/blob/main/docs/plugins.rst Install your plugin in editable mode using `sqlite-utils install -e .` to immediately reflect code changes. Alternatively, provide the path to your plugin directory. ```bash sqlite-utils install -e . sqlite-utils install -e /dev/sqlite-utils-hello-world ``` -------------------------------- ### Install sqlite-utils with Homebrew Source: https://github.com/simonw/sqlite-utils/blob/main/docs/installation.rst Use Homebrew to install the sqlite-utils command-line tool on macOS. This command installs the latest release. ```bash brew install sqlite-utils ``` -------------------------------- ### Install and Uninstall Packages with sqlite-utils Source: https://github.com/simonw/sqlite-utils/blob/main/docs/changelog.rst Use sqlite-utils install and uninstall commands to manage packages within the same virtual environment as sqlite-utils. ```bash sqlite-utils install requests ``` ```bash sqlite-utils uninstall requests ``` -------------------------------- ### Create a View Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst Use the `create-view` command to define a new view in the database. This example creates a view named 'version' that selects the SQLite version. ```bash sqlite-utils create-view mydb.db version "select sqlite_version()" ``` -------------------------------- ### Execute a custom plugin command Source: https://github.com/simonw/sqlite-utils/blob/main/docs/plugins.rst After installing a plugin with a custom command (e.g., `hello-world`), you can execute it directly from the command line. ```bash sqlite-utils hello-world ``` -------------------------------- ### List Plugins Command Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli-reference.rst This command lists the installed plugins for sqlite-utils. It has no required arguments. ```bash Usage: sqlite-utils plugins [OPTIONS] List installed plugins Options: -h, --help Show this message and exit. ``` -------------------------------- ### Install sqlite-utils with pipx Source: https://github.com/simonw/sqlite-utils/blob/main/docs/installation.rst Use pipx to install the sqlite-utils command-line tool in an isolated environment. This is recommended for command-line applications. ```bash pipx install sqlite-utils ``` -------------------------------- ### Clone the sqlite-utils repository Source: https://github.com/simonw/sqlite-utils/blob/main/docs/contributing.rst Checkout the code from the GitHub repository to start local development. ```bash git clone git@github.com:simonw/sqlite-utils cd sqlite-utils ``` -------------------------------- ### Install Trogon for sqlite-utils TUI Source: https://github.com/simonw/sqlite-utils/blob/main/docs/changelog.rst Install the optional Trogon dependency to use the experimental sqlite-utils tui interface. ```bash sqlite-utils install trogon ``` -------------------------------- ### Install sqlite-dump for iterdump fallback Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Installs the 'sqlite-dump' package, which provides an implementation for db.iterdump() if the underlying pysqlite3 or sqlean.py method is missing. ```bash pip install sqlite-dump ``` -------------------------------- ### Output of Inspect Table Schema Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst Example output showing the schema of a table. ```output table schema -------------------------------- mytable CREATE TABLE "mytable" ( "id" INTEGER PRIMARY KEY, "name" TEXT NOT NULL, "age" INTEGER NOT NULL, "is_good" INTEGER DEFAULT '1' ) ``` -------------------------------- ### Execute SQL on In-Memory Database with Imported Files Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli-reference.rst This example demonstrates how to import data from CSV and JSON files into an in-memory database and then execute a SQL query joining the data from both files. The file formats are automatically detected. ```bash sqlite-utils memory one.csv two.json \ "select * from one join two on one.two_id = two.id" ``` -------------------------------- ### Execute SQL on In-Memory Database from Standard Input Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli-reference.rst This example shows how to pipe data from a CSV file into the 'memory' command and execute a SQL query against it. The data is read from standard input, aliased as 'stdin'. ```bash cat animals.csv | sqlite-utils memory - \ "select * from stdin where species = 'dog'" ``` -------------------------------- ### Create an empty SQLite database Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst Use the 'create-database' command to create a new, empty SQLite database file. This command is fundamental for starting new projects. ```bash sqlite-utils create-database empty.db ``` -------------------------------- ### SQL Schema for Multi-Column Conversion Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst Example SQL schema created after converting a column into multiple columns. ```sql CREATE TABLE "places" ( "location" TEXT, "latitude" REAL, "longitude" REAL ); ``` -------------------------------- ### Load SQLite Extension Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst Load SQLite extension modules using the --load-extension option. This example loads the spatialite extension. ```bash sqlite-utils dogs.db "select spatialite_version()" --load-extension=spatialite ``` -------------------------------- ### Create and Populate a New SQLite Database Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Creates a new SQLite database file named 'chickens.db', adds a 'chickens' table, and inserts four records into it. This is a common starting point for using the library. ```python from sqlite_utils import Database db = Database("chickens.db") db.table("chickens").insert_all([{ "name": "Azi", "color": "blue", }, { "name": "Lila", "color": "blue", }, { "name": "Suna", "color": "gold", }, { "name": "Cardi", "color": "black", }]) ``` -------------------------------- ### View Schema of Imported Files Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli-reference.rst This example shows how to use the '--schema' option with the 'memory' command to view the SQL schema of data imported from a CSV file. ```bash sqlite-utils memory animals.csv --schema ``` -------------------------------- ### Import Python Modules for Conversion Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst Specify Python modules to be imported and used in conversion expressions with the --import option. This example uses textwrap. ```bash sqlite-utils convert content.db articles content \ '"\n".join(textwrap.wrap(value, 100))' \ --import=textwrap ``` -------------------------------- ### Table Creation with Options Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Demonstrates creating a table with various options like primary key, existence checks, and transformation. ```APIDOC ## Table Creation ### Description Creates a new table with specified columns and configuration. Supports options for primary keys, column order, foreign keys, not null constraints, default values, existence checks, and table transformation. ### Method `db.table("table_name").create(columns, **options)` ### Parameters - **columns** (dict): A dictionary mapping column names to their types. - **pk** (str or tuple, optional): The primary key column(s). - **column_order** (list, optional): The desired order of columns. - **foreign_keys** (list, optional): Foreign key constraints. - **not_null** (set, optional): Columns that cannot be null. - **defaults** (dict, optional): Default values for columns. - **if_not_exists** (bool, optional): If True, uses `CREATE TABLE IF NOT EXISTS`. - **ignore** (bool, optional): If True, ignores `OperationalError` if table exists. - **replace** (bool, optional): If True, drops and recreates the table. - **transform** (bool, optional): If True, transforms an existing table to match the new schema. - **strict** (bool, optional): If True, creates the table in `STRICT` mode. ### Request Example ```python db.table("cats").create({ "id": int, "name": str, }, pk="id", if_not_exists=True) db.table("cats").create({ "id": int, "name": str, "weight": float, }, pk="id", transform=True) db.table("cats").create({ "id": int, "breed": str, "name": str, "weight": float, }, pk=("breed", "id")) db.table("cats").create({ "id": int, "name": str, }, strict=True) ``` ### Response No specific response details are provided for table creation itself, but errors like `sqlite_utils.utils.sqlite3.OperationalError` may be raised if the table already exists and `ignore` or `if_not_exists` are not used. ``` -------------------------------- ### Explicitly Specify Input File Formats Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli-reference.rst This example demonstrates how to explicitly specify the format for input files when using the 'memory' command. It imports data from a CSV file via stdin and a newline-delimited JSON file, then queries them. ```bash cat animals.csv | sqlite-utils memory stdin:csv places.dat:nl \ "select * from stdin where place_id in (select id from places)" ``` -------------------------------- ### Specify Foreign Keys with Table Name and Column Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Sets up foreign key constraints for a table. This example explicitly defines a foreign key relationship from `author_id` to the `id` column in the `authors` table. ```python foreign_keys=[ ("author_id", "authors", "id") ] ``` -------------------------------- ### Analyze tables save JSON example Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst Example format for the 'most_common' and 'least_common' columns in the _analyze_tables_ table. ```json [ "value", "value" ] ``` -------------------------------- ### Initialize SpatiaLite on a New Database Source: https://github.com/simonw/sqlite-utils/blob/main/docs/changelog.rst Demonstrates the command-line option to initialize a newly created SQLite database with SpatiaLite support. ```bash sqlite-utils create-database --init-spatialite mydatabase.db ``` -------------------------------- ### Database Connection and Creation Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Demonstrates how to create a Database object by connecting to an existing SQLite file, creating a new one, or using an in-memory database. ```APIDOC ## Database Connection and Creation ### Description Objects are constructed by passing in either a path to a file on disk or an existing SQLite3 database connection. A new file will be created if it does not already exist. ### Connecting to a database file ```python from sqlite_utils import Database db = Database("my_database.db") ``` ### Recreating a database file To recreate a database from scratch (first removing the existing file from disk if it already exists), use the ``recreate=True`` argument: ```python db = Database("my_database.db", recreate=True) ``` ### Connecting with an existing SQLite connection Instead of a file path, you can pass in an existing SQLite connection: ```python import sqlite3 db = Database(sqlite3.connect("my_database.db")) ``` ### In-memory database To create an in-memory database: ```python db = Database(memory=True) ``` ### Named in-memory database Create a named in-memory database. These can be accessed by multiple threads if at least one reference to the database still exists. `del db` will clear the database from memory. ```python db = Database(memory_name="my_shared_database") ``` ### Connection options Connections use ``PRAGMA recursive_triggers=on`` by default. To turn them off: ```python db = Database(memory=True, recursive_triggers=False) ``` To opt out of executing plugins: ```python db = Database(memory=True, execute_plugins=False) ``` To enable SQLite STRICT mode for all tables created using this database object: ```python db = Database("my_database.db", strict=True) ``` ``` -------------------------------- ### Register a custom SQL function with prepare_connection hook Source: https://github.com/simonw/sqlite-utils/blob/main/docs/plugins.rst The `prepare_connection` hook allows you to register custom SQL functions, aggregates, or collations. This example registers a `hello` function that returns a greeting. ```python import sqlite_utils @sqlite_utils.hookimpl def prepare_connection(conn): conn.create_function( "hello", 1, lambda name: f"Hello, {name}!" ) ``` -------------------------------- ### Initialize Database with a tracer Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Pass a tracer function to the Database constructor to enable query logging during initialization. ```python db = Database(memory=True, tracer=tracer) ``` -------------------------------- ### Load SQLite extension for SpatiaLite Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst Loads the SpatiaLite extension into the SQLite database. The `spatialite` keyword is a shortcut to find the extension in common locations. This example queries the SpatiaLite version. ```bash sqlite-utils memory "select spatialite_version()" --load-extension=spatialite ``` -------------------------------- ### Transform Table Schema (Dry Run) Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst Applies complex schema transformations to a table and shows the generated SQL without executing it. This example renames a primary key, sets a default value, reorders columns, and drops a column. ```bash sqlite-utils transform fixtures.db roadside_attractions \ --rename pk id \ --default name Untitled \ --column-order id \ --column-order longitude \ --column-order latitude \ --drop address \ --sql ``` -------------------------------- ### Uninstall Python Package Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst Uninstall a package previously installed with 'sqlite-utils install'. Use -y to skip confirmation prompts. ```bash sqlite-utils uninstall beautifulsoup4 ``` -------------------------------- ### Query data from GitHub API with sqlite-utils memory Source: https://github.com/simonw/sqlite-utils/blob/main/docs/changelog.rst Use `sqlite-utils memory` to query data directly from a JSON API. This example fetches GitHub repositories, sorts them by stars, and displays the top 5. ```bash $ curl -s 'https://api.github.com/users/dogsheep/repos' \ | sqlite-utils memory - \ 'select full_name, forks_count, stargazers_count from stdin order by stargazers_count desc limit 5 ' -t ``` -------------------------------- ### List tables with schema Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst Use the --schema option to include the schema of each table when listing tables. The --table option formats the output as a table. ```bash sqlite-utils tables dogs.db --schema --table ``` -------------------------------- ### Create Table with Constraints and Defaults Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst Create a table with NOT NULL constraints and default values for columns. ```bash sqlite-utils create-table mydb.db mytable id integer name text age integer is_good integer --not-null name --not-null age --default is_good 1 --pk=id ``` -------------------------------- ### Create a database with SpatiaLite and custom extension loading Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst When SpatiaLite is not in a default location, use '--load-extension' to specify the path to the SpatiaLite library. This ensures SpatiaLite can be correctly initialized. ```bash sqlite-utils create-database empty.db --init-spatialite --load-extension /path/to/spatialite.so ``` -------------------------------- ### List views with counts, columns, and schema Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst The views command shows defined views. Use --table, --counts, --columns, and --schema to display detailed information about each view. ```bash sqlite-utils views sf-trees.db --table --counts --columns --schema ``` -------------------------------- ### Get Foreign Key Relationships Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Access the .foreign_keys property to get a list of ForeignKey named tuples, detailing the foreign key relationships for a table. This property is not available on views. ```python db.table("Street_Tree_List").foreign_keys ``` -------------------------------- ### Import and Use sqlite-utils Library Source: https://github.com/simonw/sqlite-utils/blob/main/README.md Import the library and create a database connection. This snippet demonstrates creating a table and inserting data, using 'id' as the primary key. ```python import sqlite_utils db = sqlite_utils.Database("demo_database.db") # This line creates a "dogs" table if one does not already exist: db["dogs"].insert_all([ {"id": 1, "age": 4, "name": "Cleo"}, {"id": 2, "age": 2, "name": "Pancakes"} ], pk="id") ``` -------------------------------- ### Get Table Columns as Dictionary Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Use the .columns_dict property to get a dictionary mapping column names to their Python types. This is a convenient way to quickly check the data types of columns. ```python db.table("PlantType").columns_dict ``` -------------------------------- ### Get SQLite Version Source: https://github.com/simonw/sqlite-utils/blob/main/docs/changelog.rst Accesses the SQLite version as a tuple of integers. ```python db.sqlite_version ``` -------------------------------- ### Run all tests and linters with Just Source: https://github.com/simonw/sqlite-utils/blob/main/docs/contributing.rst Execute all defined tests and linters using the Just command-line tool. ```bash just ``` -------------------------------- ### Create a database with WAL mode enabled Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst To enable Write-Ahead Logging (WAL) mode on a newly created database, use the '--enable-wal' option with the 'create-database' command. WAL mode can improve concurrency. ```bash sqlite-utils create-database empty.db --enable-wal ``` -------------------------------- ### Getting the database schema Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Retrieves the full SQL schema of the database as a string. ```APIDOC ## Getting the database schema The `db.schema` property returns the full SQL schema for the database as a string. ### Property `db.schema` ### Request Example ```python print(db.schema) ``` ### Response A string containing the SQL schema. ``` -------------------------------- ### Listing View Names Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Get a list of all view names present in the database. ```APIDOC ## Listing View Names Shows a list of views in the database. ### Method `.view_names()` ### Endpoint N/A (Python API) ### Parameters N/A ### Request Example ```python db.view_names() ``` ### Response A list of view names (strings). ``` -------------------------------- ### Query CLI Help Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli-reference.rst Shows the help text for the 'query' sub-command, which executes SQL queries against a SQLite database and returns results as JSON. It supports parameter binding for dynamic queries. ```bash Usage: sqlite-utils query [OPTIONS] PATH SQL Execute SQL query and return the results as JSON Example: sqlite-utils data.db \ "select * from chickens where age > :age" \ -p age 1 ``` -------------------------------- ### Register Deterministic Function Source: https://github.com/simonw/sqlite-utils/blob/main/docs/changelog.rst Example of registering a function with deterministic=True, with a note on SQLite version compatibility. ```python register_function(my_func, deterministic=True) ``` -------------------------------- ### Define plugin metadata with pyproject.toml Source: https://github.com/simonw/sqlite-utils/blob/main/docs/plugins.rst The `pyproject.toml` file describes your plugin, including its name, version, and entry points. The `[project.entry-points.sqlite_utils]` section specifies the module to load. ```toml [project] name = "sqlite-utils-hello-world" version = "0.1" [project.entry-points.sqlite_utils] hello_world = "sqlite_utils_hello_world" ``` -------------------------------- ### List available Just commands Source: https://github.com/simonw/sqlite-utils/blob/main/docs/contributing.rst Display all available commands managed by the Just tool. ```bash just -l ``` -------------------------------- ### View Database Tables Source: https://github.com/simonw/sqlite-utils/blob/main/docs/tutorial.ipynb Access `db.tables` to get a list of all tables currently present in the database. ```python db.tables ``` -------------------------------- ### Load SpatiaLite Extension Source: https://github.com/simonw/sqlite-utils/blob/main/docs/changelog.rst Load the SpatiaLite extension for spatial data operations. Requires SpatiaLite to be installed. ```bash --load-extension=spatialite ``` -------------------------------- ### Rename Columns Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Rename columns using the `rename` parameter. This example renames the 'age' column to 'initial_age'. ```python # Rename 'age' to 'initial_age': table.transform(rename={"age": "initial_age"}) ``` -------------------------------- ### Create a View Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli-reference.rst Create a database view based on a provided SQL SELECT query. The `--ignore` option prevents errors if the view already exists, while `--replace` will overwrite it. ```bash sqlite-utils create-view chickens.db heavy_chickens \ 'select * from chickens where weight > 3' ``` -------------------------------- ### Apply Migrations using CLI Source: https://github.com/simonw/sqlite-utils/blob/main/docs/migrations.rst Execute migrations from a Python file using the `sqlite-utils migrate` command, specifying the database file and the path to the migration script. ```bash sqlite-utils migrate creatures.db path/to/migrations.py ``` -------------------------------- ### Commit version update for release Source: https://github.com/simonw/sqlite-utils/blob/main/docs/contributing.rst Example commit message for updating the version number and changelog before a release. ```bash # Update changelog git commit -m " Release 3.29 Refs #423, #458, #467, #469, #470, #471, #472, #475" -a git push ``` -------------------------------- ### Get Last Primary Key Source: https://github.com/simonw/sqlite-utils/blob/main/docs/tutorial.ipynb Retrieve the primary key of the most recently inserted record using `table.last_pk`. ```python table.last_pk ``` -------------------------------- ### Access Table Schema Source: https://github.com/simonw/sqlite-utils/blob/main/docs/tutorial.ipynb Get a reference to a table and print its schema to understand the column types and constraints. ```python table = db["creatures"] ``` ```python print(table.schema) ``` -------------------------------- ### Create Table Indexes and Views Source: https://github.com/simonw/sqlite-utils/blob/main/docs/changelog.rst Use `table.create_index()` to define indexes on table columns and `db.create_view()` to create SQL views. ```python table.create_index(columns, index_name) ``` ```python db.create_view(name, sql) ``` -------------------------------- ### Find SpatiaLite using Utility Function Source: https://github.com/simonw/sqlite-utils/blob/main/docs/changelog.rst The `sqlite_utils.utils.find_spatialite()` function helps locate SpatiaLite installations in common directories. ```python from sqlite_utils.utils import find_spatialite spatialite_path = find_spatialite() ``` -------------------------------- ### Upgrade sqlite-utils with Homebrew Source: https://github.com/simonw/sqlite-utils/blob/main/docs/installation.rst If sqlite-utils is already installed via Homebrew, use this command to upgrade to the most recent release. ```bash brew upgrade sqlite-utils ``` -------------------------------- ### Drop Columns Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Remove columns from a table by passing their names in the `drop` set. This example drops the 'age' column. ```python # Drop the 'age' column: table.transform(drop={"age"}) ``` -------------------------------- ### Inserting data and querying Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Shows how to create a table, insert multiple records, and retrieve data using iteration and SQL queries. ```APIDOC ## Inserting data and querying ### Description This section demonstrates basic data manipulation: creating a table, inserting records, iterating over rows, and executing SQL queries. ### Creating a table and inserting records ```python from sqlite_utils import Database db = Database("chickens.db") db.table("chickens").insert_all([ {"name": "Azi", "color": "blue"}, {"name": "Lila", "color": "blue"}, {"name": "Suna", "color": "gold"}, {"name": "Cardi", "color": "black"}, ]) ``` ### Iterating over rows ```python for row in db.table("chickens").rows: print(row) # Output: # {'name': 'Azi', 'color': 'blue'} # {'name': 'Lila', 'color': 'blue'} # {'name': 'Suna', 'color': 'gold'} # {'name': 'Cardi', 'color': 'black'} ``` ### Executing SQL queries Use `db.query()` to run SQL queries: ```python for row in db.query(""" select color, count(*) from chickens group by color order by count(*) desc """): print(row) # Output: # {'color': 'blue', 'count(*)': 2} # {'color': 'gold', 'count(*)': 1} # {'color': 'black', 'count(*)': 1} ``` ``` -------------------------------- ### Setting defaults and not null constraints Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Demonstrates how to set NOT NULL constraints and default values for columns when creating or inserting data into tables using methods like `create_table`, `insert`, `insert_all`, `upsert`, and `upsert_all`. ```APIDOC ## Setting defaults and not null constraints Each of the methods that can cause a table to be created take optional arguments ``not_null=set()`` and ``defaults=dict()``. The methods that take these optional arguments are: * ``db.create_table(...)`` * ``table.create(...)`` * ``table.insert(...)`` * ``table.insert_all(...)`` * ``table.upsert(...)`` * ``table.upsert_all(...)`` You can use ``not_null=`` to pass a set of column names that should have a ``NOT NULL`` constraint set on them when they are created. You can use ``defaults=`` to pass a dictionary mapping columns to the default value that should be specified in the ``CREATE TABLE`` statement. Here's an example that uses these features: .. code-block:: python db.table("authors").insert_all( [{"id": 1, "name": "Sally", "score": 2}], pk="id", not_null={"name", "score"}, defaults={"score": 1}, ) db.table("authors").insert({"name": "Dharma"}) list(db.table("authors").rows) # Outputs: # [{'id': 1, 'name': 'Sally', 'score': 2}, # {'id': 3, 'name': 'Dharma', 'score': 1}] print(db.table("authors").schema) # Outputs: # CREATE TABLE "authors" ( # "id" INTEGER PRIMARY KEY, # "name" TEXT NOT NULL, # "score" INTEGER NOT NULL DEFAULT 1 # ) ``` -------------------------------- ### Create Table with Primary Key Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst Create a new table with specified columns and set a primary key. ```bash sqlite-utils create-table mydb.db mytable id integer name text --pk=id ``` -------------------------------- ### Get All Database Triggers as Dictionary Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst The .triggers_dict property on the database object returns all triggers across all tables as a dictionary. ```python >>> db.triggers_dict {'authors_ai': 'CREATE TRIGGER [authors_ai] AFTER INSERT...', 'authors_ad': 'CREATE TRIGGER [authors_ad] AFTER DELETE...', 'authors_au': 'CREATE TRIGGER [authors_au] AFTER UPDATE'} ``` -------------------------------- ### Get database schema Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Retrieves the full SQL schema of the database as a string. Useful for inspecting the database structure. ```python print(db.schema) ``` -------------------------------- ### Create a Table if it Does Not Exist Source: https://github.com/simonw/sqlite-utils/blob/main/docs/changelog.rst Illustrates how to create a table in SQLite, ensuring it is only created if it does not already exist. ```python db[table].create(..., if_not_exists=True) ``` -------------------------------- ### Introspect Database Tables and Indexes Source: https://github.com/simonw/sqlite-utils/blob/main/docs/changelog.rst Access database introspection properties like `db.tables` and `db.indexes` to get information about the database schema. ```python db.tables ``` ```python db.indexes ``` -------------------------------- ### Alter Column Types Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Change the data types of columns using the `types` argument. This example converts 'age' to integer and 'weight' to float. ```python # Convert the 'age' column to an integer, and 'weight' to a float table.transform(types={"age": int, "weight": float}) ``` -------------------------------- ### Connect Using an Existing SQLite3 Connection Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Initializes a Database object using a pre-existing `sqlite3.Connection` object. This allows integration with existing database connections. ```python import sqlite3 db = Database(sqlite3.connect("my_database.db")) ``` -------------------------------- ### Get Table Triggers as Dictionary Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Use .triggers_dict on a table object to retrieve triggers as a dictionary mapping trigger names to their SQL definitions. ```python >>> db.table("authors").triggers_dict {'authors_ai': 'CREATE TRIGGER [authors_ai] AFTER INSERT...', 'authors_ad': 'CREATE TRIGGER [authors_ad] AFTER DELETE...', 'authors_au': 'CREATE TRIGGER [authors_au] AFTER UPDATE'} ``` -------------------------------- ### List Tables with Columns and Counts Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst Combine --counts and --columns options with the 'tables' command to display both the row count and a list of columns for each table in the database. ```bash sqlite-utils tables dogs.db --counts --columns ``` ```output ``` -------------------------------- ### Create a Database with a Named In-Memory Instance Source: https://github.com/simonw/sqlite-utils/blob/main/docs/changelog.rst Shows how to create a named in-memory SQLite database that can be shared across multiple connections. ```python Database(memory_name="my_shared_database") ``` -------------------------------- ### Inspect Database Tables with Schema Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst List all tables in a database along with their schemas. ```bash sqlite-utils tables books.db --schema -t ``` -------------------------------- ### Get row count of a table Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Accesses the .count property to retrieve the total number of rows in a table. This is equivalent to running 'select count(*) from table'. ```python db.table("PlantType").count ``` ```python db.table("Street_Tree_List").count ``` -------------------------------- ### Execute SQL and get a cursor Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Use `db.execute(sql)` to run SQL statements that do not return rows, such as INSERT or UPDATE. It returns a sqlite3.Cursor object. ```python db = Database(memory=True) db.table("dogs").insert({"name": "Cleo"}) cursor = db.execute("update dogs set name = 'Cleopaws'") print(cursor.rowcount) # Outputs the number of rows affected by the update # In this case 2 ``` -------------------------------- ### Apply Migrations from a Directory Source: https://github.com/simonw/sqlite-utils/blob/main/docs/migrations.rst Specify a directory to search for `migrations.py` files recursively when using the `sqlite-utils migrate` command. ```bash sqlite-utils migrate creatures.db path/to/project/ ``` -------------------------------- ### Create a table with specified columns Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli-reference.rst Add a table with specified columns to an existing or new database. Columns require a name and type. Primary keys, NOT NULL constraints, default values, and foreign keys can be defined. ```bash sqlite-utils create-table my.db people \ id integer \ name text \ height float \ photo blob --pk id ``` -------------------------------- ### Query Results as JSON Arrays Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst Use the --arrays flag to get results as JSON arrays instead of objects. This can be combined with --nl. ```bash sqlite-utils dogs.db "select * from dogs" --arrays ``` ```output [[1, 4, "Cleo"], [2, 2, "Pancakes"]] ``` ```bash sqlite-utils dogs.db "select * from dogs" --arrays --nl ``` ```output [1, 4, "Cleo"] [2, 2, "Pancakes"] ``` -------------------------------- ### Apply Black formatting with Just Source: https://github.com/simonw/sqlite-utils/blob/main/docs/contributing.rst Format the code using Black via the Just command. ```bash just black ``` -------------------------------- ### Inspect Database Schema and Data Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Provides examples of how to inspect the database schema, table names, rows, and the schema of a junction table after creating a many-to-many relationship. ```python >>> db.table_names() ['dogs', 'characteristics', 'characteristics_dogs'] >>> list(db.table("dogs").rows) [{'id': 1, 'name': 'Cleo'}] >>> list(db.table("characteristics").rows) [{'id': 1, 'name': 'Playful'}, {'id': 2, 'name': 'Opinionated'}] >>> list(db.table("characteristics_dogs").rows) [{'characteristics_id': 1, 'dogs_id': 1}, {'characteristics_id': 2, 'dogs_id': 1}] >>> print(db.table("characteristics_dogs").schema) CREATE TABLE "characteristics_dogs" ( "characteristics_id" INTEGER REFERENCES "characteristics"("id"), "dogs_id" INTEGER REFERENCES "dogs"("id"), PRIMARY KEY ("characteristics_id", "dogs_id") ) ``` -------------------------------- ### Run sqlite-utils using uv Source: https://github.com/simonw/sqlite-utils/blob/main/docs/contributing.rst Use the 'uv run' command to execute the development version of the sqlite-utils tool. ```bash uv run sqlite-utils --help ``` -------------------------------- ### Connect to an Existing SQLite Database File Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Creates a Database object by connecting to an existing SQLite file. The file will be created if it does not exist. ```python from sqlite_utils import Database db = Database("my_database.db") ``` -------------------------------- ### Add a Foreign Key Constraint (Inferred) Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst If the referenced table and column are omitted, `sqlite-utils` attempts to guess them. This example adds a foreign key constraint to `books.author_id`. ```bash sqlite-utils add-foreign-key books.db books author_id ``` -------------------------------- ### Create Index CLI Source: https://github.com/simonw/sqlite-utils/blob/main/docs/changelog.rst Use the `sqlite-utils create-index` command to create indexes on tables from the command line. Specify columns as arguments. ```bash $ sqlite-utils create-index mydb.db mytable col1 [col2...] ``` -------------------------------- ### Insert files with renamed column Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst Demonstrates how to rename a column during file insertion using the -c colname:coldef syntax, for example, renaming 'mtime' to 'last_modified'. ```bash sqlite-utils insert-files gifs.db images *.gif \ -c path -c md5 -c last_modified:mtime --pk=path ``` -------------------------------- ### Execute a SQL query and iterate over results Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Use `db.query(sql)` to execute a SQL SELECT statement and get an iterator of dictionaries, where each dictionary represents a row. ```python db = Database(memory=True) db.table("dogs").insert_all([{"name": "Cleo"}, {"name": "Pancakes"}]) for row in db.query("select * from dogs"): print(row) # Outputs: # {'name': 'Cleo'} # {'name': 'Pancakes'} ``` -------------------------------- ### Create Table with Foreign Key Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst Create tables with foreign key relationships between them. ```bash sqlite-utils create-table books.db authors id integer name text --pk=id sqlite-utils create-table books.db books id integer title text author_id integer --pk=id --fk author_id authors id ``` -------------------------------- ### Chain Table Methods and Get Last Inserted ID Source: https://github.com/simonw/sqlite-utils/blob/main/docs/changelog.rst Table methods can now be chained for convenience. Access the ID of the last inserted row using `table.last_id`. ```python table.last_id ``` -------------------------------- ### List indexes for all tables Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst The indexes command lists all indexes configured for the database. Use --table to format the output. ```bash sqlite-utils indexes covid.db --table ``` -------------------------------- ### Convert column data with WHERE clause Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst Applies data conversion to specific rows in a table that match a given WHERE clause. This example converts headlines containing 'cat' to uppercase. ```bash sqlite-utils convert content.db articles headline 'value.upper()' \ --where "headline like '%cat%'" ``` -------------------------------- ### List indexes for specific tables Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst To see indexes for specific tables, list the table names after the database file. The --table option formats the output. ```bash sqlite-utils indexes covid.db johns_hopkins_csse_daily_reports --table ``` -------------------------------- ### Convert Lines to Structured Data Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst When using --lines, the --convert function receives a 'line' string. This example parses log lines into a structured dictionary. ```bash sqlite-utils insert logs.db loglines access.log --convert ' type, source, _, verb, path, _, status, _ = line.split() return { "type": type, "source": source, "verb": verb, "path": path, "status": status, }' --lines ``` -------------------------------- ### Access a table by name Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Use `db.table(table_name)` to get a reference to a database table. This method can also be used to create a table if it does not exist, using keyword arguments. ```python table = db.table("my_table") ``` -------------------------------- ### Create a table with a primary key by inserting data Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Creates a new table and specifies a primary key by passing the `pk=` parameter during the first insertion. This is only effective if the table does not already exist. ```python dogs.insert({ "id": 1, "name": "Cleo", "twitter": "cleopaws", "age": 3, "is_good_dog": True, }, pk="id") ``` -------------------------------- ### Run linters with Just Source: https://github.com/simonw/sqlite-utils/blob/main/docs/contributing.rst Execute only the linting tools using the Just command. ```bash just lint ``` -------------------------------- ### Get Table Schema as SQL Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst The .schema property outputs the table's schema as a SQL CREATE TABLE string. This is useful for understanding the exact structure and constraints of the table. ```python print(db.table("Street_Tree_List").schema) ``` -------------------------------- ### Get Default Values for Columns Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst The .default_values property returns a dictionary of default values for columns that have them defined. This is useful for understanding how new rows will be populated if values are not explicitly provided. ```python db.table("table_with_defaults").default_values ``` -------------------------------- ### List Tables with FTS Options Source: https://github.com/simonw/sqlite-utils/blob/main/docs/changelog.rst Use `sqlite-utils tables` to list tables in a database. Filter for FTS4 or FTS5 tables using the respective flags. ```bash sqlite-utils tables demo.db ``` ```bash sqlite-utils tables demo.db --fts4 ``` ```bash sqlite-utils tables demo.db --fts5 ``` -------------------------------- ### Query data and output in a table format Source: https://github.com/simonw/sqlite-utils/blob/main/README.md Execute a SQL query and display the results in a formatted table. ```bash $ sqlite-utils dogs.db "select * from dogs" --table id age name ----- ----- -------- 1 4 Cleo 2 2 Pancakes ``` -------------------------------- ### Define Custom Convert Function Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst Define a Python function named convert(value) for complex conversion logic. This example seeds the random number generator and returns a random number. ```bash sqlite-utils add-column content.db articles score float --not-null-default 1.0 sqlite-utils convert content.db articles score \ ' import random random.seed(10) def convert(value): return random.random() ' ``` -------------------------------- ### Bulk inserts with insert_all() Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Provides guidance on using the `insert_all()` method for efficient bulk insertion of records, including automatic column type detection, handling new columns with `alter=True`, and options like `ignore`, `truncate`, and `analyze`. ```APIDOC ## Bulk inserts If you have more than one record to insert, the ``insert_all()`` method is a much more efficient way of inserting them. Just like ``insert()`` it will automatically detect the columns that should be created, but it will inspect the first batch of 100 items to help decide what those column types should be. Use it like this: .. code-block:: python db.table("dogs").insert_all([{ "id": 1, "name": "Cleo", "twitter": "cleopaws", "age": 3, "is_good_dog": True, }, { "id": 2, "name": "Marnie", "twitter": "MarnieTheDog", "age": 16, "is_good_dog": True, }], pk="id", column_order=("id", "twitter", "name")) The column types used in the ``CREATE TABLE`` statement are automatically derived from the types of data in that first batch of rows. Any additional columns in subsequent batches will cause a ``sqlite3.OperationalError`` exception to be raised unless the ``alter=True`` argument is supplied, in which case the new columns will be created. The function can accept an iterator or generator of rows and will commit them according to the batch size. The default batch size is 100, but you can specify a different size using the ``batch_size`` parameter: .. code-block:: python db.table("big_table").insert_all(({ "id": 1, "name": "Name {}".format(i), } for i in range(10000)), batch_size=1000) You can skip inserting any records that have a primary key that already exists using ``ignore=True``. This works with both ``.insert({...}, ignore=True)`` and ``.insert_all([...], ignore=True)``. You can delete all the existing rows in the table before inserting the new records using ``truncate=True``. This is useful if you want to replace the data in the table. Pass ``analyze=True`` to run ``ANALYZE`` against the table after inserting the new records. ``` -------------------------------- ### List Table Triggers Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Access the .triggers property on a table object to get a list of Trigger named tuples, each containing the trigger's name, associated table, and SQL definition. ```python >>> db.table("authors").triggers [Trigger(name='authors_ai', table='authors', sql='CREATE TRIGGER [authors_ai] AFTER INSERT...'), Trigger(name='authors_ad', table='authors', sql="CREATE TRIGGER [authors_ad] AFTER DELETE..."), Trigger(name='authors_au', table='authors', sql="CREATE TRIGGER [authors_au] AFTER UPDATE")] ``` -------------------------------- ### Load SQLite Extensions using CLI Source: https://github.com/simonw/sqlite-utils/blob/main/docs/changelog.rst The `--load-extension` option for `sqlite-utils query` allows loading SQLite extensions when executing queries. ```bash sqlite-utils --load-extension=./my_extension.so my.db "SELECT my_function()" ``` -------------------------------- ### enable-wal Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli-reference.rst Enable Write-Ahead Logging (WAL) for SQLite database files. ```APIDOC ## enable-wal ### Description Enable WAL for database files. ### Usage ```bash sqlite-utils enable-wal [OPTIONS] PATH... ``` ### Parameters #### Path Parameters - **PATH** (TEXT) - Path to one or more SQLite database files. #### Options - **--load-extension TEXT** - Path to SQLite extension, with optional :entrypoint. - **-h, --help** - Show this message and exit. ``` -------------------------------- ### Get Primary Key Columns Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Retrieve the names of the primary key columns for a table using the .pks property. If the table has no explicit primary key but uses rowid, it will return ['rowid']. ```python db.table("PlantType").pks ``` -------------------------------- ### Add a Foreign Key Constraint Source: https://github.com/simonw/sqlite-utils/blob/main/docs/cli.rst Use `add-foreign-key` to add foreign key constraints to existing tables, which is not directly supported by SQLite's ALTER TABLE. This example links `books.author_id` to `authors.id`. ```bash sqlite-utils add-foreign-key books.db books author_id authors id ``` -------------------------------- ### Query data and output as CSV Source: https://github.com/simonw/sqlite-utils/blob/main/README.md Execute a SQL query and format the output as CSV. ```bash $ sqlite-utils dogs.db "select * from dogs" --csv id,age,name 1,4,Cleo 2,2,Pancakes ``` -------------------------------- ### Invoke existing commands from a custom command Source: https://github.com/simonw/sqlite-utils/blob/main/docs/plugins.rst Custom commands registered via `register_commands` can invoke other commands using `ctx.invoke`. This example shows how to use the `memory` command and return the database object. ```python @cli.command() @click.pass_context @click.argument( "paths", type=click.Path(file_okay=True, dir_okay=False, allow_dash=True), required=False, nargs=-1, ) def show_schema_for_files(ctx, paths): from sqlite_utils.cli import memory db = ctx.invoke(memory, paths=paths, return_db=True) # Now do something with that database click.echo(db.schema) ``` -------------------------------- ### Apply Migrations Programmatically Source: https://github.com/simonw/sqlite-utils/blob/main/docs/migrations.rst Apply registered migrations to a database instance using `migrations.apply(db)`. Running this multiple times is safe as already applied migrations are skipped. ```python db = Database("creatures.db") migrations.apply(db) ``` -------------------------------- ### Get Detailed Index Information Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Use .xindexes to retrieve detailed information about table indexes, leveraging SQLite's PRAGMA index_xinfo(). This returns XIndex named tuples with column details. ```python >>> db.table("ny_times_us_counties").xindexes [ XIndex( name='idx_ny_times_us_counties_date', columns=[ XIndexColumn(seqno=0, cid=0, name='date', desc=1, coll='BINARY', key=1), XIndexColumn(seqno=1, cid=-1, name=None, desc=0, coll='BINARY', key=0) ] ), XIndex( name='idx_ny_times_us_counties_fips', columns=[ XIndexColumn(seqno=0, cid=3, name='fips', desc=0, coll='BINARY', key=1), XIndexColumn(seqno=1, cid=-1, name=None, desc=0, coll='BINARY', key=0) ] ) ] ``` -------------------------------- ### Optimizing Database with VACUUM Source: https://github.com/simonw/sqlite-utils/blob/main/docs/python-api.rst Optimize the database file by running the SQLite `VACUUM` command. This can reclaim unused space and improve performance. ```python Database("my_database.db").vacuum() ```