### Start PgDog Source: https://github.com/pgdogdev/pgdog/blob/main/pgdog/tests/pgbouncer/README.md Run this script to start the PgDog service. ```bash bash run.sh ``` -------------------------------- ### Install Dependencies Source: https://github.com/pgdogdev/pgdog/blob/main/pgdog/tests/pgbouncer/README.md Run this command to install the necessary Ruby gems for the benchmark. ```bash bundle install ``` -------------------------------- ### Setup Databases and Tables Source: https://github.com/pgdogdev/pgdog/blob/main/integration/logical/README.md Run these SQL commands on all databases to create the necessary schema and table for sharding. ```sql CREATE DATABASE pgdog; CREATE USER pgdog SUPERUSER PASSWORD 'pgdog' REPLICATION; \c pgdog CREATE SCHEMA pgdog; CREATE TABLE pgdog.books ( id BIGINT PRIMARY KEY, title VARCHAR, content VARCHAR ); ``` -------------------------------- ### Start PgDog with Cargo Source: https://github.com/pgdogdev/pgdog/blob/main/README.md Run the PgDog application in release mode using Cargo. Ensure you have Rust and Cargo installed. ```bash cargo run --release ``` -------------------------------- ### Start PgBouncer Source: https://github.com/pgdogdev/pgdog/blob/main/pgdog/tests/pgbouncer/README.md Execute this command to start the PgBouncer service, providing the configuration file. ```bash pgbouncer pgbouncer.ini ``` -------------------------------- ### Create Databases and Grant Permissions Source: https://github.com/pgdogdev/pgdog/blob/main/README.md SQL commands to create the necessary databases and grant all privileges to the 'pgdog' user. This is required for local setup. ```sql CREATE DATABASE shard_0; CREATE DATABASE shard_1; GRANT ALL ON DATABASE shard_0 TO pgdog; GRANT ALL ON DATABASE shard_1 TO pgdog; ``` -------------------------------- ### Install PgDog with Helm Source: https://github.com/pgdogdev/pgdog/blob/main/README.md Install PgDog using the official Helm chart in a Kubernetes environment. Ensure your Helm repository is updated. ```bash helm repo add pgdogdev https://helm.pgdog.dev helm install pgdog pgdogdev/pgdog ``` -------------------------------- ### SQL Regression Setup Table Convention Source: https://github.com/pgdogdev/pgdog/blob/main/integration/sql/README.md Follow specific rules for creating and managing the 'sql_regression_samples' table in setup and teardown scripts to ensure consistent routing and behavior. ```sql DROP TABLE IF EXISTS sql_regression_samples; CREATE TABLE sql_regression_samples ( id BIGINT PRIMARY KEY ); ``` -------------------------------- ### Demo Data Insertion and Selection Source: https://github.com/pgdogdev/pgdog/blob/main/README.md Example SQL statements to insert data into sharded tables and select data, demonstrating basic usage within the PgDog demo environment. ```sql INSERT INTO users (id, email) VALUES (1, 'admin@acme.com'); INSERT INTO payments (id, user_id, amount) VALUES (1, 1, 100.0); ``` ```sql SELECT * FROM users WHERE id = 1; SELECT * FROM payments WHERE user_id = 1; ``` -------------------------------- ### Direct-to-Shard Query Example Source: https://github.com/pgdogdev/pgdog/blob/main/README.md Example of a direct-to-shard query where the sharding key is present in the WHERE clause. This query is sent to a single database for efficient processing. ```sql -- user_id is the sharding key. SELECT * FROM users WHERE user_id = $1; ``` -------------------------------- ### Run PgDog with Database URLs Source: https://github.com/pgdogdev/pgdog/blob/main/README.md Start PgDog using Cargo, providing database connection URLs directly. This method overrides any database configurations present in the config file. ```bash cargo run --release -- -d postgres://user:pass@localhost:5432/db1 -d postgres://user:pass@localhost:5433/db2 ``` -------------------------------- ### PgDog Load Balancer Configuration Source: https://github.com/pgdogdev/pgdog/blob/main/README.md Example TOML configuration enabling PgDog's load balancer by defining multiple hosts for a single database. This automatically distributes traffic. ```toml [[databases]] name = "prod" host = "10.0.0.1" role = "primary" [[databases]] name = "prod" host = "10.0.0.2" role = "replica" ``` -------------------------------- ### Rust Library Setup for PgDog Plugin Source: https://github.com/pgdogdev/pgdog/blob/main/docs/PLUGIN_SYSTEM.md Specifies the `Cargo.toml` configuration required to build a Rust project as a dynamic shared library (`cdylib`) suitable for use as a PgDog plugin. ```toml [lib] crate-type = ["cdylib"] ``` -------------------------------- ### SQL Regression Case File Structure Source: https://github.com/pgdogdev/pgdog/blob/main/integration/sql/README.md Organize SQL regression tests using the specified naming convention for setup, case, and teardown files. ```sql cases/ 002_select_edge_case_setup.sql # optional 002_select_edge_case_case.sql # required 002_select_edge_case_teardown.sql # optional ``` -------------------------------- ### COPY Command for Sharded Data Ingestion Source: https://github.com/pgdogdev/pgdog/blob/main/README.md Example of using the COPY command to ingest data into sharded PostgreSQL tables. PgDog automatically splits rows between shards. ```sql COPY orders (id, user_id, amount) FROM STDIN CSV HEADER; ``` -------------------------------- ### Run PgDog Integration Tests Source: https://github.com/pgdogdev/pgdog/blob/main/docs/PLUGIN_SYSTEM.md Execute the integration tests for the PgDog plugin system. This script builds test plugins, sets necessary environment variables, starts PgDog, runs RSpec tests, and stops PgDog. ```bash cd integration/plugins bash run.sh ``` -------------------------------- ### Initialize and Run Benchmark with PgDog Source: https://github.com/pgdogdev/pgdog/blob/main/examples/pgbouncer_benchmark/README.md Set the PGPORT to 6433 for PgDog, then initialize the database and run the benchmark test with 10 clients and 100,000 transactions. ```bash export PGPORT=6433 pgbench -i pgbench -c 10 -t 100000 ``` -------------------------------- ### PgDog Plugin Loading Workflow Source: https://github.com/pgdogdev/pgdog/blob/main/docs/PLUGIN_SYSTEM.md Illustrates the steps involved in loading plugins during PgDog startup, including configuration reading, library loading, symbol resolution, version checks, and initialization. ```text PgDog starts ↓ Read configuration (pgdog.toml) ↓ load_from_config() ⌼⌼⌼⌼ Extract plugin names from config.plugins[] ⌼⌼⌼⌼ Call load() ↓ For each plugin: ⌼⌼⌼⌼ Plugin::library(name) ⌼⌼⌼⌼⌼ Use libloading to dlopen() shared library ⌼ ⌼⌼⌼⌼ Plugin::load(name, &lib) ⌼⌼⌼⌼⌼ Resolve pgdog_init symbol ⌼⌼⌼⌼⌼ Resolve pgdog_fini symbol ⌼⌼⌼⌼⌼ Resolve pgdog_route symbol ⌼⌼⌼⌼⌼ Resolve pgdog_rustc_version symbol ⌼⌼⌼⌼⌼ Resolve pgdog_plugin_version symbol ⌼ ⌼⌼⌼⌼ VERSION CHECKS ⌼⌼⌼⌼⌼ Get PgDog's rustc version via comp::rustc_version() ⌼⌼⌼⌼⌼ Get plugin's rustc version via pgdog_rustc_version() ⌼⌼⌼⌼⌼ If mismatch: WARN and SKIP plugin ⌼⌼⌼⌼⌼ If match: Continue ⌼ ⌼⌼⌼⌼ plugin.init() ⌼⌼⌼⌼⌼ Call pgdog_init() if defined (synchronous) ⌼ ⌼⌼⌼⌼ Log: "loaded {name} plugin (v{version}) [timing]ms" Store plugins in static PLUGINS (OnceCell) ↓ PgDog ready to serve connections ``` -------------------------------- ### Initialize and Run Benchmark with PgBouncer Source: https://github.com/pgdogdev/pgdog/blob/main/examples/pgbouncer_benchmark/README.md Set the PGPORT to 6432 for PgBouncer, then initialize the database and run the benchmark test with 10 clients and 100,000 transactions. ```bash export PGPORT=6432 pgbench -i pgbench -c 10 -t 100000 ``` -------------------------------- ### Read-only Transaction Routing Source: https://github.com/pgdogdev/pgdog/blob/main/README.md Route read-only transactions to a replica by starting the transaction with 'BEGIN READ ONLY;'. ```sql BEGIN READ ONLY; -- This goes to a replica. SELECT * FROM users LIMIT 1; COMMIT; ``` -------------------------------- ### Client Connection Lifecycle Flowchart Source: https://github.com/pgdogdev/pgdog/blob/main/docs/CLIENT_CONNECTION.md Visualizes the steps involved in accepting a TCP connection and determining the startup message type. ```mermaid flowchart TD A[TCP accept] --> B{startup message type} B -->|Ssl| C[TLS handshake] C --> F B -->|GssEnc| D[send N — client retries] B -->|Cancel| E{verify_cancel} E -->|secret ok| E2[databases::cancel
close TCP] E -->|secret mismatch| E3[close TCP
query unaffected] B --> |Startup| F[Client::spawn] ``` -------------------------------- ### Configure Range-Based Sharding Source: https://github.com/pgdogdev/pgdog/blob/main/README.md Configure range-based sharding by specifying start and end values for each shard. The 'end' value is exclusive. ```toml start = 0 # include end = 5 # exclusive ``` -------------------------------- ### Run Go Integration Suite Source: https://github.com/pgdogdev/pgdog/blob/main/CLAUDE.md Execute the Go integration test suite. This requires Postgres to be configured beforehand. ```sh bash integration/go/run.sh # Go suite ``` -------------------------------- ### Insert Data into Sharded Schema Source: https://github.com/pgdogdev/pgdog/blob/main/README.md Example of inserting data into a table within a specific sharded schema. This query will be routed to the shard associated with 'customer_a'. ```sql INSERT INTO customer_a.orders (id, user_id, amount) VALUES ($1, $2, $3); ``` -------------------------------- ### Run Benchmark with Network Simulation Source: https://github.com/pgdogdev/pgdog/blob/main/benches/resharding/README.md Enable network simulation by setting `USE_TOXI=1`. Configure specific network conditions like bandwidth caps (`DEST_BW_KBPS`) and latency (`DEST_LATENCY_MS`). ```bash USE_TOXI=1 bash benches/resharding/copy_data/run.sh ``` ```bash DEST_BW_KBPS=5000 DEST_LATENCY_MS=20 USE_TOXI=1 bash benches/resharding/copy_data/run.sh ``` ```bash bash benches/resharding/copy_data/run.sh --save-baseline direct USE_TOXI=1 bash benches/resharding/copy_data/run.sh --baseline direct ``` -------------------------------- ### Create Database and User for PgDog Source: https://github.com/pgdogdev/pgdog/blob/main/README.md SQL commands to create the necessary database and user for PgDog to connect to. This is a prerequisite for local testing. ```sql CREATE DATABASE pgdog; CREATE USER pgdog PASSWORD 'pgdog' LOGIN; ``` -------------------------------- ### Configure Sharded Table in PgDog Source: https://github.com/pgdogdev/pgdog/blob/main/pgdog/tests/vector/README.md Add this TOML configuration to your PgDog setup to define a sharded table for embeddings, specifying the data type, column, and centroids path. ```toml [[sharded_tables]] database = "pgdog_sharded" name = "embeddings" data_type = "vector" column = "embedding" centroids_path = "centroids.json" ``` -------------------------------- ### Connect to PgDog with psql Source: https://github.com/pgdogdev/pgdog/blob/main/pgdog/README.md Connect to a running PgDog instance using psql. Assumes default credentials and port. ```bash PGPASSWORD=postgres psql -h 127.0.0.1 -p 6432 -U postgres ``` -------------------------------- ### Query Engine Pre-Dispatch Pipeline Steps Source: https://github.com/pgdogdev/pgdog/blob/main/docs/CLIENT_CONNECTION.md Details the steps involved in the query engine's pre-dispatch pipeline within QueryEngine::handle(). ```markdown | Step | Method | |---|---| | 1 | `rewrite_extended()` | | 2 | `cluster_check()` | | 3 | `parse_and_rewrite()` | | 4 | `intercept_incomplete()` | | 5 | `route_query()` | | 6 | `hooks.before_execution()` | | 7 | `backend.mirror()` | | 8 | dispatch | ``` -------------------------------- ### Connect to PgDog with psql Source: https://github.com/pgdogdev/pgdog/blob/main/README.md Connect to the PgDog instance using the `psql` command-line client. This assumes PgDog is running and accessible on the default port. ```bash psql postgres://pgdog:pgdog@127.0.0.1:6432/pgdog ``` -------------------------------- ### Basic PgDog Configuration (pgdog.toml) Source: https://github.com/pgdogdev/pgdog/blob/main/README.md A minimal pgdog.toml configuration for a single user and database. Sets the general port and default connection pool size, and defines a database connection. ```toml [general] port = 6432 default_pool_size = 10 [[databases]] name = "pgdog" host = "127.0.0.1" ``` -------------------------------- ### Resharding Macro: ok_or_abort! Source: https://github.com/pgdogdev/pgdog/blob/main/docs/RESHARDING.md This macro ensures traffic resumption after cutover starts by stopping maintenance mode and updating the cutover state on error. It's used in critical steps of the resharding process. ```rust macro_rules! ok_or_abort { ($expr:expr) => { match $expr { Ok(res) => res, Err(err) => { maintenance_mode::stop(); cutover_state(CutoverState::Abort { error: err.to_string() }); return Err(err.into()); } } }; } ``` -------------------------------- ### Documenting Configuration Structs Source: https://github.com/pgdogdev/pgdog/blob/main/pgdog-config/CONTRIBUTING.md Document public structs with a concise description of what the configuration section controls. Include any important caveats and a URL to the relevant documentation page. ```rust /// What this configuration section controls. /// /// **Note:** Any important caveat, if present. /// /// https://docs.pgdog.dev/configuration/pgdog.toml/{page}/ pub struct Foo { ... } ``` -------------------------------- ### Run Python, Ruby, Java, and SQL Integration Tests Source: https://github.com/pgdogdev/pgdog/blob/main/AGENTS.md Execute the Python, Ruby, Java, and SQL integration test suites using the main integration run script. A running Postgres instance configured by 'bash integration/setup.sh' is required. ```sh bash integration/run.sh # python + ruby + java + sql ``` -------------------------------- ### Documenting Configuration Fields Source: https://github.com/pgdogdev/pgdog/blob/main/pgdog-config/CONTRIBUTING.md Use this format for documenting public fields. Include a short description, important caveats in a 'Note:', default values if applicable, and a URL to the relevant documentation page. ```rust /// Short description of what this field controls. /// /// **Note:** Any important caveat or warning. /// /// _Default:_ `value` /// /// https://docs.pgdog.dev/configuration/pgdog.toml/{page}/#{anchor} pub field_name: Type, ``` -------------------------------- ### Run Replication Benchmark Source: https://github.com/pgdogdev/pgdog/blob/main/benches/resharding/README.md Execute the WAL replication benchmark script to measure WAL streaming throughput. ```bash bash benches/resharding/replication/run.sh ``` -------------------------------- ### Basic Benchmark Run Script Source: https://github.com/pgdogdev/pgdog/blob/main/benches/README.md Create this script to define a new benchmark suite. It sources the main bench.sh script and calls bench_run with the benchmark name and command. ```bash #!/usr/bin/env bash set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" BENCH_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" source "${BENCH_DIR}/bench.sh" bench_run "" "" "$@" ``` -------------------------------- ### Download Dataset Source: https://github.com/pgdogdev/pgdog/blob/main/pgdog/tests/vector/README.md Execute this bash script to download the necessary parquet file for the demo. ```bash bash setup.sh ``` -------------------------------- ### Basic PgDog User Configuration (users.toml) Source: https://github.com/pgdogdev/pgdog/blob/main/README.md A minimal users.toml configuration for PgDog, defining a user with their associated database and password. This file is required for database connections. ```toml [[users]] name = "alice" database = "pgdog" password = "hunter2" ``` -------------------------------- ### Run Benchmark Source: https://github.com/pgdogdev/pgdog/blob/main/pgdog/tests/pgbouncer/README.md Execute the benchmark tests using RSpec. Metrics will be sent to Datadog. ```bash bundle exec rspec benchmark_spec.rb ``` -------------------------------- ### Configure OTLP/OTEL Endpoint in pgdog.toml Source: https://github.com/pgdogdev/pgdog/blob/main/examples/datadog/README.md Set up the OTLP/OTEL ingestion endpoint in `pgdog.toml` by providing your Datadog API key and the Datadog OTLP metrics endpoint. Metrics are pushed by default every 10 seconds. ```toml # pgdog.toml [otel] datadog_api_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" endpoint = "https://otlp.us5.datadoghq.com/v1/metrics" ``` -------------------------------- ### Run Copy Data Benchmark Source: https://github.com/pgdogdev/pgdog/blob/main/benches/resharding/README.md Execute the copy data benchmark script. Use `BENCH_SCALE` to scale the dataset size. Compare performance against a saved baseline using `--baseline`. ```bash bash benches/resharding/copy_data/run.sh ``` ```bash BENCH_SCALE=10000000 bash benches/resharding/copy_data/run.sh ``` ```bash bash benches/resharding/copy_data/run.sh --save-baseline main ``` ```bash bash benches/resharding/copy_data/run.sh --baseline main ``` -------------------------------- ### Enable OpenMetrics Endpoint in pgdog.toml Source: https://github.com/pgdogdev/pgdog/blob/main/examples/datadog/README.md Configure the OpenMetrics port in the `pgdog.toml` file to enable the metrics endpoint. The endpoint will be accessible at `http://0.0.0.0:`. ```toml # pgdog.toml [general] openmetrics_port = 9090 ``` -------------------------------- ### Sharding Configuration Source: https://github.com/pgdogdev/pgdog/blob/main/README.md Configure multiple hosts for the same database with different shard numbers to enable sharding. ```toml [[databases]] name = "prod" host = "10.0.0.1" shard = 0 [[databases]] name = "prod" host = "10.0.0.2" shard = 1 ``` -------------------------------- ### Configure PgDog User Source: https://github.com/pgdogdev/pgdog/blob/main/README.md Define user credentials for PgDog in `users.toml`. This allows PgDog to authenticate with the configured databases. ```toml [[users]] database = "pgdog_sharded" name = "pgdog" password = "pgdog" ``` -------------------------------- ### Build PgDog Project Source: https://github.com/pgdogdev/pgdog/blob/main/README.md Compile the PgDog project in release mode using Cargo. This is recommended for production deployments and performance benchmarking. ```bash cargo build --release ``` -------------------------------- ### Main Client Loop with Select Macro Source: https://github.com/pgdogdev/pgdog/blob/main/docs/CLIENT_CONNECTION.md Illustrates the main event loop of a client connection, handling shutdown notifications, server messages, and client-sent messages. ```rust loop { select! { _ = shutdown.notified() => { /* check offline + can_disconnect */ } message = engine.read_backend() => server_message(message) buffer = self.buffer(state) => client_messages(buffer) } } ``` -------------------------------- ### Set Environment Variables for Database Connection Source: https://github.com/pgdogdev/pgdog/blob/main/examples/pgbouncer_benchmark/README.md Configure these environment variables before running benchmark commands to ensure pgbench connects to the correct database instance. ```bash export PGHOST=127.0.0.1 export PGPASSWORD=postgres export PGUSER=postgres export PGDATABASE=postgres ``` -------------------------------- ### Documenting Configuration Enums Source: https://github.com/pgdogdev/pgdog/blob/main/pgdog-config/CONTRIBUTING.md Document public enums with a noun phrase describing what the enum represents. Include a URL to the relevant documentation page. ```rust /// Noun phrase describing what the enum represents. /// /// https://docs.pgdog.dev/configuration/pgdog.toml/{page}/#{anchor} pub enum Bar { ... } ``` -------------------------------- ### Resharding Orchestrator Steps Source: https://github.com/pgdogdev/pgdog/blob/main/docs/RESHARDING.md Visualizes the five sequential steps involved in the resharding process managed by the Orchestrator, from schema loading to WAL draining and traffic swap. ```mermaid flowchart LR A["1. load_schema
pg_dump on source"] B["2. schema_sync_pre
pre-data to dest
reload schema cache"] C["3. data_sync
ParallelSyncManager
binary COPY"] D["4. schema_sync_post
secondary indexes"] E["5. replicate().cutover()
WAL drain
traffic swap"] A --> B --> C --> D --> E ``` -------------------------------- ### Create Publication on Primary Database Source: https://github.com/pgdogdev/pgdog/blob/main/integration/logical/README.md On the primary database (port 5432), create a publication for the 'books' table to enable logical replication. ```sql CREATE PUBLICATION books FOR TABLE pgdog.books; ``` -------------------------------- ### PgDog Query Routing Workflow Source: https://github.com/pgdogdev/pgdog/blob/main/docs/PLUGIN_SYSTEM.md Details the process of routing client queries through PgDog, including AST generation, context creation for plugins, plugin execution, and applying routing decisions. ```text Client sends query ↓ PostgreSQL Frontend Parser ⌼⌼⌼ Tokenize and parse query ⌼⌼⌼ Generate pg_query AST ↓ Query Router: QueryParser::parse() ⌼⌼⌼ Create RouterContext (shards, replicas, etc.) ⌼⌼⌼ Generate PdRouterContext for plugins ⌼⌼⌼ context.plugin_context(ast, bind_params) ⌼⌼⌼ Extract AST from pg_query ParseResult ⌼⌼⌼ Pack bind parameters into PdParameters ⌼⌼⌼ Include cluster metadata (shards, replicas, transaction state) ⌼⌼⌼ Call QueryParser::plugins() ⌼⌼⌼ For each loaded plugin: ⌼⌼⌼ plugin.route(PdRouterContext) ⌼⌼⌼ Call pgdog_route(context, &mut output) ⌼⌼⌼ Parse returned PdRoute: ⌼⌼⌼ Shard::Direct(n) → override shard to n ⌼⌼⌼ Shard::All → broadcast to all shards ⌼⌼⌼ Shard::Unknown → use default routing ⌼⌼⌼ Shard::Blocked → block query ⌼⌼⌼ ReadWrite::Read → send to replica ⌼⌼⌼ ReadWrite::Write → send to primary ⌼⌼⌼ ReadWrite::Unknown → use default logic ⌼⌼⌼ Record plugin override (if any) ⌼⌼⌼ If route provided: BREAK (first plugin wins) ⌼⌼⌼ Apply plugin overrides to routing decision ⌼⌼⌼ Continue with default routing logic ↓ Select target shard/replica ↓ Execute on database ``` -------------------------------- ### Connect to PgDog via Docker Compose Source: https://github.com/pgdogdev/pgdog/blob/main/README.md Connect to a locally running PgDog instance managed by Docker Compose using psql. Set the PGPASSWORD environment variable. ```bash docker-compose up ``` ```bash PGPASSWORD=postgres psql -h 127.0.0.1 -p 6432 -U postgres ``` -------------------------------- ### Run PgDog Unit Tests Source: https://github.com/pgdogdev/pgdog/blob/main/docs/PLUGIN_SYSTEM.md Execute the unit tests for the PgDog plugin. This command navigates to the plugin directory and runs the cargo test suite. ```bash cd pgdog-plugin cargo test ``` -------------------------------- ### Sequence Diagram of Client Connection Flow Source: https://github.com/pgdogdev/pgdog/blob/main/docs/CLIENT_CONNECTION.md Illustrates the high-level flow of a client connection, from initial TCP connect to request-response cycles. ```mermaid sequenceDiagram participant C as Client app participant L as Listener participant CL as Client participant QE as QueryEngine participant P as Pool participant S as Server C->>L: TCP connect L->>CL: Client::spawn() C->>CL: Startup message CL->>P: conn.parameters() P-->>CL: ParameterStatus CL-->>C: AuthOk + BackendKeyData + RFQ loop every request C->>CL: Q / Parse+Exec+Sync CL->>QE: client_messages() QE->>P: Pool::get() P-->>QE: Guard(Server) QE->>S: Server::send() S-->>QE: Server::read() QE->>P: Guard::drop() checkin QE-->>CL: response CL-->>C: DataRow* + RFQ end ``` -------------------------------- ### Run Logical Replication Sharding Source: https://github.com/pgdogdev/pgdog/blob/main/integration/logical/README.md Execute the sharding process using the provided cargo command. Specify source and destination databases, users, and the publication name. ```bash cargo run -- --from-database source --from-user pgdog --to-database destination --to-user pgdog --publication books ``` -------------------------------- ### Enable Shard Key Updates in pgdog.toml Source: https://github.com/pgdogdev/pgdog/blob/main/README.md Configure PgDog to allow shard key updates by setting 'enabled' to true and specifying the 'shard_key' behavior in the 'pgdog.toml' configuration file. ```toml [rewrite] enabled = true shard_key = "rewrite" ``` -------------------------------- ### Run Specific Test Suite Source: https://github.com/pgdogdev/pgdog/blob/main/AGENTS.md Execute a specific integration test suite, such as the Go suite, by directly invoking its run script. Ensure a Postgres instance is configured before running. ```sh bash integration/go/run.sh ``` -------------------------------- ### Run Clippy for Linting Source: https://github.com/pgdogdev/pgdog/blob/main/CLAUDE.md Run 'cargo clippy' to perform static analysis and identify potential code improvements and style issues where practical. ```sh cargo clippy ``` -------------------------------- ### PgDog Hash Function FFI Source: https://github.com/pgdogdev/pgdog/blob/main/docs/SHARDING.md Illustrates how PgDog's bigint(), uuid(), and varchar() hash functions delegate to PostgreSQL's native hash functions via FFI for consistent hashing. ```rust bigint(), uuid(), varchar() in [`pgdog/src/frontend/router/sharding/mod.rs`](../pgdog/src/frontend/router/sharding/mod.rs) all call into [`hashfn.c`](../pgdog/src/frontend/router/sharding/hashfn.c) via FFI ([`pgdog/src/frontend/router/sharding/ffi.rs`](../pgdog/src/frontend/router/sharding/ffi.rs)). The functions are PostgreSQL's own `hashint8extended` and `hash_bytes_extended`, so `hash(42) % N` in PgDog produces the same shard as PostgreSQL's own hash partitioning would. ``` -------------------------------- ### Run PgDog with Custom Configuration Paths Source: https://github.com/pgdogdev/pgdog/blob/main/README.md Execute PgDog using Cargo, specifying custom paths for the configuration and user files. This overrides the default 'pgdog.toml' and 'users.toml'. ```bash cargo run --release -- -c /path/to/custom/pgdog.toml -u /path/to/custom/users.toml ``` -------------------------------- ### Compare Benchmark Results Source: https://github.com/pgdogdev/pgdog/blob/main/benches/README.md Compares two hyperfine JSON files to generate a Criterion-style table with performance metrics and relative changes. Useful for analyzing benchmark outcomes. ```python python3 benches/compare.py target/pgdog_benches/before.json \ target/pgdog_benches/after.json \ --prev-label before --curr-label after ``` -------------------------------- ### Format Code with Cargo fmt Source: https://github.com/pgdogdev/pgdog/blob/main/CLAUDE.md Format the project's code according to the defined style guidelines before committing changes. ```sh cargo fmt ``` -------------------------------- ### UPDATE Flowchart for REPLICA IDENTITY FULL Source: https://github.com/pgdogdev/pgdog/blob/main/docs/REPLICATION.md This flowchart illustrates the decision process for UPDATE operations on tables with `REPLICA IDENTITY FULL`, detailing routing, shard key changes, and fast/slow path handling. ```mermaid flowchart TD GUARD{"identity == Old?"} ERR1["FullIdentityMissingOld"] ROUTE["resolve shard for old tuple
resolve shard for new tuple"] XSHARD{"shard key changed?"} FANOUT["fill 'u' cols in new from old
DELETE on old shard
INSERT on new shard"] FASTCHK{"new has
toasted cols?"} FAST["fast path: full UPDATE"] SLOW["slow path: partial UPDATE
(shape cache)"] NOOP["skip — nothing changed"] GUARD -->|yes| ROUTE GUARD -->|no| ERR1 ROUTE --> XSHARD XSHARD -->|yes| FANOUT XSHARD -->|no| FASTCHK FASTCHK -->|no| FAST FASTCHK -->|yes| SLOW SLOW -->|no non-key cols present| NOOP ``` -------------------------------- ### Enable pgvector Extension Source: https://github.com/pgdogdev/pgdog/blob/main/pgdog/tests/vector/README.md Run this SQL command to enable the pgvector extension in your PostgreSQL database. ```sql CREATE EXTENSION vector; ``` -------------------------------- ### Enable Automatic Failover Configuration Source: https://github.com/pgdogdev/pgdog/blob/main/README.md Configure PgDog for automatic failover by setting database roles to 'auto' and enabling replication monitoring. ```toml [general] lsn_check_delay = 0 [[databases]] name = "prod" host = "10.0.0.1" role = "auto" [[databases]] name = "prod" host = "10.0.0.2" role = "auto" ``` -------------------------------- ### Query Engine Flowchart Source: https://github.com/pgdogdev/pgdog/blob/main/docs/CLIENT_CONNECTION.md Visual representation of the query engine's pre-dispatch pipeline, showing the flow from ClientRequest to ReadyForQuery. ```mermaid flowchart TD A[ClientRequest] --> B[rewrite_extended] B --> C[cluster_check] C --> D[parse_and_rewrite → Route] D --> E{intercept_incomplete?} E -->|BEGIN / COMMIT / ROLLBACK| F[synthesise response
Source::Internal] E -->|no| G[route_query] G --> H[hooks.before_execution] H --> I[backend.mirror] I --> J[dispatch handler] J -->|needs Postgres| K[connect → Pool::get] K --> L[Server::send / read] L --> M[hooks.after_execution] F --> N[ReadyForQuery] M --> N ``` -------------------------------- ### PgDog Shard Value Parsing Source: https://github.com/pgdogdev/pgdog/blob/main/docs/SHARDING.md Explains the logic for shard_value(), shard_binary(), and shard_str() in handling different parameter formats and types for sharding. ```rust `shard_value()` handles text-format parameters; `shard_binary()` handles wire-format binary parameters by decoding them first. `shard_str()` is called when the type is unknown — it tries `i64` parse, then `Uuid` parse, then falls through to varchar. This is a best-effort path with a TODO noting that having the type OID would be more reliable. ``` -------------------------------- ### Enable Multi-Tuple Insert Splitting in pgdog.toml Source: https://github.com/pgdogdev/pgdog/blob/main/README.md Configure PgDog to automatically split multi-tuple INSERT statements by setting 'split_inserts' to 'rewrite' in the 'pgdog.toml' configuration file. ```toml [rewrite] enabled = true split_inserts = "rewrite" ``` -------------------------------- ### Enable Two-Phase Commit Source: https://github.com/pgdogdev/pgdog/blob/main/README.md Use these SQL statements to prepare and commit transactions for atomic cross-shard writes. PgDog handles commit failures automatically. ```sql PREPARE TRANSACTION '__pgdog_unique_id'; COMMIT PREPARED '__pgdog_unique_id'; ``` -------------------------------- ### Connection Check-in Flowchart Source: https://github.com/pgdogdev/pgdog/blob/main/docs/CLIENT_CONNECTION.md Visualizes the sequence of operations when a client connection guard is dropped, leading to cleanup and pool check-in. This includes determining the appropriate cleanup queries and handling connection health. ```mermaid flowchart TD A[Guard::drop] --> B[Cleanup::new] B --> C{dirty state?} C -->|guard.reset| D[DISCARD ALL] C -->|server.dirty| E[RESET ALL +
pg_advisory_unlock_all] C -->|schema_changed| F[DEALLOCATE ALL] C -->|clean| G[no queries] D & E & F & G --> H[drain if out of sync] H --> I[rollback if in tx] I --> J[execute_batch] J --> K[Pool::checkin] K --> L{still healthy?} L -->|yes + waiter| M[send to waiter] L -->|yes no waiter| N[idle_connections.push] L -->|no| O[drop + notify Monitor] ``` -------------------------------- ### Run Resharding Benchmark Source: https://github.com/pgdogdev/pgdog/blob/main/benches/README.md Execute the resharding benchmark. Compares against the previous run automatically. Use --save-baseline and --baseline flags for managing named baselines. ```sh bash benches/resharding/copy_data/run.sh ``` ```sh bash benches/resharding/copy_data/run.sh --save-baseline main ``` ```sh bash benches/resharding/copy_data/run.sh --baseline main ``` ```sh RUNS=1 WARMUP=0 bash benches/resharding/copy_data/run.sh ``` -------------------------------- ### Configure Default Hash Partition Sharding Source: https://github.com/pgdogdev/pgdog/blob/main/README.md Use this configuration for default hash-based sharding on a specified column. This is the default algorithm when configuring sharded tables. ```toml [[sharded_tables]] database = "prod" column = "user_id" ``` -------------------------------- ### Configure List-Based Sharding Source: https://github.com/pgdogdev/pgdog/blob/main/README.md Configure list-based sharding by defining specific values for each shard. This maps discrete values to specific shards. ```toml # Sharded table definition still required. [[sharded_tables]] database = "prod" column = "user_id" # Value-specific shard mappings. [[sharded_mapping]] database = "prod" column = "user_id" values = [1, 2, 3, 4] shard = 0 [[sharded_mapping]] database = "prod" column = "user_id" values = [5, 6, 7, 8] shard = 1 ```