### Install Dependencies

Source: https://github.com/estuary/flow/blob/master/site/README.md

Run this command to install all necessary project dependencies before starting development or building.

```bash
npm install
```

--------------------------------

### Implementing and Serving an Executor

Source: https://github.com/estuary/flow/blob/master/crates/automations/README.md

Example of defining a custom executor, registering it with the server, and starting the server to handle tasks. Ensure all necessary parameters for `serve` are provided.

```rust
// Define an executor
struct MyExecutor;

impl Executor for MyExecutor {
    const TASK_TYPE: TaskType = TaskType(42);
    type Receive = MyMessage;
    type State = MyState;
    type Outcome = Action;

    async fn poll(/* ... */) -> anyhow::Result<Action> {
        // Process inbox messages
        // Update state
        // Return action (spawn, send, sleep, etc.)
    }
}

// Register and serve
let server = Server::new().register(MyExecutor);
server.serve(permits, pool, interval, timeout, shutdown).await;
```

--------------------------------

### Start Local Development Server

Source: https://github.com/estuary/flow/blob/master/site/README.md

Starts a local development server for live previewing changes. Changes are reflected without server restarts.

```bash
npm start
```

--------------------------------

### FirstWriteWins and LastWriteWins Schema Example

Source: https://github.com/estuary/flow/blob/master/site/docs/concepts/advanced/reduction-strategies/firstwritewins-and-lastwritewins.md

Demonstrates how to configure firstWriteWins and lastWriteWins strategies within a collection schema. This setup allows tracking distinct values for the same key based on write order.

```yaml
collections:
  - name: example/reductions/fww-lww
    schema:
      type: object
      reduce: { strategy: merge }
      properties:
        key: { type: string }
        fww: { reduce: { strategy: firstWriteWins } }
        lww: { reduce: { strategy: lastWriteWins } }
      required: [key]
    key: [/key]

tests:
  "Expect we can track first- and list-written values":
    - ingest:
        collection: example/reductions/fww-lww
        documents:
          - { key: "key", fww: "one", lww: "one" }
          - { key: "key", fww: "two", lww: "two" }
    - verify:
        collection: example/reductions/fww-lww
        documents:
          - { key: "key", fww: "one", lww: "two" }
```

--------------------------------

### Start Local Supabase Instance

Source: https://github.com/estuary/flow/blob/master/CLAUDE.md

Starts a local Supabase instance for development. Use 'supabase db reset' to apply current migrations as needed.

```bash
# Start (just) local Supabase.
mise run local:supabase
# Reset with current migrations as needed
supabase db reset
```

--------------------------------

### Start Complete Local Development Stack

Source: https://github.com/estuary/flow/blob/master/CLAUDE.md

Launches the full local development stack for Estuary Flow. Refer to local/README.md for detailed setup instructions.

```bash
# Start a complete local stack (see local/README.md)
mise run local:stack
```

--------------------------------

### Install Bytewax and Kafka Client

Source: https://github.com/estuary/flow/blob/master/site/docs/reference/Connectors/materialization-connectors/Dekaf/bytewax.md

Install the necessary Python packages for Bytewax and the Kafka client. This is a prerequisite for running Bytewax dataflows that consume from Kafka.

```bash
pip install bytewax kafka-python
```

--------------------------------

### Install Mise on Host

Source: https://github.com/estuary/flow/blob/master/mise/README.md

Installs Mise on the host machine. Add the activation command to your shell's configuration file to make Mise available.

```bash
curl https://mise.run | sh
```

```bash
echo 'eval "$(~/.local/bin/mise activate bash)"' >> ~/.bashrc
source ~/.bashrc
```

--------------------------------

### Task Logs and Stats Naming Examples

Source: https://github.com/estuary/flow/blob/master/ops-catalog/README.md

Examples of naming conventions for task logs and stats collections, mirroring data-plane names.

```text
ops/tasks/public/gcp-us-central1-c1/logs
ops/tasks/public/gcp-us-central1-c1/stats

ops/tasks/public/aws-eu-west-1-c2/logs
ops/tasks/public/aws-eu-west-1-c2/stats

ops/tasks/private/acmeCo/aws-eu-west-2-c1/logs
ops/tasks/private/acmeCo/aws-eu-west-2-c1/stats
```

--------------------------------

### Elasticsearch Field Configuration Example

Source: https://github.com/estuary/flow/blob/master/site/docs/reference/Connectors/materialization-connectors/Elasticsearch.md

Example JSON configuration for the Elasticsearch connector, demonstrating field routing and mapping options for index creation.

```json
{
  "bindings": [
    {
      "resource": {
        "index": "my-elasticsearch-index"
      },
      "source": "PREFIX/source_collection",
      "fields": {
        "require": {
          "myKey": {
            "routing": true
          },
          "stringOrIntegerField": {
            "mapping": {
              "type": "keyword"
            }
          },
          "unixTimestamp": {
            "mapping": {
              "type": "date",
              "format": "epoch_seconds"
            }
          }
        },
        "recommended": true
      }
    }
  ]
}
```

--------------------------------

### Python Async Task Pipelining Example

Source: https://github.com/estuary/flow/blob/master/plans/derive-python.md

This Python code demonstrates how to manage asynchronous tasks within a transform function and `flush` method for pipelining. It uses `asyncio.create_task` to start tasks and `asyncio.gather` to await their results.

```python
async def transform(self, read):
      self.pending_tasks.append(asyncio.create_task(fetch_data(read.doc)))
      return []

  async def flush(self):
      results = await asyncio.gather(*self.pending_tasks)
      self.pending_tasks.clear()
      return [Document(...) for r in results]
```

--------------------------------

### Resource Configuration Example

Source: https://github.com/estuary/flow/blob/master/site/docs/guides/backfilling-data.md

This YAML snippet shows how to configure a resource with a specific backfill mode. Ensure you understand the implications of each mode before changing the default setting.

```yaml
bindings:
  - resource:
      namespace: public
      stream: tableName
      mode: "Only Changes"
    target: your/collection/name
```

--------------------------------

### Freshdesk Connector Configuration Sample

Source: https://github.com/estuary/flow/blob/master/site/docs/reference/Connectors/capture-connectors/freshdesk.md

This YAML configuration sample shows how to set up the Freshdesk source connector, including API key, domain, and resource bindings for incremental sync.

```yaml
captures:
  ${PREFIX}/${CAPTURE_NAME}:
    endpoint:
      connector:
        image: ghcr.io/estuary/source-freshdesk:v1
        config:
            api_key: xxxxxxxxxxxxxxxx
            domain: acmesupport.freshdesk.com
    bindings:
      - resource:
          stream: agents
          syncMode: incremental
        target: ${PREFIX}/agents
      {...}
```

--------------------------------

### Install flowctl on macOS with Homebrew

Source: https://github.com/estuary/flow/blob/master/site/docs/concepts/flowctl.md

Installs flowctl on macOS using the Homebrew package manager.

```bash
brew tap estuary/flowctl
brew install flowctl
```

--------------------------------

### Data-Plane Naming Examples

Source: https://github.com/estuary/flow/blob/master/ops-catalog/README.md

Examples of public and private data-plane names following the provider-region-cluster convention.

```text
ops/dp/public/gcp-us-central1-c1
ops/dp/public/aws-eu-west-1-c2
ops/dp/private/acmeCo/aws-eu-west-2-c1
```

--------------------------------

### Sample Configuration

Source: https://github.com/estuary/flow/blob/master/site/docs/reference/Connectors/capture-connectors/redshift-batch.md

A sample YAML configuration demonstrating how to set up the Redshift Batch connector and define a capture.

```APIDOC
### Sample Configuration

```yaml
captures:
  ${PREFIX}/${CAPTURE_NAME}:
    endpoint:
      connector:
        image: ghcr.io/estuary/source-redshift-batch:v1
        config:
          address: host:port
          user: estuary_user
          password: secret
          database: db_name
    bindings:
      - resource:
          name: main_orders
          schema: main
          table: orders
        target: ${PREFIX}/${COLLECTION_NAME}
```
```

--------------------------------

### Install flowctl on macOS

Source: https://github.com/estuary/flow/blob/master/site/docs/concepts/flowctl.md

Installs the flowctl binary for macOS systems, making it executable and adding it to the system's PATH.

```bash
sudo curl -o /usr/local/bin/flowctl -L 'https://github.com/estuary/flow/releases/latest/download/flowctl-multiarch-macos' && sudo chmod +x /usr/local/bin/flowctl
```

--------------------------------

### Pinterest Source Connector Configuration Sample

Source: https://github.com/estuary/flow/blob/master/site/docs/reference/Connectors/capture-connectors/pinterest.md

This YAML configuration sample demonstrates how to set up the Pinterest source connector, including specifying the image, start date, and authentication method with an access token.

```yaml
captures:
  ${PREFIX}/${CAPTURE_NAME}:
    endpoint:
      connector:
        image: ghcr.io/estuary/source-pinterest:v1
        config:
          start_date: 2026-01-01
          credentials:
            auth_method: access_token
            access_token: <secret>
    bindings:
      - resource:
          name: boards
          syncMode: full_refresh
        target: ${PREFIX}/${COLLECTION_NAME}
```

--------------------------------

### Install flowctl on Linux

Source: https://github.com/estuary/flow/blob/master/site/docs/concepts/flowctl.md

Installs the flowctl binary for Linux systems, making it executable and adding it to the system's PATH.

```bash
sudo curl -o /usr/local/bin/flowctl -L 'https://github.com/estuary/flow/releases/latest/download/flowctl-x86_64-linux' && sudo chmod +x /usr/local/bin/flowctl
```

--------------------------------

### Klaviyo Connector Configuration Sample

Source: https://github.com/estuary/flow/blob/master/site/docs/reference/Connectors/capture-connectors/klaviyo.md

This YAML configuration sample shows how to set up the Klaviyo source connector, including the image, API key, start date, and bindings for different resources like lists.

```yaml
captures:
  ${PREFIX}/${CAPTURE_NAME}:
    endpoint:
      connector:
        image: ghcr.io/estuary/source-klaviyo:v1
        config:
          api_key: <secret>
          start_date: 2017-01-25T00:00:00Z
    bindings:
      - resource:
          stream: lists
          syncMode: full_refresh
        target: ${PREFIX}/lists
      {...}
```

--------------------------------

### Sample Notion Connector Configuration

Source: https://github.com/estuary/flow/blob/master/site/docs/reference/Connectors/capture-connectors/notion.md

This YAML configuration sample demonstrates how to set up the Notion source connector for manual authentication using an access token and specifies bindings for capturing 'blocks' data.

```yaml
captures:
  ${PREFIX}/${CAPTURE_NAME}:
    endpoint:
      connector:
        image: ghcr.io/estuary/source-notion:v2
        config:
          credentials:
            auth_type: token
            token: {secret}
          start_date: 2021-01-25T00:00:00Z
    bindings:
      - resource:
          stream: blocks
          syncMode: incremental
        target: ${PREFIX}/blocks
      {...}
```

--------------------------------

### Original Data Row Example

Source: https://github.com/estuary/flow/blob/master/site/docs/transformations/flatten-array.md

An example of an original data row structure before derivation. This helps in understanding the transformation process.

```json
{
  "_meta": {
    ...
  },
  "id": "1",
  "name": "test1",
  "tags": "[{"name":"PFJUjs6Wec","value":"HB668r7MfN"},{"name":"aIWpjtpNnj","value":"elQ9948Wpf"}]"
}
```

--------------------------------

### L1 Rollups Naming Examples

Source: https://github.com/estuary/flow/blob/master/ops-catalog/README.md

Examples of L1 rollup names, prefixed with 'ops/rollups/L1/' and following data-plane naming conventions.

```text
ops/rollups/L1/public/gcp-us-central1-c1/catalog-stats
ops/rollups/L1/public/gcp-us-central1-c1/inferred-schemas
ops/rollups/L1/private/acmeCo/aws-eu-west-2-c1/failure-notifications
```

--------------------------------

### Install Estuary Plugins for Claude Code

Source: https://github.com/estuary/flow/blob/master/site/docs/guides/agent-skills.md

Install specific Estuary agent skills plugins after adding the marketplace to Claude Code.

```bash
/plugin install estuary-captures@estuary
/plugin install estuary-materializations@estuary
/plugin install estuary-operations@estuary
```

--------------------------------

### WooCommerce Connector Configuration Sample

Source: https://github.com/estuary/flow/blob/master/site/docs/reference/Connectors/capture-connectors/woocommerce.md

This YAML configuration sample demonstrates how to set up the WooCommerce connector, including API credentials, shop name, start date, and stream bindings for incremental data synchronization.

```yaml
captures:
  ${PREFIX}/${CAPTURE_NAME}:
    endpoint:
      connector:
        image: ghcr.io/estuary/source-woocommerce:v1
        config:
          api_key: api-key
          api_secret: secret
          shop: acmeCo
          start_date: 2025-01-01
    bindings:
      - resource:
          stream: orders
          syncMode: incremental
        target: ${PREFIX}/${COLLECTION_NAME}
```

--------------------------------

### Nested Fields Example in readSchema

Source: https://github.com/estuary/flow/blob/master/site/docs/guides/customize-dataflows/customize-materialization-fields.md

Example demonstrating how to specify deeply nested fields, such as Jira custom fields, within the readSchema.

```yaml
allOf:
  - $ref: flow://relaxed-write-schema
  - $ref: flow://inferred-schema
  - type: object
    properties:
      fields:
        type: object
        properties:
          project:
            type: object
            properties:
              id:
                type: string
```

--------------------------------

### Setting Up Environment and Reading Journals

Source: https://github.com/estuary/flow/blob/master/plans/runtime-v2/preview-harness.md

Loads environment variables for the reactor and reads journal entries using `flowctl-go`. Ensure the environment file exists and the selector matches your journal.

```bash
set -a
. ~/flow-local/env/reactor-local-cluster-8099.env
set +a
```

```bash
~/go/bin/flowctl-go journals list \
  --selector 'name:prefix=acmeCo/events/' \
  --format table
```

```bash
timeout 15 ~/go/bin/flowctl-go journals read \
  --selector 'name=acmeCo/events/14d588f6a580018e/pivot=00' \
  --output - \
  --file-root ~/flow-local/fragments
```

--------------------------------

### Sample Configuration for Google Cloud Pub/Sub Materialization

Source: https://github.com/estuary/flow/blob/master/site/docs/reference/Connectors/materialization-connectors/google-pubsub.md

This sample demonstrates manual authentication using a service account JSON key and configures a materialization to publish data to a Google Cloud Pub/Sub topic. Ensure the `credentials_json` field contains your actual service account key.

```yaml
materializations:
  ${PREFIX}/${mat_name}:
    endpoint:
      connector:
        image: ghcr.io/estuary/materialize-google-pubsub:v1
        config:
          credentials:
            auth_type: Service
            credentials_json: {secret}
          project_id: my_google_cloud_project
    bindings:
  	  - resource:
          create_default_subscription: true
      	  topic: my_new_topic
        source: ${PREFIX}/${source_collection}
```

--------------------------------

### Sample Google Search Console Connector Configuration

Source: https://github.com/estuary/flow/blob/master/site/docs/reference/Connectors/capture-connectors/google-search-console.md

This YAML configuration demonstrates how to set up the Google Search Console connector for manual authentication. It includes endpoint configuration with service account details and site URLs, as well as binding configurations for data streams.

```yaml
captures:
  ${PREFIX}/${CAPTURE_NAME}:
    endpoint:
      connector:
        image: ghcr.io/estuary/source-google-search-console:v2
          config:
            credentials:
              auth_type: Service
              service_account_info: <secret>
              email: admin@yourdomain.com
            site_urls: https://yourdomain.com
            start_date: 2022-03-01

      bindings:
        - resource:
            stream: sites
            syncMode: full_refresh
          target: ${PREFIX}/sites
       {}
```

--------------------------------

### Example Transaction ID Output

Source: https://github.com/estuary/flow/blob/master/site/docs/guides/troubleshooting/postgres-replication-slot-recovery.md

This is an example of the output from the `flowctl logs` command, showing the timestamp and the last processed transaction ID (xid).

```json
{"ts":"2026-02-27T05:13:11Z","message":"current transaction ID","fields":{"xid":583198898}}
```

--------------------------------

### Zendesk Support Connector Configuration Sample

Source: https://github.com/estuary/flow/blob/master/site/docs/reference/Connectors/capture-connectors/zendesk-support-native.md

This YAML configuration sample shows how to set up the Zendesk Support source connector using API token authentication. It specifies the connector image, credentials, start date, subdomain, and data resource bindings.

```yaml
captures:
  ${PREFIX}/${CAPTURE_NAME}:
    endpoint:
      connector:
        image: ghcr.io/estuary/source-zendesk-support-native:v1
        config:
            credentials:
              credentials: Email & API Token
              username: user@domain.com
              password: <secret>
            start_date: "2025-01-30T00:00:00Z"
            subdomain: my_subdomain
    bindings:
      - resource:
          name: tickets
        target: ${PREFIX}/tickets
```

--------------------------------

### Collection Specification Example

Source: https://github.com/estuary/flow/blob/master/site/docs/concepts/collections.md

Defines a collection named 'acmeCo/products/anvils' with a schema reference and a composite key. This is a foundational example for structuring collections in Estuary.

```yaml
collections:
  acmeCo/products/anvils:
    schema: anvils.schema.yaml
    key: [/product/id]
    projections:
    derive: ~
```

--------------------------------

### Klaviyo Source Connector Configuration Sample

Source: https://github.com/estuary/flow/blob/master/site/docs/reference/Connectors/capture-connectors/klaviyo-native.md

This YAML configuration sample shows how to set up the Klaviyo source connector, including endpoint details, credentials, and data resource bindings.

```yaml
captures:
  ${PREFIX}/${CAPTURE_NAME}:
    endpoint:
      connector:
        image: ghcr.io/estuary/source-klaviyo-native:v1
        config:
            advanced:
              window_size: PT2H
            credentials:
              credentials: API Key
              access_token: <secret>
            start_date: "2025-08-14T00:00:00Z"
    bindings:
      - resource:
          name: events
        target: ${PREFIX}/events
```

--------------------------------

### Braze Connector Configuration Sample

Source: https://github.com/estuary/flow/blob/master/site/docs/reference/Connectors/capture-connectors/braze.md

This YAML configuration sample demonstrates how to set up the Braze source connector, including endpoint details and binding configurations for data streams.

```yaml
captures:
  ${PREFIX}/${CAPTURE_NAME}:
    endpoint:
      connector:
        image: ghcr.io/estuary/source-braze:v1
        config:
          api_key: <key>
          start_date: 2017-01-25T00:00:00Z
          url: <url>
    bindings:
      - resource:
          stream: events
          syncMode: full_refresh
        target: ${PREFIX}/events
      {...}
```

--------------------------------

### Run CI Platform Build

Source: https://github.com/estuary/flow/blob/master/mise/README.md

Executes the full build suite, mirroring CI environment.

```bash
mise run ci:platform-build
```

--------------------------------

### OpenMetrics API Response Example

Source: https://github.com/estuary/flow/blob/master/site/docs/features/openmetrics-api.md

Example of the output format from the OpenMetrics API, showing metric names, types, and associated labels like task.

```text
# HELP logged_warnings_total Total log lines at level WARN, by task
# TYPE logged_warnings_total counter
logged_warnings_total{task="acmeCo/prefix/source-oracle"} 12
logged_warnings_total{task="acmeCo/prefix/materialize-mongodb"} 5
# HELP logged_errors_total Total log lines at level ERROR, by task
# TYPE logged_errors_total counter
{...}
# HELP materialized_out_docs_total Total number of post-reduce documents stored to the target, by task and source collection
```

--------------------------------

### Marketo Connector Configuration Sample

Source: https://github.com/estuary/flow/blob/master/site/docs/reference/Connectors/capture-connectors/marketo.md

This YAML configuration sample shows how to set up the Marketo source connector, including endpoint details and binding configurations for data streams.

```yaml
captures:
  ${PREFIX}/${CAPTURE_NAME}:
    endpoint:
      connector:
        image: ghcr.io/estuary/source-marketo:v1
        config:
          client_id: <secret>
          client_secret: <secret>
          start_date: 2017-01-25T00:00:00Z
          domain_url: <your domain URL>
    bindings:
      - resource:
          stream: leads
          syncMode: full_refresh
        target: ${PREFIX}/leads
      {...}
```

--------------------------------

### Example of Partitioned Data in Cloud Storage

Source: https://github.com/estuary/flow/blob/master/site/docs/concepts/advanced/projections.md

Illustrates how documents with specific partitioned field values are organized into files within cloud storage, showing the path structure based on logical and physical partitions.

```text
s3://bucket/example/sessions/country=CA/device=iPhone/network=LTE/pivot=00/utc_date=2020-11-04/utc_hour=16/<name>.gz
```

--------------------------------

### Install Python Dependencies

Source: https://github.com/estuary/flow/blob/master/tests/soak/README.md

Installs Python dependencies for the Flow project into the in-project virtual environment using Poetry. Ensure you are in the project's root directory.

```bash
cd ~/estuary/flow && poetry install --no-root
```

--------------------------------

### Install Estuary Skills for OpenAI Codex using Skills CLI

Source: https://github.com/estuary/flow/blob/master/site/docs/guides/agent-skills.md

Install all Estuary agent skills for OpenAI Codex using the cross-tool Skills CLI.

```bash
npx skills add estuary/agent-skills
```

--------------------------------

### Example Authorization Header

Source: https://github.com/estuary/flow/blob/master/site/docs/reference/Connectors/capture-connectors/datadog-ingest.md

When authentication is enabled, incoming requests must include an 'Authorization' header with a 'Bearer' token. This example shows the required header format.

```http
Authorization: Bearer mySecretToken
```

--------------------------------

### HTTP File Source Connector Configuration Sample

Source: https://github.com/estuary/flow/blob/master/site/docs/reference/Connectors/capture-connectors/http-file.md

This YAML configuration sample demonstrates how to set up the HTTP File Source connector, including the endpoint URL, parser configuration for compression and format (CSV with specific delimiters and headers), and binding to an Estuary stream.

```yaml
captures:
  ${PREFIX}/${CAPTURE_NAME}:
    endpoint:
      connector:
        image: ghcr.io/estuary/source-http-file:v1
        config:
          url: https://my-site.com/my_hosted_dataset.json.zip
          parser:
            compression: zip
            format:
              type: csv
              config:
                delimiter: ","
                encoding: UTF-8
                errorThreshold: 5
                headers: [ID, username, first_name, last_name]
                lineEnding: "\r"
                quote: "\""
    bindings:
      - resource:
          stream: my_hosted_dataset.json.zip
        target: ${PREFIX}/${COLLECTION_NAME}

```

--------------------------------

### Start Docker Compose Services

Source: https://github.com/estuary/flow/blob/master/site/docs/getting-started/tutorials/postgresql_cdc_to_snowflake.md

Starts the PostgreSQL CDC and data generator services using Docker Compose. Use the -d flag to run in the background.

```sh
docker compose up
```

--------------------------------

### Brevo Connector Configuration Sample

Source: https://github.com/estuary/flow/blob/master/site/docs/reference/Connectors/capture-connectors/brevo.md

This YAML sample demonstrates the configuration for the Brevo source connector, including endpoint details and bindings for data resources.

```yaml
captures:
  ${PREFIX}/${CAPTURE_NAME}:
    endpoint:
      connector:
        image: ghcr.io/estuary/source-brevo:v1
        config:
          api-key: {secret}
          start_date: 2025-01-01T00:00:00.000Z
    bindings:
      - resource:
          stream: contacts
          syncMode: full_refresh
        target: ${PREFIX}/contacts
      {...}
```