### Start Docker Compose and Initialize MongoDB Source

Source: https://github.com/estuary/connectors/blob/main/source-mongodb/README.md

Use these commands to start the Docker Compose environment for the MongoDB source connector and initialize it. Ensure Docker is installed and running.

```bash
docker compose -f source-mongodb/docker-compose.yaml up
```

```bash
bash source-mongodb/docker-compose-init.sh
```

--------------------------------

### Install Meltano and Plugins

Source: https://github.com/estuary/connectors/blob/main/source-shopify/source_shopify/README.md

Install Meltano if not already present, and then install any additional required plugins for the project.

```bash
meltano install
```

--------------------------------

### Install Meltano and Initialize Project

Source: https://github.com/estuary/connectors/blob/main/source-criteo/source_criteo/README.md

Install Meltano using pipx and initialize it within the tap-criteo directory.

```bash
pipx install meltano
```

```bash
cd tap-criteo
meltano install
```

--------------------------------

### Initialize Development Environment

Source: https://github.com/estuary/connectors/blob/main/source-criteo/source_criteo/README.md

Install poetry and then install project dependencies using poetry.

```bash
pipx install poetry
```

```bash
poetry install
```

--------------------------------

### Install Meltano using pipx

Source: https://github.com/estuary/connectors/blob/main/source-shopify/source_shopify/README.md

Use pipx to install the Meltano CLI. Ensure pipx is installed and configured.

```bash
pipx install meltano
```

--------------------------------

### Start Local Kafka Infrastructure

Source: https://github.com/estuary/connectors/blob/main/source-kafka/README.md

Launch a Dockerized Kafka and Zookeeper instance for local development. This command is part of the developer setup.

```bash
make start_local_infra
```

--------------------------------

### Install tap-criteo

Source: https://github.com/estuary/connectors/blob/main/source-criteo/source_criteo/README.md

Install the tap-criteo package using pipx.

```bash
pipx install git+https://github.com/edgarrmondragon/tap-criteo.git
```

--------------------------------

### Kafka Connection Configuration Example

Source: https://github.com/estuary/connectors/blob/main/source-kafka/README.md

Example `config.json` for connecting to a Kafka cluster. Ensure `bootstrap_servers` points to your Kafka instance.

```json
{
  "bootstrap_servers": "localhost:9092"
}
```

--------------------------------

### Setting Up a Connector Project

Source: https://github.com/estuary/connectors/blob/main/estuary-cdk/CLAUDE.md

Commands to create a virtual environment, activate it, and install project dependencies using Poetry.

```bash
# Create virtual environment and install dependencies
cd source-example
python -m venv .venv && source .venv/bin/activate
poetry install

# Build Docker image (from repo root)
./build-local.sh source-example
```

--------------------------------

### Example flow.yaml for Discovery

Source: https://github.com/estuary/connectors/blob/main/source-sqlserver-batch/README.md

An example `flow.yaml` configuration file demonstrating how to set up a MySQL batch source connector for discovery. It specifies the connector image and connection details.

```yaml
---
captures:
  acmeCo/source-mysql-batch:
    endpoint:
      connector:
        image: "ghcr.io/estuary/source-mysql-batch:local"
        config:
          address: "127.0.0.1:1433"
          database: test
          password: we2rie1E
          user: flow_capture
    bindings: []

```

--------------------------------

### Install Shopify Tap

Source: https://github.com/estuary/connectors/blob/main/source-shopify/source_shopify/README.md

Install the Shopify tap using pip from GitHub. Replace 'vx.x.x' with the desired version.

```bash
pip install git+https://github.com/Matatika/tap-shopify@vx.x.x
```

--------------------------------

### Kafka Catalog Configuration Example

Source: https://github.com/estuary/connectors/blob/main/source-kafka/README.md

Example `catalog.json` for reading data from a Kafka topic. Includes schema definition and optional tailing/range settings.

```json
{
  "streams": [
    {
      "name": "topic-name",
      "json_schema": {
        "type": "object"
      }
    }
  ],
  "estuary.dev/tail": true,
  "estuary.dev/range": {
    "begin": "00000000",
    "end": "ffffffff"
  }
}
```

--------------------------------

### Install pyyaml

Source: https://github.com/estuary/connectors/blob/main/tests/benchmark/materialize/README.md

Install the pyyaml package, which is required for running the benchmarks.

```bash
pip3 install pyyaml
```

--------------------------------

### Example Flow Catalog Configuration

Source: https://github.com/estuary/connectors/blob/main/source-oracle-batch/README.md

An example catalog configuration for the Flow Batch SQL Source Connector. This defines a capture source and its associated collection.

```yaml
captures:
  acmeCo/something/source-oracle-batch:
    endpoint:
      connector:
        image: "ghcr.io/estuary/source-oracle-batch:local"
        config:
          address: "localhost:5432"
          database: "oracle"
          password: "secret1234"
          user: "oracle"
    bindings:
      - resource:
          name: "foobar"
          template: >
            {{if .IsFirstQuery -}}
              SELECT ORA_ROWSCN AS TXID, {{quoteTableName .Owner .TableName}}.* FROM {{quoteTableName .Owner .TableName}} ORDER BY ORA_ROWSCN
            {{- else -}}
              SELECT ORA_ROWSCN AS TXID, {{quoteTableName .Owner .TableName}}.* FROM {{quoteTableName .Owner .TableName}}
                WHERE ORA_ROWSCN > :1
                ORDER BY ORA_ROWSCN
            {{- end}}
          cursor: ["TXID"]
          poll: 5s
        target: "acmeCo/something/foobar"

collections:
  acmeCo/something/foobar:
    key: [/_meta/polled, /_meta/index]
    schema:
      properties:
        _meta:
          type: object
          properties:
            polled: {type: string, format: date-time}
            index: {type: integer}
          required: [polled, index]
      required: [_meta]
      type: object

```

--------------------------------

### PostgreSQL User and Publication Setup for Flow Connector

Source: https://github.com/estuary/connectors/blob/main/source-postgres/README.md

SQL commands to create a user with replication privileges, grant necessary permissions, and create a publication for capturing all tables. This setup is for a more restricted environment than using superuser credentials.

```sql
CREATE USER flow_capture WITH PASSWORD 'secret' REPLICATION;

-- The `pg_read_all_data` role is new in PostgreSQL v14. For older versions:
--
--   GRANT USAGE ON SCHEMA public, <others> TO flow_capture;
--   ALTER DEFAULT PRIVILEGES IN SCHEMA public, <others> GRANT SELECT ON TABLES to flow_capture;
--   GRANT SELECT ON ALL TABLES IN SCHEMA public, <others> TO flow_capture;
--
-- can be substituted, but all schemas which will be captured from must be listed.
GRANT pg_read_all_data TO flow_capture;

CREATE PUBLICATION flow_publication FOR ALL TABLES;
ALTER PUBLICATION flow_publication SET (publish_via_partition_root = true);

-- Set WAL level to logical. Note that changing this requires a database
-- restart to take effect.
ALTER SYSTEM SET wal_level = logical;
```

--------------------------------

### Flow Configuration Example

Source: https://github.com/estuary/connectors/blob/main/source-ga4-bigquery/README.md

Example configuration for the Flow YAML file, specifying the GA4 BigQuery source connector endpoint and credentials.

```yaml
captures:
  acmeCo/source-ga4-bigquery:
    endpoint:
      connector:
        image: "ghcr.io/estuary/source-ga4-bigquery:local"
        config:
          project_id: my-gcp-project
          credentials:
            auth_type: CredentialsJSON
            credentials_json: "<REDACTED>"
          dataset: analytics_1234
    bindings: []
```

--------------------------------

### Get tap-criteo About Information

Source: https://github.com/estuary/connectors/blob/main/source-criteo/source_criteo/README.md

Run tap-criteo with the --about flag to get a list of supported settings and capabilities.

```bash
tap-criteo --about
```

--------------------------------

### Encrypted Connector Configuration Example

Source: https://github.com/estuary/connectors/blob/main/python/README.md

An example of a connector configuration file after being encrypted with SOPS. Note the `ENC[...]` fields.

```yaml
client_id: exctatic_emu@service-accounts.estuary.dev
client_secret_sops: ENC[AES256_GCM,data:c3BEsuHJLjIt7+G1hwb6x29BU7CK,iv:6LfUthR8c5DFTmucFC5NnMiOGal7v+PYixadovIm2gw=,tag:uHtTuPXLxEi4HLunVeLjkQ==,type:str]
sops:
    kms: []
    gcp_kms:
        - resource_id: projects/helpful-kingdom-273219/locations/us-central1/keyRings/dev/cryptoKeys/CI-estuary-flow
          created_at: "2023-11-06T22:16:37Z"
          enc: CiQAW8BC2JnhfMjWVLeRYPPQgnzBVM2MtLMlh/84pcfCRbQExBcSSQBgR/fKuXztEtnXLcNceSt9XGDi0A/9nqYQrFFqTD5d0R2HEATmH4Fyqg/Gn5/sYAdDegI0g3hHYZd91rJir0TaljFQ2YRAnYw=
    azure_kv: []
    hc_vault: []
    age: []
    lastmodified: "2023-11-06T22:16:37Z"
    mac: ENC[AES256_GCM,data:LzU+fTji6MHPFjXMNqnQAizwL3jBCvt9zltFz291u81ocIMSkdFiff+KRoTHz8kYvdoBVfJt8CesdCOqGiTgLKmea7teKiJuK5bBEOuzEY4lfC1fRYVkX+Dw6t1Mx5CvlLlS3ioisVtPG53eAGMcZDhZ7iJt7nm7qvo3Tkq7pSU=,iv:1OI2BJIyxo8DNnJmGn4lqU8TlFcdG9R9GhognGyyNY8=,tag:fPjklOLlS7OzDFszFK75hg==,type:str]
    pgp: []
    encrypted_suffix: _sops
    version: 3.7.3
```

--------------------------------

### Build Connector Image

Source: https://github.com/estuary/connectors/blob/main/source-sqlserver-ct/README.md

Builds a Docker image for the source-sqlserver-ct connector. Ensure Docker is installed and you are in the correct directory.

```bash
docker build --network=flow-test -t source-sqlserver-ct:local -f source-sqlserver-ct/Dockerfile .
```

--------------------------------

### Set Up Local MySQL Test Database

Source: https://github.com/estuary/connectors/blob/main/source-mysql/README.md

Run this bash command to start a local MySQL instance using Docker Compose for connector development and testing.

```bash
$ docker-compose --file source-mysql/docker-compose.yaml up --detach
```

--------------------------------

### PostgreSQL User and Schema Setup for Testing

Source: https://github.com/estuary/connectors/blob/main/source-postgres/README.md

SQL commands to create a user with replication privileges and set up a schema for testing against a Supabase PostgreSQL instance. Ensure the user and schema names match your environment.

```sql
CREATE USER flow_capture WITH PASSWORD 'secret1234' REPLICATION;
```

```sql
CREATE SCHEMA IF NOT EXISTS test;
GRANT USAGE ON SCHEMA test TO flow_capture;
ALTER DEFAULT PRIVILEGES IN SCHEMA test GRANT SELECT ON TABLES to flow_capture;
GRANT SELECT ON ALL TABLES IN SCHEMA test TO flow_capture;
```

```sql
CREATE PUBLICATION flow_publication FOR ALL TABLES;
ALTER PUBLICATION flow_publication SET (publish_via_partition_root = true);
```

--------------------------------

### Example Raw Access Token Response

Source: https://github.com/estuary/connectors/blob/main/python/README.md

An example of a raw access token response received after a successful token exchange.

```json
{
    "access": "ey...==",
    "refresh": "CRYPTOGRAPHIC_RANDOM_REFRESH",
    "expires_in": 3600
}
```

--------------------------------

### Verify Connector with `spec` Command

Source: https://github.com/estuary/connectors/blob/main/source-kafka/README.md

Use the `spec` command to verify the connector is working correctly. This is useful for initial setup and debugging.

```bash
docker run --rm -it ghcr.io/estuary/source-kafka:TAG spec
```

--------------------------------

### Run PostgreSQL Test Database

Source: https://github.com/estuary/connectors/blob/main/source-postgres-batch/README.md

Launches the test database using Docker Compose. Ensure Docker is installed and running.

```bash
docker compose -f ./source-postgres-batch/docker-compose.yaml up -d
```

--------------------------------

### Request.Spec Example

Source: https://github.com/estuary/connectors/blob/main/docs/materialize/README.md

Example structure for Request.Spec, which provides connector information to the Flow runtime. It includes protocol version, documentation URL, and JSON schemas for configuration.

```json
{
  "protocol": 3032023, 
  "documentation_url": "go.estuary.dev/materialize-postgres",
  "configSchema": {
    "type": "object",
    "properties": {
      "address": {
        "type": "string"
      },
      "credentials": {
        "type": "object",
        ...
      }
      ...
    }
  },
  "resourceConfigSchema": {
    "type": "object",
    "properties": {
      "table": {
        "type": "string"
      }
    }
  },
  "oauth2": {
    "provider": "google",
    ...
  }
}
```

--------------------------------

### Example flow.yaml for Discovery

Source: https://github.com/estuary/connectors/blob/main/source-mysql-batch/README.md

A basic `flow.yaml` configuration file used for discovering the MySQL batch source connector. It specifies the connector image and basic connection details.

```yaml
---
captures:
  acmeCo/source-mysql-batch:
    endpoint:
      connector:
        image: "ghcr.io/estuary/source-mysql-batch:local"
        config:
          address: "127.0.0.1:3306"
          password: secret1234
          user: root
    bindings: []

```

--------------------------------

### Example flow.yaml for BigQuery Source Discovery

Source: https://github.com/estuary/connectors/blob/main/source-bigquery-batch/README.md

Configuration file for the BigQuery source connector, specifying the Docker image and project details for table discovery. The credentials_json should be redacted.

```yaml
captures:
  acmeCo/source-bigquery-batch:
    endpoint:
      connector:
        image: "ghcr.io/estuary/source-bigquery-batch:local"
        config:
          project_id: pivotal-base-360421
          credentials_json: "<REDACTED>"
          advanced:
            poll: 5s
    bindings: []
```

--------------------------------

### Example Document Update

Source: https://github.com/estuary/connectors/blob/main/docs/materialize/README.md

Illustrates a document update scenario where a newer version is processed before an older version due to concurrent connector instances.

```json
{"id": 1, "counter": 1}
```

```json
{"id": 1, "counter": 2}
```

```json
{"id": 1, "counter": 2}
```

--------------------------------

### Build Materialization Connector Image

Source: https://github.com/estuary/connectors/blob/main/README.md

Build the Docker image for the materialization connector you intend to test. This example uses `ghcr.io/estuary/materialize-motherduck:local`.

```bash
docker build -t ghcr.io/estuary/materialize-motherduck:local --platform=linux/amd64 -f materialize-motherduck/Dockerfile .
```

--------------------------------

### Example Binding Configuration in YAML

Source: https://github.com/estuary/connectors/blob/main/docs/materialize/README.md

This YAML snippet shows the structure of a binding configuration, detailing resource details, source collection, fields, backfill strategy, and schema change handling.

```yaml
"bindings": [
    {
      "resource": {
        "_meta": {
          "path": [
            "public",
            "lovely_table"
          ]
        },
        "schema": "public",
        "table": "lovely_table"
      },
      "source": {
        "name": "estuary/source-something/my-lovely-collection",
      },
      "fields": {
        "require": {
          "some-optional-field-i-want-to-include": {}
        },
        "recommended": true
      },
      "backfill": 0,
      "onIncompatibleSchemaChange": "backfill"
    },
    ...
]
```

--------------------------------

### Run PostgreSQL Test Database and Connector Tests

Source: https://github.com/estuary/connectors/blob/main/source-postgres/README.md

Commands to start the test database using Docker Compose and run the complete test suite for the connector.

```bash
## Start the test database running
$ docker compose -f source-postgres/docker-compose.yaml up -d

## Run the complete test suite against the database we just launched
$ TEST_DATABASE=yes go test -v ./source-postgres/ -count=1
```

--------------------------------

### Test Meltano CLI Invocation

Source: https://github.com/estuary/connectors/blob/main/source-shopify/source_shopify/README.md

Verify the Meltano installation by invoking a specific tap with the --version flag. This confirms the CLI is accessible and the tap is recognized.

```bash
meltano invoke tap-shopify --version
```

--------------------------------

### Flow YAML Configuration for Discovery

Source: https://github.com/estuary/connectors/blob/main/source-snowflake/README.md

Example `flow.yaml` file demonstrating the configuration for discovering Snowflake CDC sources. This includes connector image, credentials, and database details.

```yaml
captures:
  acmeCo/source-snowflake:
    endpoint:
      connector:
        image: "ghcr.io/estuary/source-snowflake:v1"
        config:
          credentials:
            auth_type: jwt
            user: snowflake_user
            privateKey: |
              -----BEGIN PRIVATE KEY-----
              MIIEv....
              ...
              ...
              ...
              ...
              ...
              -----END PRIVATE KEY-----
          host: bn92689.us-central1.gcp.snowflakecomputing.com
          database: CONNECTOR_TESTING
          warehouse: COMPUTE_WH
    bindings: []
```

--------------------------------

### Configure Flowctl for PostgreSQL Preview

Source: https://github.com/estuary/connectors/blob/main/source-postgres-batch/README.md

Defines the connector configuration in a YAML file for use with flowctl. This setup uses a local Go command to run the connector directly.

```yaml
captures:
  acmeCo/example/source-postgres-batch:
    endpoint:
      local:
        command: ["go", "run", "./source-postgres-batch"]
        config:
          address: "localhost:5432"
          user: "postgres"
          password: "postgres"
          database: "postgres"
        protobuf: true
    bindings: []
```

--------------------------------

### Configure Mock Handlers for Submit and Discovery

Source: https://github.com/estuary/connectors/blob/main/source-facebook-marketing-native/tests/README.md

Use these handlers to differentiate between POST requests for job submission and GET requests for entity discovery on the same URL pattern.

```python
mock_http.add_handler(r"/act_[^/]+/insights$", handle_submit, method="POST")     # Job submission
mock_http.add_handler(r"/act_[^/]+/insights$", handle_discovery, method="GET")   # Entity discovery
```

--------------------------------

### Materialization Spec Example (YAML)

Source: https://github.com/estuary/connectors/blob/main/README.md

Define the materialization and collection specifications for your test. This YAML file configures the connector image, authentication, database details, and data schema.

```yaml
materializations:
  acmeCo/materialize-motherduck:
    endpoint:
      connector:
        image: "ghcr.io/estuary/materialize-motherduck:local"
        config:
          token: <motherduck_service_token>
          database: my_db
          schema: main
          stagingBucket:
            stagingBucketType: S3
            bucketS3: my_bucket
            awsAccessKeyId: <access_key_id>
            awsSecretAccessKey: <secret_access_key>
            region: us-east-1
          syncSchedule:
            syncFrequency: 0s
          advanced:
              no_flow_document: true
              feature_flags: allow_existing_tables_for_new_bindings
    bindings:
      - resource:
          table: tests_simple
        source: tests/simple

collections:
  tests/simple:
    schema:
      type: object
      properties:
        id: { type: integer }
        canary: { type: string }
      required: [id, canary]
    key: [/id]
```

--------------------------------

### Start and Stop Db2 Test Database

Source: https://github.com/estuary/connectors/blob/main/source-db2-batch/README.md

Use these Docker Compose commands to manage the lifecycle of the Db2 test database. Ensure the docker-compose.yaml file is in the correct directory.

```bash
docker compose -f source-db2-batch/docker-compose.yaml up -d
```

```bash
docker compose -f source-db2-batch/docker-compose.yaml down --volumes
```

--------------------------------

### Initialize Meltano Project

Source: https://github.com/estuary/connectors/blob/main/source-shopify/source_shopify/README.md

Navigate to your project directory and initialize Meltano. This command sets up the necessary files and configurations for your Meltano project.

```bash
cd tap-shopify
meltano install
```

--------------------------------

### Discover Kafka Topics and Partitions

Source: https://github.com/estuary/connectors/blob/main/source-kafka/README.md

List available topics and partitions in your Kafka cluster using the `discover` command. Mount your configuration file to `/config.json`.

```bash
docker run -it --rm -v $(pwd)/YOUR_CONFIG.json:/config.json ghcr.io/estuary/source-kafka:TAG discover --config=/config.json
```

--------------------------------

### Airtable Formula Field Error Example (#ERROR!)

Source: https://github.com/estuary/connectors/blob/main/source-airtable-native/README.md

Shows an example of an Airtable formula field error where the API returns an object with an 'error' property.

```json
{
    "myFormulaField": {
        "error": "#ERROR!"
    }
}
```

--------------------------------

### Preview Connector with Flowctl

Source: https://github.com/estuary/connectors/blob/main/source-postgres-batch/README.md

Runs the connector in preview mode using the configured flowctl YAML file. This allows for testing the connector's output without a full deployment.

```bash
flowctl preview --source acmeCo/flow.yaml
```

--------------------------------

### Parse and Apply Feature Flags

Source: https://github.com/estuary/connectors/blob/main/docs/feature_flags.md

Implement startup logic in `main.go` to parse the `FeatureFlags` property, merge with defaults, and save to the state struct.

```go
var featureFlags = common.ParseFeatureFlags(config.Advanced.FeatureFlags, featureFlagDefaults)
    if config.Advanced.FeatureFlags != "" {
        log.WithField("flags", featureFlags).Info("parsed feature flags")
    }
    // [...a bit later...]
    db.featureFlags = featureFlags
```

--------------------------------

### Run Integration Tests for Connector

Source: https://github.com/estuary/connectors/blob/main/source-kafka/README.md

Execute integration tests that connect to Kafka. Ensure `make test_setup` is run first.

```bash
cd path/to/estuary/connectors/source-kafka
make test_setup
cargo test
```

--------------------------------

### Facebook API Error Example

Source: https://github.com/estuary/connectors/blob/main/source-facebook-marketing-native/tests/README.md

Illustrates the structure of a generic Facebook API error response.

```python
create_facebook_api_http_error(code=190, message="Invalid token")
```

--------------------------------

### Download Flow Binaries

Source: https://github.com/estuary/connectors/blob/main/README.md

Run this script to download the necessary flow binaries, which are required for building most connector images. The binaries will be placed in the `./flow-bin` directory.

```bash
./fetch-flow.sh
```

--------------------------------

### Execute MySQL Command

Source: https://github.com/estuary/connectors/blob/main/source-mysql-batch/README.md

Executes a MySQL command within a Docker container. This is useful for interacting with the database directly for debugging or setup.

```bash
docker exec -it source-mysql-db-1 mysql test --user=root --port=3306 --host=127.0.0.1 --protocol=tcp --password=secret1234
```

--------------------------------

### Build and Run Integration Test for PostgreSQL Connector

Source: https://github.com/estuary/connectors/blob/main/source-postgres/README.md

Steps to build a local Docker image for the connector and execute the integration test.

```bash
## Build a local version of the connector image
$ docker build --network=host -t ghcr.io/estuary/source-postgres:local -f source-postgres/Dockerfile .

## Run the integration test
$ CONNECTOR=source-postgres VERSION=local ./tests/run.sh
```

--------------------------------

### Example Mapped Access Token Response

Source: https://github.com/estuary/connectors/blob/main/python/README.md

The transformed access token response after applying the accessTokenResponseMap, showing the desired format for connector authentication.

```json
{
    "access_token": "ey..==",
    "refresh_token": "CRYPTOGRAPHIC_RANDOM_REFRESH",
    "token_expiry_date": "2023-12-14T15:05:00Z"
}
```

--------------------------------

### Build the Connector Locally

Source: https://github.com/estuary/connectors/blob/main/source-kafka/README.md

Build the `source-kafka` connector using Cargo. Navigate to the connector's directory first.

```bash
cd path/to/estuary/connectors/source-kafka
cargo build
```

--------------------------------

### Configuring Connector for Debugpy Remote Attach

Source: https://github.com/estuary/connectors/blob/main/python/README.md

Modify your `test.flow.yaml` to include `debugpy` in the command. This setup allows for network debugging of the connector process.

```yaml
captures:
  acmeCo/source-pokemon:
    endpoint:
      local:
        command:
          - python
          - "-m"
          - "debugpy"
          - "--listen"
          - "0.0.0.0:5678"
          - "--wait-for-client"
          - "-m"
          - source-bigdata

```

--------------------------------

### Create SQS Queue and Get ARNs

Source: https://github.com/estuary/connectors/blob/main/materialize-eventbridge/testdata/README.md

Creates an SQS queue named 'integration-tests' and retrieves its URL and ARN. These are essential for configuring EventBridge rules.

```bash
# 2. Queue
QURL=$(aws sqs create-queue --queue-name integration-tests \
  --query QueueUrl --output text)
QARN=$(aws sqs get-queue-attributes --queue-url "$QURL" \
  --attribute-names QueueArn --query Attributes.QueueArn --output text)
BUS_ARN=$(aws events describe-event-bus --name integration-tests \
  --query Arn --output text)
```

--------------------------------

### Build Connector Image

Source: https://github.com/estuary/connectors/blob/main/source-sqlserver/README.md

Builds the Docker image for the source-sqlserver connector, tagging it as 'local' for development purposes.

```bash
$ docker build --network=flow-test -t source-sqlserver:local -f source-sqlserver/Dockerfile .
```

--------------------------------

### Connector Spec Output Example

Source: https://github.com/estuary/connectors/blob/main/python/README.md

This JSON output represents the specification of a 'Big Data Spec' connector, including its configuration schemas and documentation URL.

```json
{
  "configSchema": {
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "Big Data Spec",
    "type": "object",
    "required": [
      "data_name"
    ],
    "properties": {
      "data_name": {
        "type": "string",
        "description": "Name requested from the API.",
        "pattern": "^[a-z0-9_\\-]+$"
      }
    }
  },
  "resourceConfigSchema": {
    "type": "object",
    "properties": {
      "stream": {
        "type": "string"
      },
      "syncMode": {
        "enum": [
          "incremental",
          "full_refresh"
        ]
      },
      "namespace": {
        "type": "string"
      },
      "cursorField": {
        "type": "array",
        "items": {
          "type": "string"
        }
      }
    },
    "required": [
      "stream",
      "syncMode"
    ]
  },
  "documentationUrl": "https://docs.bigdata.com/integrations/sources/bigdata"
}

```

--------------------------------

### Connect to SQL Server Instance

Source: https://github.com/estuary/connectors/blob/main/source-sqlserver/README.md

Use this command to connect to a running SQL Server instance managed by Docker Compose.

```bash
$ docker-compose -f ./source-sqlserver/docker-compose.yaml exec db /opt/mssql-tools18/bin/sqlcmd -C -U sa -P gf6w6dkD
```

--------------------------------

### Airtable Formula Field Error Example (NaN)

Source: https://github.com/estuary/connectors/blob/main/source-airtable-native/README.md

Illustrates how Airtable API returns an object with a 'specialValue' property for formula fields that result in NaN.

```json
{
  "myFormulaField": {
    "specialValue": "NaN"
  }
}
```

--------------------------------

### Minimal test.flow.yaml for Connector Spec Inspection

Source: https://github.com/estuary/connectors/blob/main/python/README.md

This YAML configuration is used to define a minimal connector setup for inspecting its specification output using `flowctl`.

```yaml
--
captures:
  acmeCo/source-bigdata:
    endpoint:
      local:
        command:
          - python
          - "-m"
          - source-bigdata
        config:
          region: antarctica
    bindings:
      - resource:
          stream: bigdata
          syncMode: full_refresh
        target: acmeCo/big
collections:
  acmeCo/bigdata:
    schema:
      type: object
      properties:
        _meta:
          type: object
          properties:
            row_id:
              type: integer
          required:
            - row_id
    key:
      - "/_meta/row_id"

```

--------------------------------

### Build Local Docker Image for Go Connectors

Source: https://github.com/estuary/connectors/blob/main/CLAUDE.md

Build a local Docker image for a Go connector. This involves vendoring dependencies and modifying the Dockerfile to use the vendored modules.

```bash
go mod vendor
./build-local.sh $connector
```

--------------------------------

### Complete Bulk Editing Workflow

Source: https://github.com/estuary/connectors/blob/main/docs/feature_flags.md

Demonstrates the complete workflow for fetching task specs, adding a feature flag, and publishing the modified specs.

```bash
# Fetch task specs for every source-mysql (and variants) capture in production.
$ ../scripts/list-tasks.sh --connector=source-mysql --pull --dir=./specs
# Add the 'do_some_thing' feature flag to the local endpoint configs, with interactive diffs.
$ ../scripts/bulk-config-editor.sh --set_flag=do_some_thing --dir=./specs
# Upload modified specs to the draft, renaming the local files when successful.
$ ../scripts/bulk-publish.sh --mark --dir=./specs
```

--------------------------------

### Create Cloud Bigtable Instance

Source: https://github.com/estuary/connectors/blob/main/materialize-bigtable/testdata/README.md

Use this command to create a new Cloud Bigtable instance for testing purposes. Ensure you have the gcloud SDK installed and configured.

```bash
gcloud bigtable instances create estuary-test \
  --display-name=estuary-test \
  --cluster-config=id=estuary-test-c1,zone=us-east1-d,nodes=1 \
  --project=estuary-theatre
```

--------------------------------

### Run Flowctl Preview with Fixture

Source: https://github.com/estuary/connectors/blob/main/README.md

Execute the `flowctl preview` command with your spec and fixture files to test the materialization connector. The `RUST_LOG=DEBUG` environment variable enables detailed logging.

```bash
RUST_LOG=DEBUG flowctl preview --source ./my-spec.yaml --fixture ./my-fixture.json --output-state --output-apply
```

--------------------------------

### RingCentral Timestamp Filtering Example

Source: https://github.com/estuary/connectors/blob/main/source-ringcentral/README.md

Demonstrates millisecond-inclusive filtering for date parameters in the RingCentral API. Ensure your date parameters are formatted correctly for precise filtering.

```text
dateFrom=2026-01-26T14:16:23.561Z&dateTo=2026-01-26T14:16:24.000Z
```

--------------------------------

### Run End-to-End Connector Tests

Source: https://github.com/estuary/connectors/blob/main/source-kafka/README.md

Execute end-to-end tests for the `source-kafka` connector with Estuary Flow. Ensure developer setup is complete and specify the connector and version.

```bash
cd path/to/estuary/connectors
CONNECTOR=source-kafka VERSION=<<DOCKER_IMAGE_TAG>> ./tests/run.sh
```

--------------------------------

### Run Go Tests for MySQL Connector

Source: https://github.com/estuary/connectors/blob/main/source-mysql/README.md

Execute the connector's Go test suite with specific environment variables for the test database and log level.

```bash
$ TEST_DATABASE=yes LOG_LEVEL=debug go test -v ./source-mysql -count=1 -timeout=0
```

--------------------------------

### Tap Shopify CLI Commands

Source: https://github.com/estuary/connectors/blob/main/source-shopify/source_shopify/README.md

Execute the tap-shopify CLI for version check, help, and catalog discovery. Use '--config CONFIG' to specify a configuration file.

```bash
tap-shopify --version
```

```bash
tap-shopify --help
```

```bash
tap-shopify --config CONFIG --discover > ./catalog.json
```

--------------------------------

### Run Unit Tests for Connector

Source: https://github.com/estuary/connectors/blob/main/source-kafka/README.md

Execute unit tests for the `source-kafka` connector. Navigate to the connector's directory before running.

```bash
cd path/to/estuary/connectors/source-kafka
cargo test --lib
```

--------------------------------

### Model.json Schema Structure Example

Source: https://github.com/estuary/connectors/blob/main/source-dynamics-365-finance-and-operations/README.md

Defines the JSON structure for schema metadata files. This includes dataset information, entity definitions, and attribute details with their data types.

```json
{
  "name": "dataset_name",
  "description": "description",
  "version": "version_string",
  "entities": [
    {
      "$type": "entity_type",
      "name": "TableName",
      "description": "Table description",
      "attributes": [
        {
          "name": "ColumnName",
          "dataType": "string|int64|boolean|dateTime|decimal|guid|dateTimeOffset"
        }
      ]
    }
  ]
}
```

--------------------------------

### Python Connector Entrypoint with Airbyte Shim

Source: https://github.com/estuary/connectors/blob/main/python/README.md

The main entry point for a Python connector using the Airbyte CDK shim. Ensure the delegate and oauth2 parameters are correctly configured.

```python
from python import shim_airbyte_cdk
from source_bigdata import SourceBigData

shim_airbyte_cdk.CaptureShim(
    delegate=SourceBigData(),
    oauth2=None
).main()
```

--------------------------------

### Run a Test ELT Pipeline with Meltano

Source: https://github.com/estuary/connectors/blob/main/source-shopify/source_shopify/README.md

Execute a basic Extract, Load, Transform (ELT) pipeline using Meltano. This command runs the specified tap and target, demonstrating a full pipeline execution.

```bash
meltano elt tap-shopify target-jsonl
```

--------------------------------

### Run basic Go tests

Source: https://github.com/estuary/connectors/blob/main/source-firestore/README.md

Execute Go tests that do not require database interaction. This command skips most tests.

```bash
go test -v ./source-firestore
```

--------------------------------

### Build Docker Image

Source: https://github.com/estuary/connectors/blob/main/source-ga4-bigquery/README.md

Builds the Docker image for the source-ga4-bigquery connector locally.

```bash
docker build -t ghcr.io/estuary/source-ga4-bigquery:local -f source-ga4-bigquery/Dockerfile .
```

--------------------------------

### Mock HTTP Handler Order

Source: https://github.com/estuary/connectors/blob/main/source-facebook-marketing-native/tests/README.md

Demonstrates the correct and incorrect order for registering mock HTTP handlers to ensure specific URL patterns are matched appropriately. More specific patterns should be registered first.

```python
# CORRECT ORDER
mock_http.add_handler(r"_job_\d+/insights$", handle_results)  # More specific
mock_http.add_handler(r"_job_\d+$", handle_status)            # Less specific

# WRONG - status handler would match both URLs
mock_http.add_handler(r"_job_\d+", handle_status)   # Matches /job_1 AND /job_1/insights
mock_http.add_handler(r"_job_\d+/insights$", handle_results)  # Never reached
```

--------------------------------

### Example State JSON for Overlap Verification

Source: https://github.com/estuary/connectors/blob/main/tests/benchmark/materialize/README.md

This JSON structure represents the state of a transaction, including fresh key ranges and overlap counts. It is used to verify the correctness of connector operations.

```json
{
  "index": 1,
  "fresh":    {"start": 10000000, "count": 7000000, "op": "c"},
  "overlaps": [{"op": "u", "with": 0, "count": 2000000},
               {"op": "d", "with": 0, "count": 1000000}]
}
```

--------------------------------

### Timestamp Folder Naming Convention Example

Source: https://github.com/estuary/connectors/blob/main/source-dynamics-365-finance-and-operations/README.md

Shows the required UTC timestamp format for naming incremental data folders in ADLS Gen2. This format ensures chronological ordering and consistency.

```plaintext
yyyy-MM-ddTHH:mm:ss.SSSz
```

--------------------------------

### Build Docker Image for BigQuery Source Connector

Source: https://github.com/estuary/connectors/blob/main/source-bigquery-batch/README.md

Builds a local Docker image for the BigQuery source connector. Ensure you are in the correct directory.

```bash
$ docker build -t ghcr.io/estuary/source-bigquery-batch:local -f source-bigquery-batch/Dockerfile .
```