### Example Changelog Entry Source: https://github.com/meltano/sdk/blob/main/docs/release_process.md An example of how to format a changelog entry for a release, including highlights and links to detailed guides. ```markdown ## v0.41.0 (2024-10-02) ### Highlights - It's easier now for SQL tap developers to customize the mapping from SQL column types to JSON schema. See [the guide](https://sdk.meltano.com/en/v0.41.0/guides/sql-tap.html#custom-type-mapping) for details. ``` -------------------------------- ### Install Development Tools with uv Source: https://github.com/meltano/sdk/blob/main/docs/CONTRIBUTING.md Installs pre-commit and nox using the uv tool. Ensure uv is installed first. ```bash uv tool install pre-commit uv tool install nox ``` -------------------------------- ### Install Meltano Source: https://github.com/meltano/sdk/blob/main/cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/README.md Install Meltano using uv. ```bash uv tool install meltano ``` -------------------------------- ### Install tap-dummyjson from PyPI Source: https://github.com/meltano/sdk/blob/main/packages/meltano-tap-dummyjson/README.md Install the tap using pipx for easy command-line access. This is the recommended method for most users. ```bash pipx install tap-dummyjson ``` -------------------------------- ### Install tap-dummyjson from GitHub Source: https://github.com/meltano/sdk/blob/main/packages/meltano-tap-dummyjson/README.md Install the tap directly from its GitHub repository. Use this if you need the latest development version or are contributing to the project. ```bash pipx install git+https://github.com/ORG_NAME/tap-dummyjson.git@main ``` -------------------------------- ### Install Tap from GitHub Source: https://github.com/meltano/sdk/blob/main/cookiecutter/tap-template/{{cookiecutter.tap_id}}/README.md Installs the tap directly from its GitHub repository using the 'uv' package manager. ```bash uv tool install git+https://github.com/ORG_NAME/{{ cookiecutter.tap_id }}.git@main ``` -------------------------------- ### setup() Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.Sink.md Perform any setup actions at the beginning of a Stream. This method is executed once per Sink instance, after instantiation. If a Schema change is detected, a new Sink is instantiated and this method is called again. ```APIDOC ## setup() ### Description Perform any setup actions at the beginning of a Stream. Setup is executed once per Sink instance, after instantiation. If a Schema change is detected, a new Sink is instantiated and this method is called again. ### Return type: None ``` -------------------------------- ### Install Tap from PyPI Source: https://github.com/meltano/sdk/blob/main/cookiecutter/tap-template/{{cookiecutter.tap_id}}/README.md Installs the tap using the 'uv' package manager from the Python Package Index. ```bash uv tool install {{ cookiecutter.tap_id }} ``` -------------------------------- ### Install Meltano and Plugins Source: https://github.com/meltano/sdk/blob/main/packages/meltano-tap-dummyjson/README.md Install Meltano globally using pipx and then initialize it within the tap's directory. This prepares the environment for Meltano-based orchestration. ```bash pipx install meltano ``` ```bash cd tap-dummyjson ``` ```bash meltano install ``` -------------------------------- ### Install {{ cookiecutter.mapper_id }} from GitHub Source: https://github.com/meltano/sdk/blob/main/cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/README.md Install the mapper directly from its GitHub repository. ```bash uv tool install git+https://github.com/ORG_NAME/{{ cookiecutter.mapper_id }}.git@main ``` -------------------------------- ### Install Development Environment Source: https://github.com/meltano/sdk/blob/main/CLAUDE.md Installs the full development environment for the Meltano Singer SDK, including all groups and extras. ```bash uv sync --all-groups --all-extras --all-packages # Full development environment ``` -------------------------------- ### initialized_at Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.Target.md Start time of the plugin. Returns the start time of the plugin. ```APIDOC ## property initialized_at ### Description Start time of the plugin. ### Returns - **int** - The start time of the plugin. ``` -------------------------------- ### Install Target from GitHub Source: https://github.com/meltano/sdk/blob/main/cookiecutter/target-template/{{cookiecutter.target_id}}/README.md Install the target directly from its GitHub repository. Replace ORG_NAME with the actual organization name. ```bash uv tool install git+https://github.com/ORG_NAME/{{ cookiecutter.target_id }}.git@main ``` -------------------------------- ### Install Package Dependencies with uv Source: https://github.com/meltano/sdk/blob/main/docs/CONTRIBUTING.md Installs all package groups and extras using uv. Navigate to the project directory first. ```bash cd sdk # Install package and dependencies: uv sync --all-groups --all-extras ``` -------------------------------- ### Constant Example Usage Source: https://github.com/meltano/sdk/blob/main/docs/classes/typing/singer_sdk.typing.Constant.md An example demonstrating the creation and JSON representation of a Constant. ```APIDOC ## Example Usage ### Description This example shows how to create a Constant and view its JSON output. ### Code ```python from singer_sdk.typing import Constant t = Constant("foo") print(t.to_json(indent=2)) ``` ### Output ```json { "const": "foo" } ``` ``` -------------------------------- ### setup Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.sql.SQLSink.md Sets up the Sink by creating the required Schema and Table entities in the target database. ```APIDOC ## setup() ### Description Set up Sink. This method is called on Sink creation, and creates the required Schema and Table entities in the target database. ### Return type None ``` -------------------------------- ### Install uv and Cookiecutter Source: https://github.com/meltano/sdk/blob/main/cookiecutter/tap-template/README.md Installs the uv package manager and the cookiecutter tool. Ensure uv is installed correctly by reopening your shell if necessary. ```bash # Install uv (see https://docs.astral.sh/uv/getting-started/installation/ for more options) curl -LsSf https://astral.sh/uv/install.sh | sh # You may need to reopen your shell at this point uv tool install cookiecutter ``` -------------------------------- ### Install Target from PyPI Source: https://github.com/meltano/sdk/blob/main/cookiecutter/target-template/{{cookiecutter.target_id}}/README.md Use this command to install the target directly from the Python Package Index. ```bash uv tool install {{ cookiecutter.target_id }} ``` -------------------------------- ### Meltano Plugin Configuration Example Source: https://github.com/meltano/sdk/blob/main/docs/cli_commands.md Example `meltano.yml` configuration for a custom tap plugin. This demonstrates how to declare settings and their types, which is necessary for Meltano to manage the plugin. ```yaml plugins: extractors: - name: my-tap namespace: my_tap executable: -e . capabilities: - state - catalog - discover settings: - name: base_url kind: string - name: api_key kind: password ``` -------------------------------- ### Environment Variable Configuration Example Source: https://github.com/meltano/sdk/blob/main/cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/AGENTS.md Example of environment variables for mapper configuration, including hashing options. These should be reflected in `.env.example`. ```dotenv {{ cookiecutter.mapper_id | upper | replace('-', '_') }}_HASH_EMAIL=false {{ cookiecutter.mapper_id | upper | replace('-', '_') }}_HASH_ALGORITHM=md5 # New setting ``` -------------------------------- ### Install {{ cookiecutter.mapper_id }} from PyPI Source: https://github.com/meltano/sdk/blob/main/cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/README.md Use this command to install the mapper from the Python Package Index. ```bash uv tool install {{ cookiecutter.mapper_id }} ``` -------------------------------- ### Get tap-dummyjson About Information Source: https://github.com/meltano/sdk/blob/main/packages/meltano-tap-dummyjson/README.md Run this command to get detailed information about the tap, including supported settings and capabilities. The output can be formatted as markdown. ```bash tap-dummyjson --about ``` -------------------------------- ### Example Environment Variables for Target Source: https://github.com/meltano/sdk/blob/main/cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md Defines example environment variables for a target, including API URL, API key, and batch size. These should be set in the .env.example file. ```dotenv TARGET_{{ cookiecutter.destination_name | upper | replace(' ', '_') }}_API_URL=https://api.example.com TARGET_{{ cookiecutter.destination_name | upper | replace(' ', '_') }}_API_KEY=your_api_key_here TARGET_{{ cookiecutter.destination_name | upper | replace(' ', '_') }}_BATCH_SIZE=1000 # New setting ``` -------------------------------- ### Default Log Output Example Source: https://github.com/meltano/sdk/blob/main/docs/implementation/logging.md This example demonstrates the default log format produced by the Singer SDK, showing timestamps, log levels, logger names, and messages. ```text 2022-12-05 19:46:46,744 | INFO | my_tap | Added 'child' as child stream to 'my_stream' 2022-12-05 19:46:46,744 | INFO | my_tap | Beginning incremental sync of 'my_stream'... 2022-12-05 19:46:46,744 | INFO | my_tap | Tap has custom mapper. Using 1 provided map(s). 2022-12-05 19:46:46,745 | INFO | my_tap | Beginning full_table sync of 'child' with context: {'parent_id': 1}... 2022-12-05 19:46:46,745 | INFO | my_tap | Tap has custom mapper. Using 1 provided map(s). 2022-12-05 19:46:46,746 | INFO | singer_sdk.metrics | INFO METRIC: {"metric_type": "timer", "metric": "sync_duration", "value": 0.0005319118499755859, "tags": {"stream": "child", "context": {"parent_id": 1}, "status": "succeeded"}} 2022-12-05 19:46:46,747 | INFO | singer_sdk.metrics | INFO METRIC: {"metric_type": "counter", "metric": "record_count", "value": 3, "tags": {"stream": "child", "context": {"parent_id": 1}}} 2022-12-05 19:46:46,747 | INFO | my_tap | Beginning full_table sync of 'child' with context: {'parent_id': 2}... 2022-12-05 19:46:46,748 | INFO | singer_sdk.metrics | INFO METRIC: {"metric_type": "timer", "metric": "sync_duration", "value": 0.0004410743713378906, "tags": {"stream": "child", "context": {"parent_id": 2}, "status": "succeeded"}} 2022-12-05 19:46:46,748 | INFO | singer_sdk.metrics | INFO METRIC: {"metric_type": "counter", "metric": "record_count", "value": 3, "tags": {"stream": "child", "context": {"parent_id": 2}}} 2022-12-05 19:46:46,749 | INFO | my_tap | Beginning full_table sync of 'child' with context: {'parent_id': 3}... 2022-12-05 19:46:46,749 | INFO | singer_sdk.metrics | INFO METRIC: {"metric_type": "timer", "metric": "sync_duration", "value": 0.0004508495330810547, "tags": {"stream": "child", "context": {"parent_id": 3}, "status": "succeeded"}} 2022-12-05 19:46:46,750 | INFO | singer_sdk.metrics | INFO METRIC: {"metric_type": "counter", "metric": "record_count", "value": 3, "tags": {"stream": "child", "context": {"parent_id": 3}}} 2022-12-05 19:46:46,750 | INFO | singer_sdk.metrics | INFO METRIC: {"metric_type": "timer", "metric": "sync_duration", "value": 0.0052759647369384766, "tags": {"stream": "my_stream", "context": {}, "status": "succeeded"}} 2022-12-05 19:46:46,750 | INFO | singer_sdk.metrics | INFO METRIC: {"metric_type": "counter", "metric": "record_count", "value": 3, "tags": {"stream": "my_stream", "context": {}}} ``` -------------------------------- ### ObjectType Examples Source: https://github.com/meltano/sdk/blob/main/docs/classes/typing/singer_sdk.typing.ObjectType.md Demonstrates how to create and use ObjectType with different configurations for properties and additional properties. ```APIDOC ## Examples ```pycon >>> t = ObjectType( ... Property("name", StringType, required=True), ... Property("age", IntegerType), ... Property("height", DecimalType), ... additional_properties=False, ... ) >>> print(t.to_json(indent=2)) { "type": "object", "properties": { "name": { "type": [ "string" ] }, "age": { "type": [ "integer", "null" ] }, "height": { "type": [ "number", "null" ] } }, "required": [ "name" ], "additionalProperties": false } >>> t = ObjectType( ... Property("name", StringType, required=True), ... Property("age", IntegerType), ... Property("height", DecimalType), ... additional_properties=StringType, ... ) >>> print(t.to_json(indent=2)) { "type": "object", "properties": { "name": { "type": [ "string" ] }, "age": { "type": [ "integer", "null" ] }, "height": { "type": [ "number", "null" ] } }, "required": [ "name" ], "additionalProperties": { "type": [ "string" ] } } ``` ``` -------------------------------- ### Install Cookiecutter and Tox Source: https://github.com/meltano/sdk/blob/main/docs/dev_guide.md Install the necessary tools for project initialization and development workflows. Cookiecutter is used for scaffolding new projects, and Tox can be used for automated tasks like linting and testing. ```bash uv tool install cookiecutter # Optional: Install Tox if you want to use it to run auto-formatters, linters, tests, etc. uv tool install tox ``` -------------------------------- ### Get Mapper Configuration Options Source: https://github.com/meltano/sdk/blob/main/cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/README.md Run this command to view all supported settings and capabilities for the mapper. ```bash {{ cookiecutter.mapper_id }} --about ``` -------------------------------- ### BasicAuthenticator Example Source: https://context7.com/meltano/sdk/llms.txt Use `BasicAuthenticator` for simple username and password authentication. ```python from __future__ import annotations from singer_sdk import RESTStream from singer_sdk.authenticators import ( APIKeyAuthenticator, BearerTokenAuthenticator, BasicAuthenticator, OAuthAuthenticator, ) class BasicAuthStream(RESTStream): @property def authenticator(self): return BasicAuthenticator( username=self.config["username"], password=self.config["password"], ) ``` -------------------------------- ### Function Call for String Length Source: https://github.com/meltano/sdk/blob/main/cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/AGENTS.md Example of using the 'len()' function to get the length of a 'description' field. ```python "length": "len(description)" ``` -------------------------------- ### Run Tap in Discovery and Sync Modes Source: https://context7.com/meltano/sdk/llms.txt Command-line examples for running a tap in discovery mode to generate a catalog, and in sync mode using a configuration and catalog file. ```bash # Discovery mode tap-github --config config.json --discover > catalog.json # Sync mode with state tap-github --config config.json --catalog catalog.json --state state.json ``` -------------------------------- ### get_record_counter Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.Stream.md Get a counter for counting records retrieved from the source. This method can be overridden to customize the record counter, for example to add custom tags or modify the logging behavior. ```APIDOC ## get_record_counter(endpoint=None, log_interval=60.0) ### Description Get a counter for counting records retrieved from the source. This method can be overridden to customize the record counter, for example to add custom tags or modify the logging behavior. ### Parameters * **endpoint** (*str* | *None*) – The endpoint name to include in metrics. * **log_interval** (*float*) – The interval at which to log the count. ### Returns A counter for counting records. ### Return type *Counter* ### Versionadded Added in version 0.51.0. ``` -------------------------------- ### get_starting_replication_key_value Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.Stream.md Get starting replication key. Will return the value of the stream’s replication key when –state is passed. If no prior state exists, will return None. Developers should use this method to seed incremental processing for non-datetime replication keys. ```APIDOC ## get_starting_replication_key_value(context) ### Description Get starting replication key. Will return the value of the stream’s replication key when –state is passed. If no prior state exists, will return None. Developers should use this method to seed incremental processing for non-datetime replication keys. For datetime and date replication keys, use [`get_starting_timestamp()`](#singer_sdk.Stream.get_starting_timestamp) ### Parameters * **context** (*types.Context* | *None*) – Stream partition or context dictionary. ### Returns Starting replication value. ### Return type t.Any | None ### NOTE This method requires [`replication_key`](#singer_sdk.Stream.replication_key) to be set to a non-null value, indicating the stream should be synced incrementally. ``` -------------------------------- ### __init__ Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.Tap.md Initializes the tap with configuration, catalog, and state. ```APIDOC ## __init__ ### Description Initialize the tap. ### Parameters * **config** (*dict* *|* *PurePath* *|* *str* *|* *list* *[**PurePath* *|* *str* *]* *|* *None*) – Tap configuration. Can be a dictionary, a single path to a configuration file, or a list of paths to multiple configuration files. * **catalog** (*PurePath* *|* *str* *|* *dict* *|* *Catalog* *|* *None*) – Tap catalog. Can be a dictionary or a path to the catalog file. * **state** (*PurePath* *|* *str* *|* *dict* *|* *None*) – Tap state. Can be dictionary or a path to the state file. * **parse_env_config** (*bool*) – Whether to look for configuration values in environment variables. * **validate_config** (*bool*) – True to require validation of config settings. * **setup_mapper** (*bool*) – True to initialize the plugin mapper. * **message_writer** (*GenericSingerWriter* *|* *None*) – The class class to use for writing Singer messages. ### Return type None ``` -------------------------------- ### Execute tap-dummyjson Directly Source: https://github.com/meltano/sdk/blob/main/packages/meltano-tap-dummyjson/README.md Demonstrates direct execution of the tap for checking version, help, and discovering the catalog. Use the --config option to specify a configuration file or environment variables. ```bash tap-dummyjson --version ``` ```bash tap-dummyjson --help ``` ```bash tap-dummyjson --config CONFIG --discover > ./catalog.json ``` -------------------------------- ### Serve Documentation Locally with Live Reload Source: https://github.com/meltano/sdk/blob/main/docs/CONTRIBUTING.md Builds and serves the project documentation locally with live-reloading capabilities. This allows for real-time preview of documentation changes. ```bash nox -s docs-serve ``` -------------------------------- ### OneOf Example Usage Source: https://github.com/meltano/sdk/blob/main/docs/classes/typing/singer_sdk.typing.OneOf.md An example demonstrating the creation and JSON output of a OneOf type with StringType and IntegerType. ```APIDOC ## Example ```python >>> from singer_sdk.typing import OneOf, StringType, IntegerType >>> t = OneOf(StringType, IntegerType) >>> print(t.to_json(indent=2)) { "oneOf": [ { "type": [ "string" ] }, { "type": [ "integer" ] } ] } ``` ``` -------------------------------- ### Run --help with uv Source: https://github.com/meltano/sdk/blob/main/docs/cli_commands.md Execute the `--help` command for a tap using `uv` for environment management. ```bash uv sync && \ uv run tap-mysource --help ``` -------------------------------- ### Run in discovery mode with uv Source: https://github.com/meltano/sdk/blob/main/docs/cli_commands.md Execute a tap in discovery mode using `uv`, specifying a configuration file. ```bash uv sync && \ uv run tap-mysource --discover \ --config singer_sdk/samples/sample_tap_parquet/parquet-config.sample.json ``` -------------------------------- ### Build Documentation Source: https://github.com/meltano/sdk/blob/main/CLAUDE.md Builds the Sphinx documentation for the SDK using Nox. ```bash nox -s docs # Build Sphinx documentation ``` -------------------------------- ### Initialize Development Environment Source: https://github.com/meltano/sdk/blob/main/cookiecutter/tap-template/{{cookiecutter.tap_id}}/README.md Synchronizes development dependencies using 'uv', a Python package and environment manager. ```bash uv sync ``` -------------------------------- ### Run in sync mode with catalog file using uv Source: https://github.com/meltano/sdk/blob/main/docs/cli_commands.md Execute a tap in sync mode using `uv`, specifying both a configuration and a catalog file. ```bash uv sync && \ uv run tap-mysource \ --config singer_sdk/samples/sample_tap_parquet/parquet-config.sample.json \ --catalog singer_sdk/samples/sample_tap_parquet/parquet-catalog.sample.json ``` -------------------------------- ### Example Metrics JSONL Output Source: https://github.com/meltano/sdk/blob/main/docs/implementation/logging.md This is an example of the JSON Lines (jsonl) format for metrics output, showing timer and counter metrics with associated tags. ```json {"created": 1705709074.883021, "point": {"type": "timer", "metric": "http_request_duration", "value": 0.501743, "tags": {"stream": "continents", "endpoint": "", "http_status_code": 200, "status": "succeeded"}}} {"created": 1705709074.897184, "point": {"type": "counter", "metric": "http_request_count", "value": 1, "tags": {"stream": "continents", "endpoint": ""}}} {"created": 1705709074.897256, "point": {"type": "timer", "metric": "sync_duration", "value": 0.7397160530090332, "tags": {"stream": "continents", "context": {}, "status": "succeeded"}}} {"created": 1705709074.897292, "point": {"type": "counter", "metric": "record_count", "value": 7, "tags": {"stream": "continents", "context": {}}}} {"created": 1705709075.397254, "point": {"type": "timer", "metric": "http_request_duration", "value": 0.392148, "tags": {"stream": "countries", "endpoint": "", "http_status_code": 200, "status": "succeeded"}}} {"created": 1705709075.421888, "point": {"type": "counter", "metric": "http_request_count", "value": 1, "tags": {"stream": "countries", "endpoint": ""}}} {"created": 1705709075.422001, "point": {"type": "timer", "metric": "sync_duration", "value": 0.5258760452270508, "tags": {"stream": "countries", "context": {}, "status": "succeeded"}}} {"created": 1705709075.422047, "point": {"type": "counter", "metric": "record_count", "value": 250, "tags": {"stream": "countries", "context": {}}}} ``` -------------------------------- ### Test Mapper with Sample Input Source: https://github.com/meltano/sdk/blob/main/cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/AGENTS.md Install dependencies and test your mapper by piping sample data through it. This is useful for verifying transformations. ```bash # Install dependencies uv sync # Test mapper with sample input cat sample_input.singer | {{ cookiecutter.mapper_id }} --config config.json # Full pipeline test tap-something | {{ cookiecutter.mapper_id }} --config mapper_config.json | target-something --config target_config.json ``` -------------------------------- ### Execute Target Directly Source: https://github.com/meltano/sdk/blob/main/cookiecutter/target-template/{{cookiecutter.target_id}}/README.md Demonstrates how to run the target executable directly from the command line, including version check, help display, and piping data from a tap. ```bash {{ cookiecutter.target_id }} --version ``` ```bash {{ cookiecutter.target_id }} --help ``` ```bash # Test using the "Smoke Test" tap: tap-smoke-test | {{ cookiecutter.target_id }} --config /path/to/{{ cookiecutter.target_id }}-config.json ``` -------------------------------- ### Run --help with Poetry Source: https://github.com/meltano/sdk/blob/main/docs/cli_commands.md Execute the `--help` command for a tap using `poetry` for environment management. ```bash poetry sync && \ poetry run tap-mysource --help ``` -------------------------------- ### Run in discovery mode with catalog file using uv Source: https://github.com/meltano/sdk/blob/main/docs/cli_commands.md Execute a tap in discovery mode using `uv`, specifying both configuration and catalog files. Note: The `--catalog` option may not be supported by all connectors during discovery. ```bash uv sync && \ uv run tap-mysource --discover \ --config singer_sdk/samples/sample_tap_parquet/parquet-config.sample.json \ --catalog singer_sdk/samples/sample_tap_parquet/parquet-catalog.sample.json ``` -------------------------------- ### __init__ Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.sql.SQLSink.md Initializes the SQLSink object with target, stream name, schema, key properties, and an optional connector. ```APIDOC ## __init__ ### Description Initialize SQL Sink. ### Parameters * **target** ([*Target*](singer_sdk.Target.md#singer_sdk.Target)) – The target object. * **stream_name** (*str*) – The source tap’s stream name. * **schema** (*dict*) – The JSON Schema definition. * **key_properties** (*t.Sequence* *[**str* *]* *|* *None*) – The primary key columns. * **connector** ( *\_C* *|* *None*) – Optional connector to reuse. ### Return type None ``` -------------------------------- ### Context Dictionary Example for Path Replacement Source: https://github.com/meltano/sdk/blob/main/docs/parent_streams.md Example of a context dictionary passed from a parent stream to a child stream. These keys are used to automatically replace placeholders in the child stream's `path`. ```python { "group_id": 123, "epic_id": 456, "epic_iid": 789, } ``` -------------------------------- ### Parent-Child Stream Implementation Example Source: https://github.com/meltano/sdk/blob/main/docs/parent_streams.md Example demonstrating a child stream inheriting from a parent stream. Configure `parent_stream_type` and `ignore_parent_replication_key` for child streams. The `path` parameter is auto-populated using parent context keys. ```python class GitlabStream(RESTStream): # Base stream definition with auth and pagination logic # This logic works for other base classes as well, including Stream, GraphQLStream, etc. class EpicsStream(GitlabStream): name = "epics" # ... def get_child_context(self, record: dict, context: Optional[dict]) -> dict: """Return a context dictionary for child streams.""" return { "group_id": record["group_id"], "epic_id": record["id"], "epic_iid": record["iid"], } class EpicIssuesStream(GitlabStream): # Note that this class inherits from the GitlabStream base class, and not from # the EpicsStream class. name = "epic_issues" # EpicIssues streams should be invoked once per parent epic: parent_stream_type = EpicsStream # Assume epics don't have `updated_at` incremented when issues are changed: ignore_parent_replication_key = True # Path is auto-populated using parent context keys: path = "/groups/{group_id}/epics/{epic_iid}/issues" # ... ``` -------------------------------- ### Execute Tap Directly Source: https://github.com/meltano/sdk/blob/main/cookiecutter/tap-template/{{cookiecutter.tap_id}}/README.md Demonstrates direct execution of the tap for version checking, help information, and catalog discovery. ```bash {{ cookiecutter.tap_id }} --version {{ cookiecutter.tap_id }} --help {{ cookiecutter.tap_id }} --config CONFIG --discover > ./catalog.json ``` -------------------------------- ### logger Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.Stream.md Get stream logger. ```APIDOC ## property logger: Logger ### Description Get stream logger. ``` -------------------------------- ### state_manager Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.Stream.md Get stream state manager. ```APIDOC ## property state_manager: StreamStateManager ### Description Get stream state manager. ``` -------------------------------- ### emit_activate_version_messages Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.Stream.md Get whether to emit ACTIVATE_VERSION messages. ```APIDOC ## property emit_activate_version_messages: bool ### Description Get whether to emit ACTIVATE_VERSION messages. ``` -------------------------------- ### Run in discovery mode with Poetry Source: https://github.com/meltano/sdk/blob/main/docs/cli_commands.md Execute a tap in discovery mode using `poetry`, specifying a configuration file. ```bash poetry sync && \ poetry run tap-mysource --discover \ --config singer_sdk/samples/sample_tap_parquet/parquet-config.sample.json ``` -------------------------------- ### user_agent Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.GraphQLStream.md Get the user agent string for the stream. ```APIDOC ## Property: user_agent ### Description Get the user agent string for the stream. ### Returns: The user agent string. ``` -------------------------------- ### Adding a New Configuration Setting Source: https://github.com/meltano/sdk/blob/main/cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md Illustrates how to add a new configuration setting (`batch_size`) to the tap's schema, meltano.yml, and .env.example file. ```python # {{ cookiecutter.library_name }}/tap.py config_jsonschema = th.PropertiesList( th.Property("api_url", th.StringType, required=True), th.Property("api_key", th.StringType, required=True, secret=True), th.Property("batch_size", th.IntegerType, default=100), # New setting ).to_dict() ``` ```yaml # meltano.yml plugins: extractors: - name: {{ cookiecutter.tap_id }} settings: - name: api_url kind: string - name: api_key kind: string sensitive: true - name: batch_size # New setting kind: integer value: 100 ``` ```bash # .env.example TAP_{{ cookiecutter.source_name | upper | replace(' ', '_') }}_API_URL=https://api.example.com TAP_{{ cookiecutter.source_name | upper | replace(' ', '_') }}_API_KEY=your_api_key_here TAP_{{ cookiecutter.source_name | upper | replace(' ', '_') }}_BATCH_SIZE=100 # New setting ``` -------------------------------- ### get_new_paginator Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.GraphQLStream.md Get a fresh paginator for this API endpoint. ```APIDOC ## get_new_paginator() ### Description Get a fresh paginator for this API endpoint. ### Returns A paginator instance, or `None` to indicate pagination is not supported. ### Return type BaseAPIPaginator | None ``` -------------------------------- ### Run in sync mode with uv Source: https://github.com/meltano/sdk/blob/main/docs/cli_commands.md Execute a tap in sync mode using `uv` and specifying a configuration file. ```bash uv sync && \ uv run tap-mysource \ --config singer_sdk/samples/sample_tap_parquet/parquet-config.sample.json ``` -------------------------------- ### get_http_request_counter Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.GraphQLStream.md Get the HTTP request counter for the stream. ```APIDOC ## get_http_request_counter() ### Description Get the HTTP request counter for the stream. ### Returns The HTTP request counter for the stream. ### Return type Counter ``` -------------------------------- ### PropertiesList Initialization and Usage Source: https://github.com/meltano/sdk/blob/main/docs/classes/typing/singer_sdk.typing.PropertiesList.md Demonstrates how to initialize a PropertiesList with various Property types, including secret properties and properties that require others. It also shows how to convert the schema to a JSON representation. ```APIDOC ## PropertiesList ### Description Properties list. A convenience wrapper around the ObjectType class. ### Example ```pycon >>> schema = PropertiesList( ... # username/password ... Property("username", StringType, requires_properties=["password"]), ... Property("password", StringType, secret=True), ... # OAuth ... Property( ... "client_id", ... StringType, ... requires_properties=["client_secret", "refresh_token"], ... ), ... Property("client_secret", StringType, secret=True), ... Property("refresh_token", StringType, secret=True), ... ) >>> print(schema.to_json(indent=2)) { "type": "object", "properties": { "username": { "type": [ "string", "null" ] }, "password": { "type": [ "string", "null" ], "secret": true, "writeOnly": true }, "client_id": { "type": [ "string", "null" ] }, "client_secret": { "type": [ "string", "null" ], "secret": true, "writeOnly": true }, "refresh_token": { "type": [ "string", "null" ], "secret": true, "writeOnly": true } }, "dependentRequired": { "username": [ "password" ], "client_id": [ "client_secret", "refresh_token" ] }, "$schema": "https://json-schema.org/draft/2020-12/schema" } ``` ### Methods #### __init__(*args, additional_properties=None, **kwargs) Initialize PropertiesList. * Parameters: * *args* (*Any*) – Arguments to pass to the parent class. * *additional_properties* (*W* *|* *type* *[**W* *]* *|* *bool* *|* *None*) – Additional properties to allow. * **kwargs** (*Any*) – Keyword arguments to pass to the parent class. * Return type: None #### append(property) Append a property to the property list. * Parameters: **property** ([*Property*](singer_sdk.typing.Property.md#singer_sdk.typing.Property)) – Property to add * Return type: None #### items() Get wrapped properties. * Returns: List of (name, property) tuples. * Return type: ItemsView[str, [*Property*](singer_sdk.typing.Property.md#singer_sdk.typing.Property)] #### *property* type_dict *: dict* Get type dictionary. * Returns: A dictionary describing the type. ``` -------------------------------- ### Property.type_dict Source: https://github.com/meltano/sdk/blob/main/docs/typing.md Get the dictionary representation of the property's type. ```APIDOC ## Property.type_dict ### Description Get type dictionary. ### Returns A dictionary describing the type. ### Raises **ValueError** – If the type dict is not valid. ``` -------------------------------- ### Standard State Format Example Source: https://github.com/meltano/sdk/blob/main/docs/implementation/state.md Illustrates the basic structure for storing stream state, including a 'bookmarks' object with per-stream entries. ```js { "bookmarks": { "orders": { // 'orders' state stored here }, "customers": { // 'customers' state stored here } } } ``` -------------------------------- ### connection Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.sql.SQLSink.md Property to get or set the SQLAlchemy connection for this sink. ```APIDOC ## connection ### Description Get or set the SQLAlchemy connection for this sink. ### Returns A connection object. ``` -------------------------------- ### Initialize Tap Streams with Different Types Source: https://github.com/meltano/sdk/blob/main/docs/code_samples.md This example shows how to initialize a collection of tap streams where each stream can be of a different type. The `discover_streams` method should return a list of stream instances. ```python class TapCountries(Tap): # ... def discover_streams(self) -> List[Stream]: """Return a list containing one each of the two stream types.""" return [ CountriesStream(tap=self), ContinentsStream(tap=self), ] ``` -------------------------------- ### requests_session Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.GraphQLStream.md Get the requests session object for HTTP requests. ```APIDOC ## Property: requests_session ### Description Get requests session. ### Returns: The `requests.Session` object for HTTP requests. ``` -------------------------------- ### oauth_request_payload (property) Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.authenticators.OAuthAuthenticator.md Gets the request body, which can be plain or encrypted. ```APIDOC ## oauth_request_payload ### Description Gets the request body, which can be plain or encrypted. ### Returns A plain (OAuth) or encrypted (JWT) request body. ### Return type dict ``` -------------------------------- ### Run in discovery mode with catalog file using Poetry Source: https://github.com/meltano/sdk/blob/main/docs/cli_commands.md Execute a tap in discovery mode using `poetry`, specifying both configuration and catalog files. Note: The `--catalog` option may not be supported by all connectors during discovery. ```bash poetry sync && \ poetry run tap-mysource --discover \ --config singer_sdk/samples/sample_tap_parquet/parquet-config.sample.json \ --catalog singer_sdk/samples/sample_tap_parquet/parquet-catalog.sample.json ``` -------------------------------- ### Run Standard Tap Tests Source: https://github.com/meltano/sdk/blob/main/docs/code_samples.md This snippet demonstrates how to run the standard tap tests provided by the Meltano SDK. Import `get_standard_tap_tests` and your tap class, then iterate through the tests. ```python # Import the tests from singer_sdk.testing import get_standard_tap_tests # Import our tap class from tap_parquet.tap import TapParquet SAMPLE_CONFIG = { # ... } def test_sdk_standard_tap_tests(): """Run the built-in tap tests from the SDK.""" tests = get_standard_tap_tests(TapParquet, config=SAMPLE_CONFIG) for test in tests: test() ``` -------------------------------- ### IntegerType Initialization Source: https://github.com/meltano/sdk/blob/main/docs/classes/typing/singer_sdk.typing.IntegerType.md Demonstrates how to initialize and use the IntegerType class with different constraints. ```APIDOC ## IntegerType ### Description Represents an integer data type. ### Class Methods #### __new__(*args, **kwargs) ### Initializer #### __init__(minimum=None, maximum=None, exclusive_minimum=None, exclusive_maximum=None, multiple_of=None, **kwargs) Initialize IntegerType. * **Parameters**: * **minimum** (*int* | *None*) – Minimum numeric value. See the [JSON Schema reference](https://json-schema.org/understanding-json-schema/reference/numeric#range) for details. * **maximum** (*int* | *None*) – Maximum numeric value. [JSON Schema reference](https://json-schema.org/understanding-json-schema/reference/numeric#range) for details. * **exclusive_minimum** (*int* | *None*) – Exclusive minimum numeric value. [JSON Schema reference](https://json-schema.org/understanding-json-schema/reference/numeric#range) for details. * **exclusive_maximum** (*int* | *None*) – Exclusive maximum numeric value. See the [JSON Schema reference](https://json-schema.org/understanding-json-schema/reference/numeric#range) for details. * **multiple_of** (*int* | *None*) – A number that the value must be a multiple of. See the [JSON Schema reference](https://json-schema.org/understanding-json-schema/reference/numeric#multiples) for details. * **kwargs** (*Any*) – Additional keyword arguments to pass to the parent class. * **Return type:** None ### Examples ```pycon >>> IntegerType.type_dict {'type': ['integer']} >>> IntegerType().type_dict {'type': ['integer']} >>> IntegerType(allowed_values=[1, 2]).type_dict {'type': ['integer'], 'enum': [1, 2]} >>> IntegerType(minimum=0, maximum=10).type_dict {'type': ['integer'], 'minimum': 0, 'maximum': 10} >>> IntegerType(exclusive_minimum=0, exclusive_maximum=10).type_dict {'type': ['integer'], 'exclusiveMinimum': 0, 'exclusiveMaximum': 10} >>> IntegerType(multiple_of=2).type_dict {'type': ['integer'], 'multipleOf': 2} ``` ``` -------------------------------- ### replication_key Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.Stream.md Get replication key. Returns the replication key for the stream. ```APIDOC ## property replication_key: str | None ### Description Get replication key. ### Returns Replication key for the stream. ``` -------------------------------- ### descendent_streams Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.Stream.md Get child streams. Returns a list of all children, recursively. ```APIDOC ## property descendent_streams: list[Stream] ### Description Get child streams. ### Returns A list of all children, recursively. ``` -------------------------------- ### Custom SQLToJSONSchema Initialization Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.sql.connector.SQLToJSONSchema.md Demonstrates how to extend SQLToJSONSchema to include custom initialization logic using a configuration dictionary. This is useful for taps that require specific settings during converter instantiation. ```python class CustomSQLToJSONSchema(SQLToJSONSchema): def __init__(self, *, my_custom_option, **kwargs): super().__init__(**kwargs) self.my_custom_option = my_custom_option @classmethod def from_config(cls, config): return cls(my_custom_option=config.get("my_custom_option")) ``` -------------------------------- ### records_jsonpath Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.GraphQLStream.md Get the JSONPath expression to extract records from an API response. ```APIDOC ## Property: records_jsonpath ### Description Get the JSONPath expression to extract records from an API response. ### Returns: JSONPath expression string. ``` -------------------------------- ### VSCode Launch Configuration for Tap Discovery Source: https://github.com/meltano/sdk/blob/main/docs/dev_guide.md Example launch configuration for VSCode to debug a tap's discovery process. Ensure your VSCode interpreter is set to the project's virtual environment. ```json { // launch.json "version": "0.2.0", "configurations": [ { "name": "tap-snowflake discovery", "type": "python", "request": "launch", "module": "tap_snowflake.tap", "args": ["--config", "config.json", "--discover"], "python": "${command:python.interpreterPath}", // Set to true to debug third-party library code "justMyCode": false, } ] } ``` -------------------------------- ### config Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.InlineMapper.md Gets the configuration for the plugin. The configuration is provided as a read-only dictionary. ```APIDOC ## config ### Description Get config. ### Returns A frozen (read-only) config dictionary map. ### Return type Mapping[str, Any] ``` -------------------------------- ### Migrate project from Poetry to uv Source: https://github.com/meltano/sdk/blob/main/docs/guides/migrate-to-uv.md Run this command in your project directory to initiate the migration from Poetry to uv. ```bash uvx migrate-to-uv ``` -------------------------------- ### oauth_scopes (property) Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.authenticators.OAuthAuthenticator.md Gets the OAuth scopes as a string or None if not set. ```APIDOC ## oauth_scopes ### Description Gets the OAuth scopes as a string or None if not set. ### Returns String of OAuth scopes, or None if not set. ### Return type *str | None* ``` -------------------------------- ### max_parallelism Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.Target.md Get max parallel sinks. The default is 8 if not overridden. ```APIDOC ## property max_parallelism ### Description Get max parallel sinks. The default is 8 if not overridden. ### Returns - **int** - Max number of sinks that can be drained in parallel. ``` -------------------------------- ### Build a Simple Target Source: https://github.com/meltano/sdk/blob/main/docs/index.md Create a data loader by subclassing `singer_sdk.Target` and `singer_sdk.sinks.RecordSink`. Implement the `process_record` method to define how incoming records are handled. This example prints each record. ```python from singer_sdk import Target from singer_sdk.sinks import RecordSink class MySink(RecordSink): """My custom sink.""" def process_record(self, record: dict, context: dict) -> None: """Process a single record.""" # Your custom logic here print(f"Processing: {record}") class MyTarget(Target): """My custom target.""" name = "target-mydb" config_jsonschema = th.PropertiesList( th.Property("host", th.StringType, required=True), th.Property("port", th.IntegerType, default=5432), th.Property("database", th.StringType, required=True), th.Property("username", th.StringType, required=True), th.Property("password", th.StringType, required=True, secret=True), ).to_dict() default_sink_class = MySink if __name__ == "__main__": MyTarget.cli() ``` -------------------------------- ### oauth_request_body (property) Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.authenticators.OAuthAuthenticator.md Gets the formatted request body for OAuth authorization. ```APIDOC ## oauth_request_body ### Description Gets the formatted request body for OAuth authorization. ### Sample implementation ```python @property def oauth_request_body(self) -> dict: return { "grant_type": "password", "scope": "https://api.powerbi.com", "resource": "https://analysis.windows.net/powerbi/api", "client_id": self.client_id, "username": self.username, "password": self.password, } ``` ### Raises **NotImplementedError** – If derived class does not override this method. ### Return type dict ``` -------------------------------- ### auth_endpoint (property) Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.authenticators.OAuthAuthenticator.md Gets the authorization endpoint URL for OAuth 2.0. ```APIDOC ## auth_endpoint ### Description Gets the authorization endpoint URL for OAuth 2.0. ### Returns The API authorization endpoint if it is set. ### Raises **ValueError** – If the endpoint is not set. ### Return type *str* ``` -------------------------------- ### Run in sync mode with catalog file using Poetry Source: https://github.com/meltano/sdk/blob/main/docs/cli_commands.md Execute a tap in sync mode using `poetry`, specifying both a configuration and a catalog file. ```bash poetry sync && \ poetry run tap-mysource \ --config singer_sdk/samples/sample_tap_parquet/parquet-config.sample.json \ --catalog singer_sdk/samples/sample_tap_parquet/parquet-catalog.sample.json ``` -------------------------------- ### BearerTokenAuthenticator Example Source: https://context7.com/meltano/sdk/llms.txt Employ `BearerTokenAuthenticator` for token-based authentication, commonly used with JWTs. ```python from __future__ import annotations from singer_sdk import RESTStream from singer_sdk.authenticators import ( APIKeyAuthenticator, BearerTokenAuthenticator, BasicAuthenticator, OAuthAuthenticator, ) class BearerStream(RESTStream): @property def authenticator(self): return BearerTokenAuthenticator(token=self.config["token"]) ``` -------------------------------- ### Date/Time Formatting Expression Source: https://github.com/meltano/sdk/blob/main/cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/AGENTS.md Example of formatting a date using '.strftime()' for 'formatted_date'. ```python "formatted_date": "created_at.strftime('%Y-%m-%d')" ``` -------------------------------- ### replication_method Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.Stream.md Get replication method. Returns the replication method to be used for this stream. ```APIDOC ## property replication_method: str ### Description Get replication method. ### Returns Replication method to be used for this stream. ``` -------------------------------- ### __init__ Source: https://github.com/meltano/sdk/blob/main/docs/classes/singer_sdk.sql.SQLStream.md Initializes the SQLStream object. If a connector is not provided, a new one will be created. ```APIDOC ## __init__(tap, catalog_entry, connector=None) ### Description Initialize the database stream. If connector is omitted, a new connector will be created. ### Parameters * **tap** (*Tap*) – The parent tap object. * **catalog_entry** (*dict*) – Catalog entry dict. * **connector** (*SQLConnector | None*) – Optional connector to reuse. ### Return type None ```