### Install dlt Python Library

Source: https://dlthub.com/docs/intro/walkthroughs

To get started with dlt, install the library using pip. It is recommended to use a clean virtual environment for experiments.

```bash
pip install dlt
```

--------------------------------

### Install dlt-plus Python Package

Source: https://dlthub.com/docs/intro/devel/plus/getting-started/installation

Installs the dlt-plus Python package using pip. This is the quickstart method for getting dlt+ installed. A valid license is required to proceed after installation.

```Shell
pip install dlt-plus
```

--------------------------------

### Install dlt Python library

Source: https://dlthub.com/docs/intro/intro

Installs the dlt library using pip. It is recommended to use a clean virtual environment for your experiments to avoid conflicts.

```python
pip install dlt
```

--------------------------------

### Install dlt Library with PostgreSQL Support

Source: https://dlthub.com/docs/intro/pipelines/filesystem-gcs/load-data-with-python-from-filesystem-gcs-to-postgres

Command to install the dlt library with the necessary PostgreSQL extras using pip.

```bash
pip install "dlt[postgres]"
```

--------------------------------

### List Available dlt Sources

Source: https://dlthub.com/docs/intro/pipelines/sql_database/load-data-with-python-from-sql_database-to-snowflake

This command uses the dlt command-line interface to display a comprehensive list of all verified data sources that can be initialized and used with dlt pipelines.

```bash
dlt init --list
```

--------------------------------

### Install dlt library with filesystem extras

Source: https://dlthub.com/docs/intro/pipelines/rest_api/load-data-with-python-from-rest_api-to-filesystem

Installs the dlt library using pip, including the necessary 'filesystem' extras for integration with destinations like AWS S3.

```bash
pip install "dlt[filesystem]"
```

--------------------------------

### Install dlt Python Library

Source: https://dlthub.com/docs/intro/getting-started

This snippet shows how to install the dlt library using pip. It is recommended to use a clean virtual environment for your experiments to manage dependencies effectively.

```bash
pip install dlt
```

--------------------------------

### Install dlt-init-openapi CLI Tool

Source: https://dlthub.com/docs/intro/pipelines/digitalocean/load-data-with-python-from-digitalocean-to-yugabyte

Installs the `dlt-init-openapi` command-line interface tool, which is used to generate `dlt` sources from OpenAPI specifications.

```bash
pip install dlt-init-openapi
```

--------------------------------

### Install dlt with Snowflake Extras

Source: https://dlthub.com/docs/intro/pipelines/sql_database/load-data-with-python-from-sql_database-to-snowflake

This command installs the dlt library along with the necessary dependencies for integrating with Snowflake, ensuring all required components are available for pipeline development.

```bash
pip install "dlt[snowflake]"
```

--------------------------------

### Create and Navigate Project Directory

Source: https://dlthub.com/docs/intro/plus/getting-started/tutorial

This command creates a new directory named 'tutorial' and then changes the current working directory into it, preparing the environment for a new dlt+ project.

```bash
mkdir tutorial && cd tutorial
```

--------------------------------

### Install dlt with PostgreSQL Extras

Source: https://dlthub.com/docs/intro/pipelines/sql_database_mssql/load-data-with-python-from-sql_database_mssql-to-postgres

Installs the `dlt` library along with the necessary extras for PostgreSQL integration using pip.

```Shell
pip install "dlt[postgres]"
```

--------------------------------

### Create and navigate dlt+ project directory

Source: https://dlthub.com/docs/intro/devel/plus/getting-started/tutorial

This command creates a new directory named 'tutorial' and then changes the current working directory into it, preparing the environment for a new dlt+ project.

```bash
mkdir tutorial && cd tutorial
```

--------------------------------

### Visualize dlt Pipeline Output with Streamlit

Source: https://dlthub.com/docs/intro/pipelines/sql_database/load-data-with-python-from-sql_database-to-snowflake

Install Streamlit to create a web-based interface for inspecting the contents of your `dlt` pipeline's destination. After installation, run the Streamlit application for your pipeline directly using the `dlt` CLI to visualize the loaded data.

```bash
# install streamlit
pip install streamlit
# run the streamlit app for your pipeline with the dlt cli:
dlt pipeline rfam_database show
```

--------------------------------

### Initialize dlt Pipeline for SQL Database to Snowflake

Source: https://dlthub.com/docs/intro/pipelines/sql_database/load-data-with-python-from-sql_database-to-snowflake

These commands set up a new dlt project directory, initialize a pipeline configured to extract data from a SQL database and load it into Snowflake, and then install all necessary Python dependencies listed in the generated `requirements.txt` file.

```bash
mkdir my-sql_database-pipeline
cd my-sql_database-pipeline
dlt init sql_database snowflake
pip install -r requirements.txt
```

--------------------------------

### Initialize a new dlt+ Project with source and destination

Source: https://dlthub.com/docs/intro/devel/plus/getting-started/tutorial

Initializes a new dlt+ project named 'tutorial' (derived from the folder name) with an Arrow source and a DuckDB destination. This command automatically sets up a pipeline and a dataset.

```bash
# Initialize a dlt+ Project named "tutorial", the name is derived from the folder name
dlt project init arrow duckdb
```

--------------------------------

### Example Generated requirements.txt Content

Source: https://dlthub.com/docs/intro/pipelines/looker/load-data-with-python-from-looker-to-filesystem-local

This snippet shows an example of the `requirements.txt` file content, indicating the minimum required version for the `dlt` library that is installed as a dependency for the generated pipeline.

```text
dlt>=0.4.12
```

--------------------------------

### dlt Initialization Post-Setup Instructions

Source: https://dlthub.com/docs/intro/walkthroughs/add-a-verified-source

This snippet shows the typical command-line output after a successful dlt project initialization, providing crucial next steps such as reviewing usage examples, configuring credentials, and installing required Python dependencies.

```text
Verified source pipedrive was added to your project!
* See the usage examples and code snippets to copy from pipedrive_pipeline.py
* Add credentials for bigquery and other secrets in .dlt/secrets.toml
* Add the required dependencies to pyproject.toml:
  dlt[bigquery]>=0.3.1
  If the dlt dependency is already added, make sure you install the extra for bigquery to it
  If you are using poetry you may issue the following command:
  poetry add dlt -E bigquery

* Read https://dlthub.com/docs/walkthroughs/create-a-pipeline for more information
```

--------------------------------

### Install dlt with Synapse extras

Source: https://dlthub.com/docs/intro/pipelines/sql_database/load-data-with-python-from-sql_database-to-synapse

Installs the `dlt` library along with the necessary extra dependencies for integrating with Synapse, preparing the environment for data loading operations.

```bash
pip install "dlt[synapse]"
```

--------------------------------

### Initialize a New dlt+ Project

Source: https://dlthub.com/docs/intro/plus/getting-started/tutorial

Initializes a new dlt+ project named 'tutorial' (derived from the folder name) with an Arrow source and a DuckDB destination. This sets up the basic project structure including a pipeline and a dataset.

```bash
# Initialize a dlt+ Project named "tutorial", the name is derived from the folder name
dlt project init arrow duckdb
```

--------------------------------

### Install dlt with Filesystem Extras

Source: https://dlthub.com/docs/intro/pipelines/sql_database_mssql/load-data-with-python-from-sql_database_mssql-to-filesystem-gcs

Installs the `dlt` library with necessary dependencies for filesystem operations, specifically for Google Cloud Storage, using pip.

```bash
pip install "dlt[filesystem]"
```

--------------------------------

### Get help for dlt init command

Source: https://dlthub.com/docs/intro/devel/walkthroughs/add-a-verified-source

Displays available options and arguments for the `dlt init` command, providing detailed usage information.

```bash
dlt init --help
```

--------------------------------

### Install pymssql Python Driver

Source: https://dlthub.com/docs/intro/pipelines/sql_database_mssql/load-data-with-python-from-sql_database_mssql-to-postgres

This command installs the `pymssql` Python DB API driver, which is necessary for connecting to Microsoft SQL Server when using the `dlt` example pipeline.

```shell
pip install pymssql
```

--------------------------------

### List available dlt sources via CLI

Source: https://dlthub.com/docs/intro/devel/plus/getting-started/tutorial

Use this command to display a comprehensive list of all currently supported sources that can be integrated when initializing a new dlt+ project.

```bash
dlt source list-available
```

--------------------------------

### Install Python DB API Driver for MS SQL Server

Source: https://dlthub.com/docs/intro/pipelines/sql_database_mssql/load-data-with-python-from-sql_database_mssql-to-filesystem-gcs

This command installs `pymssql`, a Python DB API driver necessary for connecting to Microsoft SQL Server with the example dlt pipeline.

```Bash
pip install pymssql
```

--------------------------------

### Install Streamlit and Visualize dlt Pipeline Data

Source: https://dlthub.com/docs/intro/pipelines/clubhouse/load-data-with-python-from-clubhouse-to-biganimal

These commands guide users on installing Streamlit and then using the dlt CLI to launch a Streamlit application that visualizes the data loaded by the `clubhouse_pipeline`. This is useful for inspecting the destination contents.

```Bash
pip install streamlit
dlt pipeline clubhouse_pipeline show
```

--------------------------------

### Access Developer Tutorial

Source: https://dlthub.com/docs/intro/pipelines/vimeo/load-data-with-python-from-vimeo-to-biganimal

Provides access to a developer tutorial endpoint, designed to help users learn how to use the Vimeo API in conjunction with the Getting Started guide.

```APIDOC
        # This method, in conjunction with our [Getting Started](https://developer.vimeo.com/api/guides/start) guide, can help you learn how to use the Vimeo API.
        {
            "name": "developer_tutorial",
            "table_name": "tutorial",
            "endpoint": {
                "path": "/tutorial"
            }
        }
```

--------------------------------

### Example dlt config.toml File

Source: https://dlthub.com/docs/intro/pipelines/rest_api/load-data-with-python-from-rest_api-to-yugabyte

Shows the default content of the `config.toml` file, which holds general pipeline configurations like log level and telemetry settings.

```toml
# put your configuration values here

[runtime]
log_level="WARNING"  # the system log level of dlt
# use the dlthub_telemetry setting to enable/disable anonymous usage data reporting, see https://dlthub.com/docs/telemetry
dlthub_telemetry = true
```

--------------------------------

### List Available dlt Sources

Source: https://dlthub.com/docs/intro/plus/getting-started/tutorial

This command lists all currently available sources that can be used with `dlt project init`. It helps users discover supported data sources for their dlt+ projects.

```bash
dlt source list-available
```

--------------------------------

### Install Streamlit and Visualize dlt Pipeline Data

Source: https://dlthub.com/docs/intro/pipelines/adobe_analytics/load-data-with-python-from-adobe_analytics-to-alloydb

These commands guide users to install Streamlit and then use the dlt CLI to launch a Streamlit application that visualizes the data loaded by a specific dlt pipeline in the destination (e.g., AlloyDB).

```bash
# install streamlit
pip install streamlit
# run the streamlit app for your pipeline with the dlt cli:
dlt pipeline adobe_analytics_pipeline show
```

--------------------------------

### Example: Get Bitbucket Issue Import Status with cURL

Source: https://dlthub.com/docs/intro/pipelines/bitbucket/load-data-with-python-from-bitbucket-to-biganimal

This cURL command demonstrates how to query the Bitbucket API for the status of an issue import job. It includes an example of the initial PENDING response and describes the subsequent STARTED and final SUCCESS/FAILURE states.

```Shell
$ curl -u <username> -X GET https://api.bitbucket.org/2.0/repositories/<owner_username>/<repo_slug>/issues/import
```

```JSON
< HTTP/1.1 202 Accepted
{
    "type": "issue_job_status",
    "status": "PENDING",
    "phase": "Attachments",
    "total": 15,
    "count": 0,
    "percent": 0
}
```

--------------------------------

### Main dlt Pipeline Script Example (Python)

Source: https://dlthub.com/docs/intro/pipelines/clubhouse/load-data-with-python-from-clubhouse-to-neondb

Provides an example of the main Python script (`clubhouse_pipeline.py`) generated by `dlt-init-openapi`. It demonstrates how to initialize a dlt pipeline, specify its name, destination (e.g., `duckdb`), dataset, and run it with a source.

```python
import dlt

from clubhouse import clubhouse_source

if __name__ == "__main__":
    pipeline = dlt.pipeline(
        pipeline_name="clubhouse_pipeline",
        destination='duckdb',
        dataset_name="clubhouse_data",
        progress="log",
        export_schema_path="schemas/export"
    )
    source = clubhouse_source()
    info = pipeline.run(source)
    print(info)
```

--------------------------------

### Example dlt secrets.toml File

Source: https://dlthub.com/docs/intro/pipelines/rest_api/load-data-with-python-from-rest_api-to-yugabyte

Displays the default structure of the `secrets.toml` file, used for storing sensitive information like API tokens and database credentials for sources and destinations.

```toml
# put your secret values and credentials here. do not share this file and do not push it to github

[sources.rest_api]
github_token = "github_token" # please set me up!

[destination.postgres]
dataset_name = "dataset_name" # please set me up!

[destination.postgres.credentials]
database = "database" # please set me up!
password = "password" # please set me up!
username = "username" # please set me up!
host = "host" # please set me up!
port = 5432
connect_timeout = 15
```

--------------------------------

### Install dlt Python Library

Source: https://dlthub.com/docs/intro/devel/intro

This snippet provides the basic command to install the `dlt` library using pip. It is recommended to use a clean virtual environment for your experiments to manage dependencies effectively.

```bash
pip install dlt
```

--------------------------------

### Install Streamlit and Visualize dlt Pipeline Data

Source: https://dlthub.com/docs/intro/pipelines/magento/load-data-with-python-from-magento-to-filesystem-aws

These commands guide users to install Streamlit, a Python library for creating web apps, and then use the dlt CLI to launch a Streamlit application that visualizes and allows inspection of the data loaded by the 'magento_pipeline' in its destination.

```bash
# install streamlit
pip install streamlit
# run the streamlit app for your pipeline with the dlt cli:
dlt pipeline magento_pipeline show
```

--------------------------------

### dlt ai setup Command Arguments and Options

Source: https://dlthub.com/docs/intro/devel/reference/command-line-interface

Detailed documentation for the `dlt ai setup` command's positional arguments and options, including their purpose and usage. Files are fetched from `https://github.com/dlt-hub/verified-sources` by default.

```APIDOC
dlt ai setup
  Positional arguments:
    {cursor,continue,cline,claude_desktop} - Specifies the IDE for which to generate configuration.
  Options:
    -h, --help - Show this help message and exit.
    --location LOCATION - Advanced. Specify git url or local path to rules files and config.
    --branch BRANCH - Advanced. Specify git branch to fetch rules files and config.
```

--------------------------------

### MongoDB Connection URL Examples

Source: https://dlthub.com/docs/intro/devel/dlt-ecosystem/verified-sources/mongodb

Provides various connection URL formats for different MongoDB deployment types, including local installations, Docker deployments, MongoDB Atlas, managed cloud services, and more. These examples illustrate the typical structure for connecting to a MongoDB instance.

```APIDOC
MongoDB Connection URL Examples:
  Local installation: "mongodb://dbuser:passwd@host.or.ip:27017"
  Docker: "mongodb://dbuser:passwd@docker.host:27017"
  MongoDB Atlas: "mongodb+srv://dbuser:passwd@cluster.mongodb.net"
  Managed cloud: "mongodb://dbuser:passwd@managed.cloud:27017"
  Configuration tools: "mongodb://dbuser:passwd@config.tool:27017"
  Replica set: "mongodb://dbuser:passwd@replica.set:27017"
  Sharded cluster: "mongodb://dbuser:passwd@shard.cluster:27017"
  Kubernetes: "mongodb://dbuser:passwd@k8s.cluster:27017"
  Manual tarball: "mongodb://dbuser:passwd@tarball.host:27017"
```

--------------------------------

### Display dlt init help

Source: https://dlthub.com/docs/intro/walkthroughs/add-a-verified-source

Shows the help message for the `dlt init` command, providing information on its usage and available options.

```Shell
dlt init --help
```

--------------------------------

### Configure dlt pipeline runtime settings in config.toml

Source: https://dlthub.com/docs/intro/pipelines/rest_api/load-data-with-python-from-rest_api-to-filesystem

Example content for the `config.toml` file, used to set general pipeline configurations such as runtime log level and dlt telemetry settings.

```toml
# put your configuration values here

[runtime]
log_level="WARNING"  # the system log level of dlt
# use the dlthub_telemetry setting to enable/disable anonymous usage data reporting, see https://dlthub.com/docs/telemetry
dlthub_telemetry = true
```

--------------------------------

### dlt Codespaces Demo Environment Setup

Source: https://dlthub.com/docs/intro/pipelines/chargebee/load-data-with-python-from-chargebee-to-neondb

Describes a dlt demo environment available on GitHub Codespaces, guiding users to fork the repository and follow the README for further steps, leveraging the Continue VSCode extension.

```APIDOC
This demo works on codespaces. Codespaces is a development environment available for free to anyone with a Github account. You'll be asked to fork the demo repository and from there the README guides you with further steps.
The demo uses the Continue VSCode extension.
Off to codespaces! (https://github.com/codespaces/new/dlt-hub/dlt-llm-code-playground?ref=create-pipeline)
```

--------------------------------

### dlt ai setup Command-Line Arguments and Options

Source: https://dlthub.com/docs/intro/reference/command-line-interface

Detailed documentation for the arguments and options available with the `dlt ai setup` command, explaining their purpose and usage.

```APIDOC
Positional Arguments:
  {cursor,continue,cline,claude_desktop} - Specifies the AI IDE for which to generate configuration and rules files.

Options:
  -h, --help - Show this help message and exit.
  --location LOCATION - Advanced. Specify a git URL or local path to rules files and configuration.
  --branch BRANCH - Advanced. Specify a git branch to fetch rules files and configuration.
```

--------------------------------

### Install Streamlit and Inspect dlt Pipeline with Streamlit App

Source: https://dlthub.com/docs/intro/pipelines/adobe_analytics/load-data-with-python-from-adobe_analytics-to-cockroachdb

These commands guide you through installing Streamlit, a Python library for creating web apps, and then using the `dlt` CLI to launch a Streamlit application. This app allows for visual inspection of the data loaded into your `CockroachDB` (or other) destination by the `dlt` pipeline.

```bash
# install streamlit
pip install streamlit
# run the streamlit app for your pipeline with the dlt cli:
dlt pipeline adobe_analytics_pipeline show

```

--------------------------------

### Generate dlt Pipeline from OpenAPI Spec and Install Dependencies

Source: https://dlthub.com/docs/intro/pipelines/adobe_analytics/load-data-with-python-from-adobe_analytics-to-cockroachdb

Demonstrates how to generate a `dlt` pipeline for a specific API (e.g., Adobe Analytics) using its OpenAPI specification. It includes options for global data limits and guides through installing the generated pipeline's Python dependencies from `requirements.txt`.

```Bash
# generate pipeline
# NOTE: add_limit adds a global limit, you can remove this later
# NOTE: you will need to select which endpoints to render, you
# can just hit Enter and all will be rendered.
dlt-init-openapi adobe_analytics --url https://raw.githubusercontent.com/dlt-hub/openapi-specs/main/open_api_specs/Business/adobe_analytics.yaml --global-limit 2
cd adobe_analytics_pipeline
# install generated requirements
pip install -r requirements.txt
```

--------------------------------

### dlt Project Folder Structure

Source: https://dlthub.com/docs/intro/pipelines/rest_api/load-data-with-python-from-rest_api-to-yugabyte

Illustrates the standard directory and file structure created by `dlt init`, including configuration files, source-specific files, the main pipeline script, and dependency list.

```text
rest_api_pipeline/
├── .dlt/
│   ├── config.toml          # configs for your pipeline
│   └── secrets.toml         # secrets for your pipeline
├── rest_api/                   # folder with source specific files
│   └── ...
├── rest_api_pipeline.py        # your main pipeline script
├── requirements.txt         # dependencies for your pipeline
└── .gitignore               # ignore files for git (not required)
```

--------------------------------

### Visualize dlt Pipeline Data with Streamlit

Source: https://dlthub.com/docs/intro/pipelines/filesystem-gcs/load-data-with-python-from-filesystem-gcs-to-postgres

These commands guide you through installing Streamlit and then launching a Streamlit application via the `dlt` CLI. This application allows for visual inspection of the data loaded into your PostgreSQL destination by the `filesystem_pipeline`.

```shell
# install streamlit
pip install streamlit
# run the streamlit app for your pipeline with the dlt cli:
dlt pipeline filesystem_pipeline show
```

--------------------------------

### Run dlt Pipeline Script

Source: https://dlthub.com/docs/intro/pipelines/sql_database/load-data-with-python-from-sql_database-to-snowflake

This command executes the main Python script generated by `dlt init`, which orchestrates the data extraction, loading, and transformation process from the configured source to the destination.

```bash
python sql_database_pipeline.py
```

--------------------------------

### List Available dlt Sources

Source: https://dlthub.com/docs/intro/pipelines/sql_database/load-data-with-python-from-sql_database-to-bigquery

Use this dlt CLI command to display a comprehensive list of all verified data sources that can be initialized with dlt, helping users discover available integrations.

```bash
dlt init --list
```

--------------------------------

### Get Revenue Automation Start Date API

Source: https://dlthub.com/docs/intro/pipelines/zuora/load-data-with-python-from-zuora-to-neondb

Describes how to get the revenue automation start date. Request and response field descriptions and sample code are provided.

```APIDOC
{
    "name": "get_revenue_automation_start_date",
    "table_name": "revenue_automation_start_date",
    "endpoint": {
        "path": "/v1/settings/finance/revenue-automation-start-date"
    }
}
```

--------------------------------

### Generated dlt project folder structure

Source: https://dlthub.com/docs/intro/pipelines/google_analytics/load-data-with-python-from-google_analytics-to-neondb

Illustrates the typical directory structure created by the `dlt init` command, showing configuration files, source-specific files, the main pipeline script, and dependencies.

```text
google_analytics_pipeline/
├── .dlt/
│   ├── config.toml          # configs for your pipeline
│   └── secrets.toml         # secrets for your pipeline
├── google_analytics/                   # folder with source specific files
│   └── ...
├── google_analytics_pipeline.py        # your main pipeline script
├── requirements.txt         # dependencies for your pipeline
└── .gitignore               # ignore files for git (not required)
```

--------------------------------

### Generated dlt Pipeline Requirements

Source: https://dlthub.com/docs/intro/pipelines/sql_database/load-data-with-python-from-sql_database-to-snowflake

This `requirements.txt` file lists the Python packages and their minimum versions required for the initialized dlt pipeline, including `sqlalchemy` for database interaction and `dlt` with Snowflake extras.

```python
sqlalchemy>=1.4
dlt[snowflake]>=0.3.5
```

--------------------------------

### Overview of Generated dlt Pipeline Project Structure

Source: https://dlthub.com/docs/intro/pipelines/chargebee/load-data-with-python-from-chargebee-to-alloydb

This illustrates the standard folder and file layout created by `dlt-init-openapi` for a new pipeline. It highlights key directories like `.dlt` for configurations, `rest_api` for the verified source, and the main pipeline script, `chargebee_pipeline.py`.

```text
chargebee_pipeline/
├── .dlt/
│   ├── config.toml          # configs for your pipeline
│   └── secrets.toml         # secrets for your pipeline
├── rest_api/                # The rest api verified source
│   └── ...
├── chargebee/
│   └── __init__.py          # TODO: possibly tweak this file
├── chargebee_pipeline.py        # your main pipeline script
├── requirements.txt         # dependencies for your pipeline
└── .gitignore               # ignore files for git (not required)
```

--------------------------------

### dlt Pipeline Configuration (config.toml)

Source: https://dlthub.com/docs/intro/pipelines/sql_database/load-data-with-python-from-sql_database-to-snowflake

This `config.toml` file contains general runtime settings for the dlt pipeline, such as the system log level and telemetry reporting preferences, allowing users to customize pipeline behavior.

```toml
# put your configuration values here

[runtime]
log_level="WARNING"  # the system log level of dlt
# use the dlthub_telemetry setting to enable/disable anonymous usage data reporting, see https://dlthub.com/docs/telemetry
dlthub_telemetry = true
```

--------------------------------

### Main Execution Block for dlt Pipeline Examples

Source: https://dlthub.com/docs/intro/pipelines/sql_database_mysql/load-data-with-python-from-sql_database_mysql-to-biganimal

Provides the main entry point for running various dlt pipeline examples, including loading selected tables, selecting columns, and loading entire databases.

```python
if __name__ == "__main__":
    # Load selected tables with different settings
    load_select_tables_from_database()

    # load a table and select columns
    # select_columns()

    # load_entire_database()
    # select_with_end_value_and_row_order()

    # Load tables with the standalone table resource
    # load_standalone_table_resource()

    # Load all tables from the database.
    # Warning: The sample database is very large
    # load_entire_database()
```

--------------------------------

### Configure dlt pipeline secrets and credentials in secrets.toml

Source: https://dlthub.com/docs/intro/pipelines/rest_api/load-data-with-python-from-rest_api-to-filesystem

Example content for the `secrets.toml` file, used to store sensitive information like API tokens and cloud credentials for sources (Rest API) and destinations (Filesystem/AWS S3).

```toml
# put your secret values and credentials here. do not share this file and do not push it to github

[sources.rest_api]
github_token = "github_token" # please set me up!

[destination.filesystem]
dataset_name = "dataset_name" # please set me up!
bucket_url = "bucket_url" # please set me up!

[destination.filesystem.credentials]
aws_access_key_id = "aws_access_key_id" # please set me up!
aws_secret_access_key = "aws_secret_access_key" # please set me up!
```

--------------------------------

### dlt Local Pipeline Setup Steps

Source: https://dlthub.com/docs/intro/pipelines/chargebee/load-data-with-python-from-chargebee-to-neondb

An outline of the steps required to set up and run a dlt data loading pipeline locally, covering prerequisites, installation, configuration, code adjustment, initial execution, and result inspection.

```APIDOC
0. Prerequisites
1. Install dlt and dlt-init-openapi
1.1. Tweak chargebee/__init__.py
2. Configuring your source and destination credentials
2.1. Adjust the generated code to your usecase
3. Running your pipeline for the first time
4. Inspecting your load result
5. Next steps to get your pipeline running in production
```

--------------------------------

### Install dlt with Postgres Extras

Source: https://dlthub.com/docs/intro/pipelines/rest_api/load-data-with-python-from-rest_api-to-yugabyte

Installs the dlt library using pip, including the necessary extras for connecting to a Postgres-compatible database like YugabyteDB.

```bash
pip install "dlt[postgres]"
```

--------------------------------

### List Available dlt Sources

Source: https://dlthub.com/docs/intro/devel/walkthroughs/add-a-verified-source

Use this command to display a comprehensive list of all verified data sources that can be initialized with dlt, along with their brief descriptions and capabilities.

```Bash
dlt init --list-sources
```

--------------------------------

### Inspect dlt Pipeline Information via CLI

Source: https://dlthub.com/docs/intro/pipelines/sql_database/load-data-with-python-from-sql_database-to-snowflake

Use the `dlt` command-line interface to view detailed information about a specific pipeline, such as its status and configuration. This command provides an overview of the pipeline's current state.

```bash
dlt pipeline rfam_database info
```

--------------------------------

### Run dlt pipelines with dev and prod profiles

Source: https://dlthub.com/docs/intro/devel/plus/getting-started/tutorial

Executes the `my_pipeline` using both 'dev' and 'prod' profiles, demonstrating how `dlt` applies profile-specific configurations from TOML files.

```Shell
dlt pipeline --profile dev my_pipeline run
dlt pipeline --profile prod my_pipeline run
```

--------------------------------

### Inspect Destination Data with Streamlit and dlt CLI

Source: https://dlthub.com/docs/intro/pipelines/chargebee/load-data-with-python-from-chargebee-to-biganimal

These shell commands guide the user to install Streamlit and then use the dlt CLI to launch a Streamlit application, enabling visual inspection of the data loaded into the pipeline's destination, such as EDB BigAnimal.

```Shell
pip install streamlit
# run the streamlit app for your pipeline with the dlt cli:
dlt pipeline chargebee_pipeline show
```

--------------------------------

### List available dlt sources via CLI

Source: https://dlthub.com/docs/intro/pipelines/sql_database/load-data-with-python-from-sql_database-to-synapse

Executes the `dlt` command-line interface to display a comprehensive list of all pre-built and available data sources that can be used to initialize a new pipeline.

```bash
dlt init --list
```

--------------------------------

### Visualize dlt Pipeline Data with Streamlit (Shell)

Source: https://dlthub.com/docs/intro/pipelines/rest_api/load-data-with-python-from-rest_api-to-filesystem

This shell snippet guides users to install Streamlit, a Python library for creating web apps, and then use the `dlt` CLI to launch a Streamlit-based UI. The command `dlt pipeline rest_api_github show` opens an interactive dashboard for visualizing the data and state of the specified `dlt` pipeline.

```shell
# install streamlit
pip install streamlit
# run the streamlit app for your pipeline with the dlt cli:
dlt pipeline rest_api_github show
```

--------------------------------

### Install dlt+ packaged project locally using uv

Source: https://dlthub.com/docs/intro/devel/plus/getting-started/advanced_tutorial

This command installs the dlt+ packaged project into the current `uv` environment directly from a local file path, making it available for import and use.

```shell
uv pip install /Volumes/my_drive/my_folder
```

--------------------------------

### Install dlt with PostgreSQL Extras

Source: https://dlthub.com/docs/intro/pipelines/sql_database_hana/load-data-with-python-from-sql_database_hana-to-cosmosdb

Installs the dlt library with necessary dependencies for PostgreSQL, enabling database interactions.

```bash
pip install "dlt[postgres]"
```

--------------------------------

### Initialize dlt Pipeline for Zendesk to PostgreSQL

Source: https://dlthub.com/docs/intro/pipelines/zendesk/load-data-with-python-from-zendesk-to-postgres

Provides a sequence of shell commands to set up a new `dlt` project. It creates a directory, initializes a pipeline configured for `Zendesk` as a source and `PostgreSQL` as a destination, and then installs all required Python dependencies from the generated `requirements.txt` file.

```bash
mkdir zendesk_pipeline
cd zendesk_pipeline
dlt init zendesk postgres
pip install -r requirements.txt
```

--------------------------------

### Access Vimeo API Developer Tutorial

Source: https://dlthub.com/docs/intro/pipelines/vimeo/load-data-with-python-from-vimeo-to-neondb

This method, in conjunction with the Getting Started guide, can help you learn how to use the Vimeo API.

```Python
{
            "name": "developer_tutorial",
            "table_name": "tutorial",
            "endpoint": {
                "path": "/tutorial"
            }
        }
```

```APIDOC
Method: developer_tutorial
Table Name: tutorial
Path: /tutorial
Description: Provides access to a tutorial for learning the Vimeo API.
Parameters: None
```

--------------------------------

### Generated dlt Project Folder Structure

Source: https://dlthub.com/docs/intro/pipelines/google_analytics/load-data-with-python-from-google_analytics-to-duckdb

Illustrates the directory and file structure created by the `dlt init` command, showing the location of configuration files, source-specific files, the main pipeline script, and dependencies.

```Text
google_analytics_pipeline/
├── .dlt/
│   ├── config.toml          # configs for your pipeline
│   └── secrets.toml         # secrets for your pipeline
├── google_analytics/                   # folder with source specific files
│   └── ...
├── google_analytics_pipeline.py        # your main pipeline script
├── requirements.txt         # dependencies for your pipeline
└── .gitignore               # ignore files for git (not required)
```

--------------------------------

### Initialize an Empty dlt Project

Source: https://dlthub.com/docs/intro/devel/plus/getting-started/tutorial

This command creates a new dlt project in the current directory without any pre-configured sources, destinations, datasets, or pipelines.

```Shell
dlt project init
```

--------------------------------

### Install Pipeline Dependencies

Source: https://dlthub.com/docs/intro/devel/dlt-ecosystem/verified-sources/google_ads

Before running the pipeline, ensure all required Python packages are installed. This command reads dependencies from `requirements.txt` and installs them using pip.

```shell
pip install -r requirements.txt
```

--------------------------------

### Example dlt Pipeline Script

Source: https://dlthub.com/docs/intro/pipelines/crypt_api/load-data-with-python-from-crypt_api-to-filesystem-az

This Python script shows a basic `dlt` pipeline setup. It initializes a pipeline named `crypt_api_pipeline` with `duckdb` as the destination, runs a `crypt_api_source`, and prints the load information. This serves as a starting point for custom pipeline development.

```python
import dlt

from crypt_api import crypt_api_source

if __name__ == "__main__":
    pipeline = dlt.pipeline(
        pipeline_name="crypt_api_pipeline",
        destination='duckdb',
        dataset_name="crypt_api_data",
        progress="log",
        export_schema_path="schemas/export"
    )
    source = crypt_api_source()
    info = pipeline.run(source)
    print(info)

```

--------------------------------

### dlt Pipeline Project Folder Structure

Source: https://dlthub.com/docs/intro/pipelines/sql_database_mssql/load-data-with-python-from-sql_database_mssql-to-filesystem-gcs

Illustrates the typical directory structure of a `dlt` project after initialization, showing configuration files, source-specific files, the main pipeline script, and dependency list.

```text
sql_database_mssql_pipeline/
├── .dlt/
│   ├── config.toml          # configs for your pipeline
│   └── secrets.toml         # secrets for your pipeline
├── sql_database/                   # folder with source specific files
│   └── ...
├── sql_database_pipeline.py        # your main pipeline script
├── requirements.txt         # dependencies for your pipeline
└── .gitignore               # ignore files for git (not required)

```

--------------------------------

### Inspect dlt pipeline destination with Streamlit

Source: https://dlthub.com/docs/intro/pipelines/sql_database_mssql/load-data-with-python-from-sql_database_mssql-to-timescale

These commands guide the user on how to install Streamlit and then launch a Streamlit application via the dlt CLI. This allows for visual inspection of the data loaded into the pipeline's destination, such as a Timescale database.

```bash
pip install streamlit
dlt pipeline sql_database show
```

--------------------------------

### Initialize dlt Project for SQL Database

Source: https://dlthub.com/docs/intro/devel/dlt-ecosystem/verified-sources/sql_database/setup

Use `dlt init` to create a new project, configuring it for a SQL database source and a specified destination like DuckDB. This command generates essential project files and sets up the basic pipeline structure.

```bash
dlt init sql_database duckdb
```

--------------------------------

### Simplest dlt Source Configuration

Source: https://dlthub.com/docs/intro/devel/general-usage/credentials/setup

Demonstrates basic configuration for a dlt source using a single API key, showing both TOML file content and environment variable export methods for straightforward pipelines.

```TOML
api_key="some_value"
```

```Shell
export API_KEY="some_value"
```

--------------------------------

### Install dlt-init-openapi CLI Tool

Source: https://dlthub.com/docs/intro/pipelines/magento/load-data-with-python-from-magento-to-timescale

Installs the `dlt-init-openapi` command-line interface tool, which is essential for generating dlt pipelines from OpenAPI specifications. This tool is installed via pip.

```bash
pip install dlt-init-openapi
```

--------------------------------

### Configure Simple Source with TOML and Environment Variables

Source: https://dlthub.com/docs/intro/general-usage/credentials/config_providers

Demonstrates the simplest way to configure a source using a direct key-value pair in TOML or an environment variable. This approach is suitable for straightforward pipelines with minimal configuration.

```TOML
api_key="some_value"
```

```Shell
export API_KEY="some_value"
```

--------------------------------

### Verify dlt+ License Installation

Source: https://dlthub.com/docs/intro/devel/plus/getting-started/installation

This command-line snippet shows how to verify that the dlt+ license has been correctly installed and is valid. Running `dlt license show` will display the license status.

```bash
$ dlt license show
```

--------------------------------

### Initialize dlt pipeline for Filesystem to Postgres

Source: https://dlthub.com/docs/intro/pipelines/filesystem-local/load-data-with-python-from-filesystem-local-to-neondb

These commands create a new project directory, navigate into it, initialize a dlt pipeline connecting a filesystem source to a Postgres destination, and then install all required dependencies listed in `requirements.txt`.

```bash
mkdir filesystem_local_pipeline
cd filesystem_local_pipeline
dlt init filesystem postgres
pip install -r requirements.txt
```

--------------------------------

### Install dlt library with Postgres extras

Source: https://dlthub.com/docs/intro/pipelines/filesystem-local/load-data-with-python-from-filesystem-local-to-neondb

This command installs the dlt library along with the necessary extras for PostgreSQL, enabling interaction with Neon Serverless Postgres. It's the first step to set up a dlt pipeline.

```bash
pip install "dlt[postgres]"
```

--------------------------------

### Install dlt with PostgreSQL Extras

Source: https://dlthub.com/docs/intro/pipelines/stripe_analytics/load-data-with-python-from-stripe_analytics-to-timescale

Installs the `dlt` library along with the necessary dependencies for connecting to PostgreSQL, which is used by TimescaleDB.

```bash
pip install "dlt[postgres]"
```

--------------------------------

### dlt Project Initialization Success Output and Dependency Instructions

Source: https://dlthub.com/docs/intro/devel/walkthroughs/add-a-verified-source

This snippet displays the console output after a successful dlt project initialization, providing crucial next steps. It includes instructions for adding credentials and installing required dependencies using various package managers like pip, poetry, and uv.

```plaintext
Verified source pipedrive was added to your project!
* See the usage examples and code snippets to copy from pipedrive_pipeline.py
* Add credentials for bigquery and other secrets in .dlt/secrets.toml
* Add the required dependencies to pyproject.toml:
  dlt[bigquery]>=0.3.1
  If the dlt dependency is already added, make sure you install the extra for bigquery to it
  If you are using poetry you may issue the following command:
  poetry add dlt -E bigquery
  For uv, use:
  uv add "dlt[bigquery]"

* Read https://dlthub.com/docs/walkthroughs/create-a-pipeline for more information
```

--------------------------------

### Install dlt with Postgres Extras

Source: https://dlthub.com/docs/intro/pipelines/sql_database_oracle/load-data-with-python-from-sql_database_oracle-to-biganimal

Installs the dlt library with the necessary extras for connecting to a Postgres-compatible destination like EDB BigAnimal.

```bash
pip install "dlt[postgres]"
```

--------------------------------

### dlt Pipeline Project Folder Structure

Source: https://dlthub.com/docs/intro/pipelines/stripe_analytics/load-data-with-python-from-stripe_analytics-to-timescale

Illustrates the typical directory layout of a `dlt` project after initialization, showing key configuration files, source-specific files, the main pipeline script, and dependency list.

```text
stripe_analytics_pipeline/
├── .dlt/
│   ├── config.toml          # configs for your pipeline
│   └── secrets.toml         # secrets for your pipeline
├── stripe_analytics/                   # folder with source specific files
│   └── ...
├── stripe_analytics_pipeline.py        # your main pipeline script
├── requirements.txt         # dependencies for your pipeline
└── .gitignore               # ignore files for git (not required)
```

--------------------------------

### Visualize dlt Pipeline State with Streamlit

Source: https://dlthub.com/docs/intro/pipelines/filesystem-az/load-data-with-python-from-filesystem-az-to-biganimal

These commands guide you through installing Streamlit and then using the dlt CLI to launch a Streamlit application. This application allows for a visual inspection of your pipeline's state and the contents of your destination database.

```shell
# install streamlit
pip install streamlit
# run the streamlit app for your pipeline with the dlt cli:
dlt pipeline filesystem_pipeline show
```