### Run dltHub Quickstart

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/hub/getting-started/introduction.md

Execute this command to start a guided first experience with dltHub, which includes setting up a local workspace, installing dependencies, logging into the platform, and running a sample pipeline.

```sh
uvx dlthub-start@latest
```

--------------------------------

### Start dltHub with pipx

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/hub/getting-started/installation.md

Run this command if you don't have `uv` installed; `pipx` will offer to install it for you.

```sh
pipx run dlthub-start
```

--------------------------------

### Quick start: Load data into Lance

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/destinations/lance.md

A basic example demonstrating how to initialize a dlt pipeline with the Lance destination and load a list of dictionaries into a Lance table.

```python
import dlt

movies = [
    {"id": 1, "title": "Blade Runner", "year": 1982},
    {"id": 2, "title": "Ghost in the Shell", "year": 1995},
    {"id": 3, "title": "The Matrix", "year": 1999},
]

pipeline = dlt.pipeline(
    pipeline_name="movies",
    destination="lance",
    dataset_name="movies_db",
)

info = pipeline.run(movies, table_name="movies")
```

--------------------------------

### Quick Start: Access Dataset and Load to Pandas/PyArrow

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/general-usage/dataset-access/dataset.md

Demonstrates how to retrieve data from a dlt pipeline and load it into a Pandas DataFrame or a PyArrow Table. This is a comprehensive example for immediate use.

```python
import dlt
import pandas as pd
from pyarrow import Table

# Assuming 'pipeline' is an initialized dlt.Pipeline object
# For example:
# pipeline = dlt.load_resource(...) or dlt.load_package(...) etc.
# For demonstration purposes, let's assume a pipeline object exists
# In a real scenario, you would have run a pipeline first.

# Mock pipeline for demonstration if not run
# In a real scenario, this would be your actual pipeline object
class MockPipeline:
    def __init__(self):
        self.dataset_name = "my_dataset"

    def dataset(self):
        # Mock dataset object
        return MockDataset(self.dataset_name)

class MockDataset:
    def __init__(self, name):
        self.name = name

    def __call__(self, table_name=None):
        # Mock relation object
        return MockRelation(self.name, table_name)

class MockRelation:
    def __init__(self, dataset_name, table_name):
        self.dataset_name = dataset_name
        self.table_name = table_name

    def to_pandas(self):
        print(f"Fetching data from dataset '{self.dataset_name}', table '{self.table_name or 'all'}' to Pandas...")
        # Return a dummy Pandas DataFrame for demonstration
        return pd.DataFrame({
            'col1': [1, 2, 3],
            'col2': ['A', 'B', 'C']
        })

    def to_arrow(self):
        print(f"Fetching data from dataset '{self.dataset_name}', table '{self.table_name or 'all'}' to PyArrow...")
        # Return a dummy PyArrow Table for demonstration
        return Table.from_pandas(self.to_pandas())

# Use the mock pipeline if no real pipeline is available
try:
    pipeline # Check if pipeline object exists
except NameError:
    pipeline = MockPipeline()

# Access the dataset
my_dataset = pipeline.dataset()

# Retrieve data as a Pandas DataFrame
print("\n--- Loading to Pandas DataFrame ---")
pandas_df = my_dataset.to_pandas() # Access all tables or specify a table name
print(pandas_df)

# Retrieve data as a PyArrow Table
print("\n--- Loading to PyArrow Table ---")
arrow_table = my_dataset.to_arrow() # Access all tables or specify a table name
print(arrow_table)

# Example accessing a specific table
print("\n--- Accessing a specific table 'my_table' ---")
specific_table_df = my_dataset('my_table').to_pandas()
print(specific_table_df)

```

--------------------------------

### Basic Shopify Data Load

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/verified-sources/shopify.md

Example of how to configure a dlt pipeline and load data for 'products', 'orders', and 'customers' from a specified start date.

```APIDOC
## Create your own pipeline

### Example: Load specific resources

1.  **Configure the pipeline**: Specify the pipeline name, destination, and dataset.
    ```python
    pipeline = dlt.pipeline(
        pipeline_name="shopify",
        destination="duckdb",
        dataset_name="shopify_data"
    )
    ```

2.  **Load data**: Specify the resources to load and the start date.
    ```python
    # Add your desired resources to the list...
    resources = ["products", "orders", "customers"]

    load_data = shopify_source(start_date="2023-01-01").with_resources(*resources)
    load_info = pipeline.run(load_data)
    print(load_info)
    ```
```

--------------------------------

### MS SQL Full and Incremental Load Example

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/hub/ingestion/ms-sql.md

This Python script demonstrates how to perform an initial full load of a table from MS SQL Server and then set up and run incremental loads using change tracking. It includes setup for DLT pipeline, explicit database connection, and applying hints for DLT system columns.

```python
import dlt

from sqlalchemy import create_engine

from dlt.sources.sql_database import sql_table
from dlthub.sources.mssql import (
    create_change_tracking_table,
    get_current_change_tracking_version,
)


def single_table_initial_load(connection_url: str, schema_name: str, table_name: str) -> None:
    """Performs an initial full load and sets up tracking version and incremental loads"""
    # Create a new pipeline
    pipeline = dlt.pipeline(
        pipeline_name=f"{schema_name}_{table_name}_sync",
        destination="duckdb",
        dataset_name=schema_name,
    )

    # Explicit database connection
    engine = create_engine(connection_url, isolation_level="SNAPSHOT")

    # Initial full load
    initial_resource = sql_table(
        credentials=engine,
        schema=schema_name,
        table=table_name,
        reflection_level="full",
        write_disposition="merge",
    )

    # Get the current tracking version before you run the pipeline to make sure
    # you do not miss any records
    tracking_version = get_current_change_tracking_version(engine)
    print(f"will track from: {tracking_version}")  # noqa

    # Apply hints to create _DLT_DELETED and _DLT_SYS_CHANGE_VERSION columns on the initial load
    # This is an optional step
    initial_resource.apply_hints(
        columns=[
            {"name": "_dlt_sys_change_version", "data_type": "bigint"},
            {"name": "_dlt_deleted", "data_type": "text", "precision": 10},
        ]
    )

    # Run the pipeline for the initial load
    # NOTE: we always drop data and state from the destination on initial load
    print(pipeline.run(initial_resource, refresh="drop_resources"))  # noqa

    # Incremental loading resource
    incremental_resource = create_change_tracking_table(
        credentials=engine,
        table=table_name,
        schema=schema_name,
        initial_tracking_version=tracking_version,
    )

    # Run the pipeline for incremental load
    print(pipeline.run(incremental_resource))  # noqa


def single_table_incremental_load(connection_url: str, schema_name: str, table_name: str) -> None:
    """Continues loading incrementally"""
    # Make sure you use the same pipeline and dataset names in order to continue incremental
    # loading.
    pipeline = dlt.pipeline(
        pipeline_name=f"{schema_name}_{table_name}_sync",
        destination="duckdb",
        dataset_name=schema_name,
    )

    engine = create_engine(connection_url, isolation_level="SNAPSHOT")
    # We do not need to pass the tracking version anymore
    incremental_resource = create_change_tracking_table(
        credentials=engine,
        table=table_name,
        schema=schema_name,
    )
    print(pipeline.run(incremental_resource))  # noqa


if __name__ == "__main__":
    # Change Tracking already enabled here
    test_db = "my_database83ed099d2d98a3ccfa4beae006eea44c"
    # A test run with a local mssql instance
    connection_url = (
        f"mssql+pyodbc://sa:Strong%21Passw0rd@localhost:1433/{test_db}"
        "?driver=ODBC+Driver+18+for+SQL+Server&TrustServerCertificate=yes"
    )
    single_table_initial_load(
        connection_url,
        "my_dlt_source",
        "app_user",
    )
    single_table_incremental_load(
        connection_url,
        "my_dlt_source",
        "app_user",
    )

```

--------------------------------

### Initialize a Custom Pipeline Example

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/hub/ingestion/init.md

Example of initializing a custom pipeline named 'my_github_pipeline' with DuckDB as the destination.

```sh
dlthub pipeline init my_github_pipeline duckdb
```

--------------------------------

### Install dlt with Qdrant support

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/destinations/qdrant.md

Install the dlt library with the necessary qdrant extra to enable the destination.

```bash
pip install "dlt[qdrant]"
```

--------------------------------

### Initial Data for Schema Example

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/general-usage/schema.md

Example of initial data with an integer ID, used to demonstrate schema creation.

```python
data = [
  {"id": 1, "human_name": "Alice"}
]
```

--------------------------------

### Install dlt with Filesystem Dependencies

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/destinations/filesystem.md

Install the dlt library with the necessary filesystem dependencies, including s3fs and botocore.

```sh
pip install "dlt[filesystem]"
```

--------------------------------

### Install Preview Tools and Show Data

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/verified-sources/openapi-generator.md

Install `pandas` and `marimo` for data previewing. Then, use `dlt pipeline show` to view the loaded data within the workspace dashboard.

```bash
pip install pandas marimo
dlt pipeline pokemon_pipeline show
```

--------------------------------

### Install Transformations Toolkit

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/hub/transformations/explore-and-transform.md

Install the transformations toolkit using uv. Ensure uv is set up and the coding assistant is initialized if this is a fresh project.

```sh
uv run dlthub ai toolkit install transformations
```

--------------------------------

### Install Specific dlt Version

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/reference/installation.md

Installs a specific version of dlt, for example, any version prior to 0.5.0.

```sh
uv pip install "dlt<0.5.0"
```

--------------------------------

### Install Marimo for Data Exploration

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/tutorial/sql-database.md

Install the marimo library to enable the dlt browser dashboard for data exploration.

```sh
pip install marimo
```

--------------------------------

### Install dlt with DuckDB

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/education/dlt-fundamentals-course/lesson_5_write_disposition_and_incremental_loading.ipynb

Installs the dlt library with DuckDB support, which is a lightweight, in-process analytical data warehouse. This is often a prerequisite for running dlt examples locally.

```bash
#!/bin/bash
!pip install "dlt[duckdb]"
```

--------------------------------

### Comprehensive Databricks Configuration Example

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/destinations/databricks.md

A complete example demonstrating partitioning, table properties, documentation, and column hints for Databricks tables using `databricks_adapter`.

```python
import dlt
from dlt.destinations.adapters import databricks_adapter

@dlt.resource(
    columns=[
        {"name": "transaction_date", "data_type": "date"},
        {"name": "customer_id", "data_type": "bigint"},
        {"name": "amount", "data_type": "decimal"},
        {"name": "region", "data_type": "text"},
        {"name": "year", "data_type": "bigint"},
        {"name": "month", "data_type": "bigint"}
    ]
)
def transactions():
    yield from load_transactions()

# Apply comprehensive Databricks optimizations
databricks_adapter(
    transactions,

    # Partitioning for efficient data organization
    partition=["year", "month"],

    # Note: Cannot use both partition and cluster together in Databricks
    # cluster=["region", "customer_id"],  # Would need separate table without partitioning

    # Table format (DELTA is default)
    table_format="DELTA",

    # Table properties for performance tuning
    table_properties={
        "delta.appendOnly": False,
        "delta.dataSkippingStatsColumns": "transaction_date,amount,customer_id",
        "delta.autoOptimize.optimizeWrite": True,
        "delta.targetFileSize": "128mb",
        "custom.owner": "finance-team"
    },

    # Table documentation
    table_comment="Financial transactions with optimized storage",
    table_tags=["financial", {"compliance": "sox"}],

    # Column documentation
    column_hints={
        "customer_id": {
            "column_comment": "Unique customer identifier",
            "column_tags": ["pii"]
        },
        "amount": {
            "column_comment": "Transaction amount in USD"
        }
    }
)
```

--------------------------------

### Run the Pipeline

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/verified-sources/workable.md

Instructions on how to install dependencies and run the Workable pipeline.

```APIDOC
## Run the pipeline

1. Before running the pipeline, ensure that you have installed all the necessary dependencies by
   running the command:

   ```sh
   pip install -r requirements.txt
   ```

1. You're now ready to run the pipeline! To get started, run the following command:

   ```sh
   python workable_pipeline.py
   ```

1. Once the pipeline has finished running, you can verify that everything loaded correctly by using
   the following command:

   ```sh
   dlt pipeline <pipeline_name> show
   ```

   For example, the `pipeline_name` for the above pipeline example is `workable`, you may also use
   any custom name instead.

For more information, read the guide on [how to run a pipeline](../../walkthroughs/run-a-pipeline).
```

--------------------------------

### dlt Credential Lookup Order Example

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/general-usage/credentials/setup.md

Illustrates the order in which dlt searches for credential values for a source function, starting with the most specific path.

```python
# module: notion.py

@dlt.source
def notion_databases(api_key: str = dlt.secrets.value):
    pass
```

--------------------------------

### Initialize dlt Project with Zendesk Source

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/walkthroughs/zendesk-weaviate.md

Initializes a dlt project by downloading and installing the verified Zendesk source.

```sh
dlt init zendesk weaviate
```

--------------------------------

### JSONResponseCursorPaginator with Query Parameters

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/verified-sources/rest_api/advanced.md

Use JSONResponseCursorPaginator for APIs that return a cursor for the next page. This example shows pagination using GET requests with query parameters.

```python
client = RESTClient(
    base_url="https://api.example.com",
    paginator=JSONResponseCursorPaginator(
        cursor_path="cursors.next",
        cursor_param="cursor"
    )
)
```

--------------------------------

### Load Example Data for Transformations

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/hub/transformations/index.md

This snippet shows how to load example data, which is a prerequisite for demonstrating transformations. It assumes a 'fruitshop' dataset is available.

```python
import dlt

@dlt.resource(write_disposition="load")
def fruitshop():
    yield {"name": "apple", "price": 1.0}
    yield {"name": "banana", "price": 0.5}
    yield {"name": "orange", "price": 0.75}

@dlt.resource(write_disposition="load")
def customers():
    yield {"name": "Alice", "city": "New York"}
    yield {"name": "Bob", "city": "Los Angeles"}
    yield {"name": "Charlie", "city": "New York"}
    yield {"name": "David", "city": "Chicago"}
    yield {"name": "Eve", "city": "Los Angeles"}

```

--------------------------------

### Apply BigQuery Hints to a Resource

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/destinations/bigquery.md

Use the bigquery_adapter to specify partitioning and clustering for a resource. This example also shows how to apply a table description and configure streaming inserts.

```python
import dlt
from dlt.destinations.adapters import bigquery_adapter


@dlt.resource(
    columns=[
        {"name": "event_date", "data_type": "date"},
        {"name": "user_id", "data_type": "bigint"},
        # Other columns.
    ]
)
def event_data():
    yield from [
        {"event_date": datetime.date.today() + datetime.timedelta(days=i)} for i in range(100)
    ]


# Apply column options.
bigquery_adapter(
    event_data, partition="event_date", cluster=["event_date", "user_id"]
)

# Apply table level options.
bigquery_adapter(event_data, table_description="Dummy event data.")

# Load data in "streaming insert" mode (only available with
# write_disposition=\"append\").
bigquery_adapter(event_data, insert_api="streaming")
```

--------------------------------

### Get Last Visits Resource

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/verified-sources/matomo.md

Retrieves site visits within a specified timeframe. If a start date is given, it begins from that date. If not, it retrieves all visits up until now. This is an incremental resource.

```APIDOC
## Resource `get_last_visits`

This function retrieves site visits within a specified timeframe. If a start date is given, it begins from that date. If not, it retrieves all visits up until now.

```python
@dlt.resource(
    name="visits", write_disposition="append", primary_key="idVisit", selected=True
)
def get_last_visits(
    client: MatomoAPIClient,
    site_id: int,
    last_date: dlt.sources.incremental[float],
    visit_timeout_seconds: int = 1800,
    visit_max_duration_seconds: int = 3600,
    rows_per_page: int = 2000,
) -> Iterator[TDataItem]:
   ...
```

### Parameters

*   `client` (MatomoAPIClient): Interface for Matomo API calls.
*   `site_id` (int): Unique ID for each Matomo site.
*   `last_date` (dlt.sources.incremental[float]): Last resource load date, if it exists.
*   `visit_timeout_seconds` (int): Time (in seconds) until a session is inactive and deemed closed. Default: 1800.
*   `visit_max_duration_seconds` (int): Maximum duration (in seconds) of a visit before closure. Default: 3600.
*   `rows_per_page` (int): Number of rows on each page.

:::note
This is an [incremental](../../general-usage/incremental-loading) resource method and loads the "last_date" from the state of the last pipeline run.
:::
```

--------------------------------

### Initialize DLT Documentation Environment

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/README.md

Commands to install necessary Python and Node.js dependencies required for building and maintaining the DLT documentation website.

```bash
make dev
cd website && npm install
```

--------------------------------

### Deploy pipedrive pipeline with Airflow Composer

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/walkthroughs/deploy-a-pipeline/deploy-with-airflow-composer.md

Example of initializing deployment for a specific pipeline (pipedrive) with Airflow Composer. This generates deployment files and provides instructions for credential setup.

```sh
dlt deploy pipedrive_pipeline.py airflow-composer
```

--------------------------------

### Run dbt standalone with DuckDB profile

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/transformations/dbt/dbt.md

Execute dbt transformations independently of a dlt pipeline. This example assumes dbt is installed and a `profile.yml` is present. It uses a DuckDB profile configuration.

```python
# This snippet is defined in dbt-snippets.py and not directly in the markdown.
# It is intended to be run standalone.
# Example usage:
# RUNTIME__LOG_LEVEL=DEBUG python dbt_standalone.py
pass
```

--------------------------------

### Drop resources using regex patterns

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/reference/command-line-interface.md

You can use regular expressions to select resources for dropping. Prepend the resource name with `re:` to indicate a regex pattern. This example drops all resources starting with 'repo'.

```sh
dlt pipeline github_events drop "re:^repo"
```

--------------------------------

### Example: Generate Fact Table for 'orders'

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/hub/transformations/dbt-transformations.md

This command demonstrates generating a fact table for the 'orders' table within the 'example_shop' pipeline.

```sh
dlthub dbt generate example_shop --fact orders
```

--------------------------------

### Create Project and Virtual Environment

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/walkthroughs/zendesk-weaviate.md

Sets up a new project directory and initializes a Python virtual environment for managing dependencies.

```sh
mkdir zendesk-weaviate
cd zendesk-weaviate
python -m venv venv
source venv/bin/activate
```

--------------------------------

### Load Workable Data from a Specific Date

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/verified-sources/workable.md

Load data starting from a specified date, including dependent endpoints by setting `load_details` to True. This example loads data from January 1, 2022.

```python
load_data = workable_source(start_date=pendulum.DateTime(2022, 1, 1), load_details=True)
load_info = pipeline.run(load_data)
print(load_info)
```

--------------------------------

### Partitioning Iceberg Tables with Athena Adapter

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/destinations/athena.md

Example of using the `athena_adapter` to partition an Iceberg table by category and the month of a date column. This setup is useful for optimizing queries on large datasets in Athena.

```python
from datetime import date

import dlt
from dlt.destinations.adapters import athena_partition, athena_adapter

data_items = [
    (1, "A", date(2021, 1, 1)),
    (2, "A", date(2021, 1, 2)),
    (3, "A", date(2021, 1, 3)),
    (4, "A", date(2021, 2, 1)),
    (5, "A", date(2021, 2, 2)),
    (6, "B", date(2021, 1, 1)),
    (7, "B", date(2021, 1, 2)),
    (8, "B", date(2021, 1, 3)),
    (9, "B", date(2021, 2, 1)),
    (10, "B", date(2021, 3, 2)),
]

@dlt.resource(table_format="iceberg")
def partitioned_data():
    yield [{"id": i, "category": c, "created_at": d} for i, c, d in data_items]

# Add partitioning hints to the table
athena_adapter(
    partitioned_data,
    partition=[
        # Partition per category and month
        "category",
        athena_partition.month("created_at"),
    ],
)


pipeline = dlt.pipeline("athena_example")
pipeline.run(partitioned_data)
```

--------------------------------

### Create Your Own Pipeline

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/verified-sources/personio.md

Example of how to create and run a dlt pipeline using the Personio source.

```APIDOC
## Customization

### Create your own pipeline

If you wish to create your own pipelines, you can leverage source and resource methods from this
verified source.

1. Configure the [pipeline](../../general-usage/pipeline) by specifying the pipeline name, destination, and dataset as follows:

   ```py
   pipeline = dlt.pipeline(
      pipeline_name="personio",  # Use a custom name if desired
      destination="duckdb",  # Choose the appropriate destination (e.g., duckdb, redshift, post)
      dataset_name="personio_data"  # Use a custom name if desired
   )
   ```

1. To load employee data:

   ```py
   load_data = personio_source().with_resources("employees")
   print(pipeline.run(load_data))
   ```

1. To load data from all supported endpoints:

   ```py
   load_data = personio_source()
   print(pipeline.run(load_data))
   ```
```

--------------------------------

### NewsAPI Authentication and Request with dlt

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/education/dlt-advanced-course/lesson_1_custom_sources_restapi_source_and_restclient.ipynb

Demonstrates how to authenticate with the NewsAPI using an API key and make a GET request for 'everything' articles about Python. It utilizes dlt's RESTClient and APIKeyAuth for secure and simplified request setup.

```python
import os
from dlt.sources.helpers.rest_client import RESTClient
from dlt.sources.helpers.rest_client.auth import APIKeyAuth
from google.colab import userdata

api_key = userdata.get("NEWS_API_KEY")

news_api_client = RESTClient(
    base_url="https://newsapi.org/v2/",
    auth=APIKeyAuth(name="apiKey", api_key=api_key, location="query"),
)

response = news_api_client.get("everything", params={"q": "python", "page": 1})
print(response.json())
```

--------------------------------

### Initialize a Custom Pipeline (Manual)

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/hub/ingestion/init.md

Scaffolds a pipeline template for manual setup using a specified source name and DuckDB as the destination.

```sh
dlthub pipeline init {source_name} duckdb
```

--------------------------------

### Install Data Exploration and DLTHub Platform Toolkits

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/hub/ingestion/rest-api-source.md

Install the 'data-exploration' toolkit for interactive data analysis and the 'dlthub-platform' toolkit for production deployment, scheduling, and monitoring.

```bash
uv run dlthub ai toolkit install data-exploration
```

```bash
uv run dlthub ai toolkit install dlthub-platform
```

--------------------------------

### Drop Tables Using Regex for Resource Selection

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/hub/command-line-interface.md

This command allows you to drop tables by specifying a regex pattern to select resources. Prepend the resource name with 're:' to indicate a regex pattern. This example drops all resources starting with 'repo'.

```sh
dlt pipeline github_events drop "re:^repo"
```

--------------------------------

### Install Dependencies and Activate Environment

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/hub/getting-started/platform-tutorial.md

Install project dependencies using uv and activate the virtual environment. This ensures all necessary packages, including dlt[hub], are available.

```sh
cd jaffle_shop_workspace
uv sync
source .venv/bin/activate
```

--------------------------------

### Update and Deduplicate Issues with Merge Disposition

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/tutorial/load-data-from-an-api.md

Combine incremental loading with the 'merge' write disposition to always get fresh content of issues. This example uses a 'primary_key' to identify issues and tracks the 'updated_at' field to load issues modified since the last run.

```python
import dlt
from dlt.sources.incremental import incremental

@dlt.resource(write_disposition="merge", primary_key="id")
def github_issues_merge(owner: str = "dlt-hub", repo: str = "dlt") -> dlt.typing.Iterable[dlt.typing.TDict]:
    # Use incremental to track the 'updated_at' field
    # This will filter issues updated since the last pipeline run
    issues_to_load = incremental(key="updated_at")

    # Request issues ordered by update date (descending)
    # Use the 'since' parameter to filter issues updated after a specific date
    # 'updated_at.last_value' holds the last 'updated_at' value from the previous run
    for issue in issues_to_load.get_new_issues(owner=owner, repo=repo, since=issues_to_load.updated_at.last_value):
        yield issue

# To run this pipeline:
# pipeline = dlt.pipeline(pipeline_name='github_issues_merge', destination='duckdb')
# data = github_issues_merge()
# info = pipeline.run(data)
# print(info)
```

--------------------------------

### Access and Select Resources from a Source

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/general-usage/source.md

Demonstrates how to access all available resources in a source and how to select specific resources (e.g., 'companies', 'deals') for loading using the `with_resources` method.

```python
from hubspot import hubspot

source = hubspot()
# "resources" is a dictionary with all resources available, the key is the resource name
print(source.resources.keys())  # print names of all resources
# print resources that are selected to load
print(source.resources.selected.keys())
# load only "companies" and "deals" using the "with_resources" convenience method
pipeline.run(source.with_resources("companies", "deals"))
```

--------------------------------

### Install project requirements

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/destinations/dremio.md

Install project dependencies, including those for Dremio and S3, by running 'pip install -r requirements.txt' or directly installing the dlt package with extras.

```sh
pip install -r requirements.txt
```

```sh
pip install "dlt[dremio,s3]"
```

--------------------------------

### Basic Iceberg Partitioning with iceberg_adapter

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/destinations/iceberg.md

Demonstrates partitioning an Iceberg table by a string column and by the month of a date column using the `iceberg_adapter` and `iceberg_partition` helpers.

```python
from datetime import date

import dlt
from dlt.destinations.adapters import iceberg_adapter, iceberg_partition

data_items = [
    {"id": 1, "category": "A", "created_at": date(2025, 1, 1)},
    {"id": 2, "category": "A", "created_at": date(2025, 1, 15)},
    {"id": 3, "category": "B", "created_at": date(2025, 2, 1)},
]

@dlt.resource(table_format="iceberg")
def events():
    yield data_items

# Partition by category and month of created_at
iceberg_adapter(
    events,
    partition=[
        "category",  # identity partition (shorthand)
        iceberg_partition.month("created_at"),
    ],
)

pipeline = dlt.pipeline("iceberg_example", destination="filesystem")
pipeline.run(events)
```

--------------------------------

### Install SQLAlchemy Dependencies

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/destinations/sqlalchemy.md

Install the necessary dependencies for SQLAlchemy, either by installing from requirements.txt or directly.

```sh
pip install -r requirements.txt
```

```sh
pip install "dlt[sqlalchemy]"
```

--------------------------------

### Configure ClickHouse Sorting and Partitioning with Column Names

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/destinations/clickhouse.md

Example of using the `clickhouse_adapter` with a sequence of column names for `sort` and `partition` parameters. This is the recommended approach when no column transformations are needed.

```python
clickhouse_adapter(
   my_resource,
   sort=["timestamp", "street"],
   partition=["town"]
)
```

--------------------------------

### Install Pip and Uv on Windows

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/reference/installation.md

Installs or upgrades pip and installs the uv package manager on Windows.

```bat
C:\> pip3 install -U pip
C:\> pip3 install uv
```

--------------------------------

### Configure and Run Pipeline with All Resources

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/verified-sources/personio.md

Configure a pipeline with a specific name, destination, and dataset, then load all supported resources from the Personio source.

```python
pipeline = dlt.pipeline(
   pipeline_name="personio",  # Use a custom name if desired
   destination="duckdb",  # Choose the appropriate destination (e.g., duckdb, redshift, post)
   dataset_name="personio_data"  # Use a custom name if desired
)
load_data = personio_source()
print(pipeline.run(load_data))
```

--------------------------------

### Install DLT with SQLAlchemy

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/destinations/sqlalchemy.md

Install dlt with the sqlalchemy extra dependency. Database drivers must be installed separately.

```sh
pip install "dlt[sqlalchemy]"
```

--------------------------------

### Configure and Run Incremental Resource for First Time

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/hub/ingestion/ms-sql.md

Set up the incremental resource using `create_change_tracking_table` for the first run, providing the `initial_tracking_version`. This initializes incremental loading and stores the tracking version in DLT state. An optional engine isolation level can be configured.

```python
from dlthub.sources.mssql import create_change_tracking_table

# Optional: Configure engine isolation level
# use it if you create an Engine implicitly
def configure_engine_isolation_level(engine):
    return engine.execution_options(isolation_level="SERIALIZABLE")

incremental_resource = create_change_tracking_table(
    credentials=engine,
    table=table_name,
    schema=schema_name,
    initial_tracking_version=tracking_version,
    engine_adapter_callback=configure_engine_isolation_level,
)

pipeline.run(incremental_resource)
```

--------------------------------

### Install dlthub Plugin Directly

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/hub/getting-started/installation.md

Installing a plugin like `dlthub` directly might result in a version mismatch with the installed `dlt` core. This command installs the latest `dlthub` plugin, which may not be compatible with the current `dlt` version.

```sh
uv pip install dlthub
```

--------------------------------

### Install dlt with Hugging Face Support

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/destinations/huggingface.md

Install dlt with the necessary dependencies for Hugging Face integration. This command installs the `huggingface_hub` package.

```sh
pip install "dlt[hf]"
```

--------------------------------

### Initialize filesystem source

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/education/dlt-fundamentals-course/lesson_4_using_pre_build_sources_and_destinations.ipynb

Initializes the dlt template for the filesystem source using the init command. This is an alternative to directly importing the source.

```bash
!yes | dlt init filesystem duckdb
```

--------------------------------

### dlt Pipeline Initialization for Qdrant

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/destinations/qdrant.md

Shows how to initialize a dlt pipeline targeting the Qdrant destination. This example demonstrates setting the pipeline name and destination, and how to optionally skip the dataset name prefix for Qdrant collection names if desired.

```python
pipeline = dlt.pipeline(
    pipeline_name="movies",
    destination="qdrant",
)
```

--------------------------------

### Install Latest LTS Node.js Version

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/README.md

Installs and uses the latest stable version of Node.js and npm if an older version is detected. This is a prerequisite for the npm install command.

```bash
nvm install --lts
```

--------------------------------

### Install DLT with Parquet extra

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/file-formats.md

Install the pyarrow package required for Parquet support by using the dlt extra. This command installs dlt along with the necessary dependencies for Parquet.

```sh
pip install "dlt[parquet]"
```

--------------------------------

### Configure Filesystem Source with GCP Credentials in Python

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/tutorial/filesystem.md

Instantiate the filesystem source with bucket URL, GCP credentials, and a file glob pattern. Sensitive credentials like private key can be read from environment variables.

```python
import os

from dlt.common.configuration.specs import GcpClientCredentials
from dlt.sources.filesystem import filesystem, read_csv

files = filesystem(
    bucket_url="gs://filesystem-tutorial",
    # please, do not specify sensitive information directly in the code,
    # instead, you can use env variables to get the credentials
    credentials=GcpClientCredentials(
        client_email="public-access@dlthub-sandbox.iam.gserviceaccount.com",
        project_id="dlthub-sandbox",
        private_key=os.environ["GCP_PRIVATE_KEY"]  # type: ignore
    ),
    file_glob="encounters*.csv") | read_csv()
```

--------------------------------

### Perform Basic GET Request with RESTClient

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/verified-sources/rest_api/advanced.md

Use the get() method to perform a basic HTTP GET request to a specified endpoint relative to the client's base URL.

```python
client = RESTClient(base_url="https://api.example.com")
response = client.get("/posts/1")
```

--------------------------------

### Simple Identity Partitioning with Column Hint

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/destinations/iceberg.md

Define a dlt resource with the Iceberg table format and configure simple identity partitioning using the `partition` column hint.

```python
@dlt.resource(
    table_format="iceberg",
    columns={"region": {"partition": True}}
)
def my_iceberg_resource():
    yield [
        {"id": 1, "region": "US", "amount": 100},
        {"id": 2, "region": "EU", "amount": 200},
    ]
```

--------------------------------

### Install Playwright Dependencies

Source: https://github.com/dlt-hub/dlt/blob/devel/CONTRIBUTING.md

Installs Playwright dependencies for end-to-end browser testing.

```bash
playwright install
```

--------------------------------

### Filesystem Source Example with S3

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/intro.md

Extracts data from an S3 bucket using a specified file glob pattern. This example demonstrates setting up a dlt pipeline for filesystem data.

```python
from dlt.sources.filesystem import filesystem

resource = filesystem(
    bucket_url="s3://example-bucket",
    file_glob="*.csv"
)

pipeline = dlt.pipeline(
    pipeline_name="filesystem_example",
    destination="duckdb",
    dataset_name="filesystem_data",
)

load_info = pipeline.run(resource)

# print load info and the "example" table as data frame
print(load_info)
print(pipeline.dataset().example.df())
```

--------------------------------

### Initialize Filesystem Resource with Code

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/verified-sources/filesystem/index.md

Initialize the filesystem resource directly in Python code, specifying the bucket URL and file glob pattern.

```python
from dlt.sources.filesystem import filesystem

filesystem_source = filesystem(
  bucket_url="file://Users/admin/Documents/csv_files",
  file_glob="*.csv"
)
```

--------------------------------

### Install Dashboard Testing Dependencies

Source: https://github.com/dlt-hub/dlt/blob/devel/CONTRIBUTING.md

Installs dependencies required for dashboard testing.

```bash
uv sync --group dashboard-tests
```

--------------------------------

### Initialize Simulation Pipeline

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/verified-sources/pg_replication.md

Sets up a dlt pipeline for simulating changes in a PostgreSQL source, intended for local testing and demonstrating replication scenarios.

```python
# Simulation pipeline
sim_pl = dlt.pipeline(
    pipeline_name="simulation_pipeline",
    destination="postgres",
    dataset_name="source_dataset",
    dev_mode=True,
)
```

--------------------------------

### Install dlt with DuckDB

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/general-usage/data-enrichments/url-parser-data-enrichment.md

Installs the dlt library with the DuckDB destination dependency.

```bash
pip install "dlt[duckdb]"

```

--------------------------------

### Install dlt for Reverse ETL

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/destinations/destination.md

Install the dlt library without additional dependencies.

```sh
pip install dlt
```

--------------------------------

### Install dlt with DuckDB support

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/education/dlt-fundamentals-course/lesson_3_pagination_and_authentication_and_dlt_configuration.ipynb

Install the dlt library with the necessary dependencies for DuckDB.

```python
%%capture
!pip install "dlt[duckdb]"
```

--------------------------------

### Apply Hints for Merging and Renaming Table

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/verified-sources/filesystem/index.md

This example shows how to apply hints to a filesystem pipeline to specify a write disposition (merge) and a merge key. It also renames the target table to 'table_name' before running the pipeline.

```python
import dlt
from dlt.sources.filesystem import filesystem, read_csv
ilesystem_pipe = filesystem(bucket_url="file://Users/admin/Documents/csv_files", file_glob="*.csv") | read_csv()
# Tell dlt to merge on dateilesystem_pipe.apply_hints(write_disposition="merge", merge_key="date")

# We load the data into the table_name table
pipeline = dlt.pipeline(pipeline_name="my_pipeline", destination="duckdb")
load_info = pipeline.run(filesystem_pipe.with_name("table_name"))
print(load_info)
```

--------------------------------

### Simplest Source Configuration (TOML)

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/general-usage/credentials/setup.md

Use this TOML format to configure a simple source. It directly maps keys to values.

```toml
api_key="some_value"
```

--------------------------------

### Install Marimo for Data Browser

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/tutorial/rest-api.md

Install the marimo library to enable the dlt data browser.

```shell
pip install marimo

```

--------------------------------

### Creating a dlt Pipeline with rest_api Source

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/education/dlt-advanced-course/lesson_1_custom_sources_restapi_source_and_restclient.ipynb

Example of how to define a RESTAPIConfig, create a dlt source, and run a pipeline.

```APIDOC
## Creating a dlt Pipeline with rest_api Source

### Description
This example demonstrates how to define a `RESTAPIConfig` for the NewsAPI, create a `dlt` source using `rest_api_source`, and then run a `dlt` pipeline to load the data.

### Method
Python script using `dlt` library

### Endpoint
N/A (This is a client-side script)

### Parameters
N/A

### Request Body
N/A

### Request Example
```python
import dlt
from dlt.sources.rest_api import rest_api_source

# Define the configuration for the NewsAPI
news_config = {
    "client": {
        "base_url": "https://newsapi.org/v2",
        "auth": {
            "type": "apiKey",
            "key_name": "apiKey", # As specified by NewsAPI documentation
            "key_value": "YOUR_NEWSAPI_KEY" # Replace with your actual API key
        }
    },
    "resources": [
        {
            "name": "top_headlines",
            "endpoint": "/top-headlines",
            "query_params": {
                "country": "us",
                "category": "business"
            },
            "data_path": "articles"
        },
        {
            "name": "everything",
            "endpoint": "/everything",
            "query_params": {
                "q": "dlt",
                "language": "en"
            },
            "data_path": "articles"
        }
    ]
}

# Create the rest_api source
news_source = rest_api_source(news_config)

# Initialize the dlt pipeline
pipeline = dlt.pipeline(
  pipeline_name="news_pipeline",
  destination="duckdb", # Or your preferred destination
  dataset_name="news_data"
)

# Run the pipeline
print("Running the pipeline...")
load_info = pipeline.run(news_source)

print(f"Pipeline finished. Loaded {load_info.row_counts} rows.")
print(f"Schema: {pipeline.default_schema.name}")
```

### Response
#### Success Response (200)
Upon successful execution, the pipeline will load data into the specified destination. The `load_info` object contains details about the loading process.

- **`load_info.row_counts`** (dict) - A dictionary mapping table names to the number of rows loaded.
- **`load_info.load_id`** (str) - The unique identifier for this load operation.

#### Response Example
```
Running the pipeline...
Pipeline finished. Loaded {'top_headlines': 100, 'everything': 250} rows.
Schema: default
```
```

--------------------------------

### Install dlt via Conda

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/reference/installation.md

Installs the dlt library from the conda-forge channel using Conda.

```sh
conda install -c conda-forge dlt
```

--------------------------------

### Initialize dlt project with Dremio destination

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/destinations/dremio.md

Initialize a new dlt project specifying 'chess' as the source and 'dremio' as the destination, using filesystem staging.

```sh
dlt init chess dremio
```

--------------------------------

### Install dlt Library

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/reference/installation.md

Installs or upgrades the dlt library to the latest version using uv.

```sh
uv pip install -U dlt
```

--------------------------------

### Install Delta Lake Dependency

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/hub/ingestion/delta.md

Install the necessary Python package for Delta Lake integration.

```sh
pip install deltalake
```

--------------------------------

### Initialize Project with SQLAlchemy

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/destinations/sqlalchemy.md

Initialize a new dlt project with the SQLAlchemy destination. This command sets up the basic project structure and configuration.

```sh
dlt init chess sqlalchemy
```

--------------------------------

### Configure SQL Database via TOML

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/verified-sources/sql_database/advanced.md

Demonstrates how to define database credentials, backend settings, and incremental loading parameters within a TOML configuration file.

```toml
[sources.sql_database]
credentials="mssql+pyodbc://loader.database.windows.net/dlt_data?trusted_connection=yes&driver=ODBC+Driver+17+for+SQL+Server"
schema="data"
backend="pandas"
chunk_size=1000

[sources.sql_database.chat_message.incremental]
cursor_path="updated_at"
```

--------------------------------

### Install SQLAlchemy for PyIceberg

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/destinations/iceberg.md

Install or upgrade SQLAlchemy to version 2.0.18 or higher, which is a dependency for PyIceberg.

```shell
pip install 'sqlalchemy>=2.0.18'
```

--------------------------------

### Install dlt

Source: https://github.com/dlt-hub/dlt/blob/devel/README.md

Install the dlt library using pip. Ensure you have Python 3.10 or higher.

```sh
pip install dlt
```

--------------------------------

### Initialize sql_database source

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/education/dlt-fundamentals-course/lesson_4_using_pre_build_sources_and_destinations.ipynb

Initializes the dlt template for sql_database using the init command. This is an alternative to directly importing the source.

```bash
!yes | dlt init sql_database duckdb
```

--------------------------------

### Install Dependencies

Source: https://github.com/dlt-hub/dlt/blob/devel/docs/website/docs/dlt-ecosystem/verified-sources/zendesk.md

Before running your dlt pipeline, ensure all required packages are installed by running this command.

```sh
pip install -r requirements.txt

```