Sling

https://docs.slingdata.io
Admin
Sling is a modern data movement and transformation platform that simplifies and streamlines data...
Tokens:142,661
Snippets:868
Trust Score:7.6
Update:11 months ago
Show doc for...
Tokens:
Raw
### YAML Configuration for Replication Hooks

Source: https://docs.slingdata.io/concepts/hooks

Provides a comprehensive YAML configuration example for setting up replication hooks in Slingdata IO. It includes hooks for the start and end of replication, with options for query execution and validation checks.

```yaml
# replication level hooks need to be set at the root of the YAML
hooks:
  # runs in order before replication starts.
  start:
      - type: query
        query: select ....
        id: my_query # can use `{state.my_query.result[0].col1}` later
        on_failure: abort
      
  # runs in order after all streams have completed.
  end:
      # check for any errors. if errored, do not proceed (break)
      - type: check
        check: execution.status.error == 0
        on_failure: break
      
      - type: query
        query: update ....
        into: result # can use `{result.col1}` later
        on_failure: abort
```

--------------------------------

### Data Replication with Sling Python API

Source: https://docs.slingdata.io/sling-cli/getting-started

Example of using the Sling Python library to define and run a data replication task. It showcases how to configure source and target connections, define streams with SQL queries, and specify output objects.

```python
from sling import Replication, ReplicationStream

replication = Replication(
  source='MY_PG',
  target='MY_AWS_S3',
  steams={
    "my_table": ReplicationStream(
      sql="select * from my_table",
      object='my_folder/new_file.csv',
    ),
  }
)

replication.run()
```

--------------------------------

### Specify SQLite Binary Path | Bash

Source: https://docs.slingdata.io/connections/database-connections/sqlite

Provides an example of how to explicitly set the path to an existing SQLite binary installation on the system. This is useful if SlingData should not download its own SQLite binary.

```bash
export SQLITE_PATH='/opt/homebrew/bin/sqlite'
```

--------------------------------

### Basic Sling Pipeline with Logging and Replication

Source: https://docs.slingdata.io/concepts/pipeline/examples

A straightforward Sling Pipeline that demonstrates basic step sequencing. It includes logging at the start and end, a replication step, and a command execution step to display the replication status. It utilizes built-in functions like `date_format`, `now`, and `upper`.

```yaml
steps:
  - type: log
    message: 'Starting pipeline execution on {date_format(now(), "%Y-%m-%d %H:%M:%S")}. Runtime state: {runtime_state}'

  - type: replication
    path: path/to/your/replication.yaml
    id: main_replication

  - type: log
    message: 'Pipeline completed. Final state: {runtime_state}'
    level: info

  - type: command
    command: 'echo "Replication status: {upper(state.main_replication.status)}"' 
    print: true
```

--------------------------------

### Sling CLI Examples

Source: https://docs.slingdata.io/sling-platform/platform/cli-usage

Provides practical examples of using the `sling project` commands for common operations like initialization, status checks, job management, and file synchronization.

```bash
# Initialize a new project
$ sling project init

# Check project status
$ sling project status

# List all jobs
$ sling project jobs list

# Trigger a specific job
$ sling project jobs trigger job_abc123

# Sync local changes
$ sling project sync

# Force sync without confirmation
$ sling project sync --force

```

--------------------------------

### Specify DuckDB Version (Bash)

Source: https://docs.slingdata.io/connections/database-connections/duckdb

Environment variable example for specifying a particular DuckDB version to be used by Sling. Sling will automatically download the specified version if it's not found.

```bash
export DUCKDB_VERSION='0.7.0'
```

--------------------------------

### Install Sling CLI with Scoop (Windows)

Source: https://docs.slingdata.io/sling-cli/getting-started

Installs the Sling CLI on Windows using the Scoop package manager. It first adds the official Sling bucket and then installs the 'sling' package. The command 'sling -h' can verify the installation.

```powershell
scoop bucket add sling https://github.com/slingdata-io/scoop-sling.git
scoop install sling

# You're good to go!
 sling -h
```

--------------------------------

### Install Sling CLI on Linux

Source: https://docs.slingdata.io/sling-cli/getting-started

Installs the Sling CLI on Linux by downloading the latest binary release. It fetches the tarball, extracts the executable, removes the tarball, and makes the 'sling' binary executable. The command './sling -h' verifies the installation.

```shell
# download latest binary
curl -LO 'https://github.com/slingdata-io/sling-cli/releases/latest/download/sling_linux_amd64.tar.gz' \
  && tar xf sling_linux_amd64.tar.gz \
  && rm -f sling_linux_amd64.tar.gz \
  && chmod +x sling

# You're good to go!
./sling -h
```

--------------------------------

### Specify DuckDB Binary Path and Extensions (Bash)

Source: https://docs.slingdata.io/connections/database-connections/duckdb

Environment variables to configure Sling to use an existing local DuckDB installation instead of downloading it. Also includes an option to prevent Sling from downloading extensions.

```bash
export DUCKDB_PATH='/opt/homebrew/bin/duckdb'
export DUCKDB_USE_INSTALLED_EXTENSIONS='true'  # tells sling not to download extensions
```

--------------------------------

### Python Incremental Replication Example

Source: https://docs.slingdata.io/examples/database-to-database

A minimal Python example demonstrating how to set up incremental data replication using Slingdata.io. It defines the source and target connections and configures a single stream with its object mapping, replication mode, primary key, and update key.

```python
from sling import Replication, ReplicationStream, SourceOptions, TargetOptions, Mode

# Single stream example
replication = Replication(
    source='MY_SOURCE_DB',
    target='MY_TARGET_DB',
    streams={
        'source_schema.source_table': ReplicationStream(
            object='target_schema.target_table',
            mode=Mode.INCREMENTAL,
            primary_key=['id'],
            update_key='last_modified_dt'
        )
    }
)

replication.run()
```

--------------------------------

### Configure DuckLake with Environment Variables

Source: https://docs.slingdata.io/connections/datalake-connections/ducklake

Examples of configuring DuckLake connections using environment variables. This demonstrates setting the DUCKLAKE environment variable with JSON objects containing catalog and storage configurations, including S3 properties and options for non-HTTPS MinIO setups.

```bash
# configuration with s3 properties
export DUCKLAKE='{ 
  type: ducklake, 
  catalog_type: postgres,
  catalog_conn_string: "host=db.example.com port=5432 user=ducklake password=secret dbname=ducklake_catalog",
  data_path: "s3://bucket/path/to/data/folder/",
  s3_access_key_id: "AKIAIOSFODNN7EXAMPLE",
  s3_secret_access_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  s3_region: "us-east-1"
}'

# configuration with Non-HTTPS MinIO properties
export DUCKLAKE='{ 
  type: ducklake, 
  catalog_type: postgres,
  catalog_conn_string: "host=db.example.com port=5432 user=ducklake password=secret dbname=ducklake_catalog",
  data_path: "s3://bucket/path/to/data/folder/",
  s3_endpoint: machine:9000,
  s3_access_key_id: "AKIAIOSFODNN7EXAMPLE",
  s3_secret_access_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  s3_region: "minio",
  url_style: "path",
  use_ssl: false
}'
```

--------------------------------

### Set Proton Connection using Environment Variables

Source: https://docs.slingdata.io/connections/database-connections/proton

Provides examples of setting up a Proton connection via environment variables. Supports both URL format and JSON-like string format for the connection details. The `export` command is used to set these variables.

```bash
export PROTON='proton://myuser:mypass@host.ip:8463/mydatabase'
export PROTON='{ type: proton, user: "myuser", password: "mypass", host: "host.ip", port: 8463, database: "mydatabase" }'
```

--------------------------------

### Sling Python Cloud Storage Replication Setup

Source: https://docs.slingdata.io/examples/database-to-file

Initializes Sling Python replication by setting environment variables for database and cloud storage connections. This setup is a prerequisite for running replication tasks programmatically using the Sling library in Python. The `os.environ` calls are placeholders for actual connection details.

```python
from sling import Replication, ReplicationStream, TargetOptions, Format, Compression
import os

# Set environment variables
os.environ['MY_SOURCE_DB'] = '...'
os.environ['MY_CLOUD_STORAGE'] = '...'
```

--------------------------------

### Set Google Storage Connection via Environment Variable

Source: https://docs.slingdata.io/connections/file-connections/gs

This example demonstrates setting up a Google Cloud Storage connection using an environment variable. The variable `GOOGLE_STORAGE` should be assigned a JSON string containing the connection details: `type` (set to 'gs'), `bucket` name, and the `key_file` path. This method is useful for programmatic access.

```bash
export GOOGLE_STORAGE='{type: gs, bucket: sling-bucket, key_file: /path/to/keyfile.json}'
```

--------------------------------

### Database Object Location Strings (Examples)

Source: https://docs.slingdata.io/sling-cli/environment

Provides examples of location strings for referencing database objects. Covers various database types like PostgreSQL, MySQL, Snowflake, BigQuery, Oracle, and SQL Server.

```text
postgres/public.users
mysql_db/analytics.events
snowflake/DATABASE.SCHEMA.TABLE
bigquery/project.dataset.table_name
oracle_db/HR.EMPLOYEES
mssql/dbo.customers
```

--------------------------------

### Cloudflare R2 Connection Configuration in Environment Variable

Source: https://docs.slingdata.io/connections/file-connections/r2

Example of configuring Cloudflare R2 connection details as an environment variable in JSON/YAML format. This allows for easy management of credentials.

```bash
export R2_BUCKET='{type: s3, bucket: sling-bucket, access_key_id: ACCESS_KEY_ID, secret_access_key: "SECRET_ACCESS_KEY", endpoint: <endpoint>}'
```

--------------------------------

### Example Sling Data IO Pipeline

Source: https://docs.slingdata.io/concepts/pipeline

A comprehensive example of a Sling Data IO pipeline showcasing various step types including environment variable definition, logging, replication, validation (check), database queries, HTTP requests, and final logging. It utilizes environment variables and step outputs.

```yaml
env:
  DATABASE: production
  NOTIFY_URL: https://api.example.com/webhook

steps:
  # Log the start of execution
  - type: log
    message: "Starting pipeline execution"

  # Run a replication
  - type: replication
    path: replications/daily_sync.yaml
    id: daily_sync
    on_failure: warn

  # Validate the results
  - type: check
    check: state.daily_sync.status == "success"
    message: "Daily sync failed"
    on_failure: abort

  # Update status in database
  - type: query
    connection: "{env.DATABASE}"
    query: |
      UPDATE pipeline_status 
      SET last_run = current_timestamp
      WHERE name = 'daily_sync'
    on_failure: warn

  # Send notification
  - type: http
    url: "{env.NOTIFY_URL}"
    method: POST
    payload: |
      {
        "pipeline": "daily_sync",
        "status": "success",
      }

  # Log completion
  - type: log
    message: "Pipeline completed successfully"
```

--------------------------------

### Set DuckLake Connections using sling conns

Source: https://docs.slingdata.io/connections/datalake-connections/ducklake

Examples of setting up different DuckLake connections using the 'sling conns set' command. This includes configurations for local DuckDB catalogs with local or S3 data, SQLite catalogs, PostgreSQL catalogs, and MySQL catalogs, demonstrating various credential and storage path options.

```bash
# Default DuckDB catalog with local data
$ sling conns set DUCKLAKE type=ducklake catalog_conn_string=metadata.ducklake

# DuckDB catalog with S3 data storage
$ sling conns set DUCKLAKE type=ducklake catalog_conn_string=metadata.ducklake data_path=data

# SQLite catalog for multi-client access
$ sling conns set DUCKLAKE type=ducklake catalog_type=sqlite catalog_conn_string=metadata.sqlite

# PostgreSQL catalog for multi-user access
$ sling conns set DUCKLAKE type=ducklake catalog_type=postgres catalog_conn_string="host=localhost port=5432 user=myuser password=mypass dbname=ducklake_catalog"

# MySQL catalog with Azure storage
$ sling conns set DUCKLAKE type=ducklake catalog_type=mysql catalog_conn_string="host=localhost user=root password=pass db=ducklake_catalog" data_path=data
```

--------------------------------

### Example: Read Query Template (YAML)

Source: https://docs.slingdata.io/concepts/hooks/read

Shows an example of reading a SQL query template from a file using the Read hook and then utilizing it within a query hook.

```yaml
hooks:
  pre:
    - type: read
      from: "local/queries/cleanup_template.sql"
      into: "cleanup_query"
      
    - type: query
      connection: target_db
      query: "{store.cleanup_query}"
```

--------------------------------

### Configure Clickhouse Connections via Environment Variables

Source: https://docs.slingdata.io/connections/database-connections/clickhouse

Provides examples for configuring Clickhouse connections using environment variables. It shows how to set the connection string directly or using a JSON object for more detailed configuration. This method is useful for automating connection setups.

```bash
export CLICKHOUSE='clickhouse://myuser:mypass@host.ip:9000/mydatabase'
export CLICKHOUSE='{ type: clickhouse, user: "myuser", password: "mypass", host: "host.ip", port: 9000, database: "mydatabase", export_stream_format: "CSVWithNames" }'

# ClickHouse Cloud (secure=true is required)
export CLICKHOUSE_CLOUD='clickhouse://myuser:mypass@host.clickhouse.cloud:9440/mydatabase?secure=true'
```

--------------------------------

### Install Sling CLI with Homebrew (macOS)

Source: https://docs.slingdata.io/sling-cli/getting-started

Installs the Sling CLI using Homebrew on macOS. Ensure Homebrew is installed first. After installation, the 'sling' command should be available for use.

```shell
# Install with brew first
brew install slingdata-io/sling/sling

# Once, installed, `sling` should be available
 sling -h
```

--------------------------------

### Pull and Run Sling CLI with Docker

Source: https://docs.slingdata.io/sling-cli/getting-started

Pulls the latest Sling CLI Docker image and runs it to display help information. This is a convenient way to use Sling without local installation.

```bash
docker pull slingdata/sling

docker run --rm -i slingdata/sling --help
```

--------------------------------

### Configure Proton Connection in Sling Env File (YAML)

Source: https://docs.slingdata.io/connections/database-connections/proton

Illustrates how to define Proton connections within the `env.yaml` file for Sling CLI. Shows configuration for both direct parameter setup and using a connection URL under the `connections` key. Supports parameters like `type`, `host`, `user`, `port`, `database`, `schema`, and `password`.

```yaml
connections:
  PROTON:
    type: proton
    host: <host>
    user: <user>
    port: <port>
    database: <database>
    schema: <schema>
    password: <password>

  PROTON_URL:
    url: "proton://myuser:mypass@host.ip:8463/mydatabase"
```

--------------------------------

### Example: Read Configuration File (YAML)

Source: https://docs.slingdata.io/concepts/hooks/read

Demonstrates how to use the Read hook to read a database configuration file and make its content available to subsequent hooks.

```yaml
hooks:
  pre:
    - type: read
      from: "s3/config/database-config.json"
      into: "db_config"
      
    - type: log
      message: "Database config: {store.db_config}"
```

--------------------------------

### REST API Integration Example

Source: https://docs.slingdata.io/concepts/hooks/http

Example of fetching configuration from an external API before starting replication using an HTTP GET request.

```APIDOC
## REST API Integration

Fetch configuration from an external API before starting replication:

### Method

GET

### Endpoint

`https://api.company.com/v1/config/{run.stream.name}`

### Headers

```yaml
Authorization: "Bearer {source.api_key}"
Content-Type: "application/json"
```

### Example Usage in Hooks

```yaml
hooks:
  pre:
    - type: http
      url: "https://api.company.com/v1/config/{run.stream.name}"
      method: GET
      headers:
        Authorization: "Bearer {source.api_key}"
        Content-Type: "application/json"
```
```

--------------------------------

### Initialize Sling Project

Source: https://docs.slingdata.io/sling-platform/platform/cli-usage

Links a local directory to a Sling project, generates a `.sling.json` configuration file, and sets up project settings. Navigate to your project's root folder before running this command.

```bash
# navigate to the root folder of your project first
$ sling project init

```

--------------------------------

### REST API Integration Http Hook - YAML

Source: https://docs.slingdata.io/concepts/hooks/http

Example YAML configuration for fetching configuration from a REST API before replication starts using an HTTP GET request. It includes dynamic URL parameters and authentication headers.

```yaml
hooks:
  pre:
    - type: http
      url: "https://api.company.com/v1/config/{run.stream.name}"
      method: GET
      headers:
        Authorization: "Bearer {source.api_key}"
        Content-Type: "application/json"
```

--------------------------------

### Multiple Condition Check Example

Source: https://docs.slingdata.io/concepts/hooks/check

Validates multiple conditions before starting a replication process. This example checks for non-empty schemas and specific hour ranges.

```yaml
hooks:
  pre:
    - type: check
      check: |
        run.stream.schema != '' && 
        run.object.schema != '' && 
        timestamp.hour >= 1 && 
        timestamp.hour <= 23
      on_failure: abort

```

--------------------------------

### Sling Project CLI Help

Source: https://docs.slingdata.io/sling-platform/platform/cli-usage

Displays the main help information for the `sling project` command, outlining its usage, subcommands, and available flags. This is useful for understanding the scope of project management capabilities.

```bash
$ sling project
project - Manage a project on the Sling Data Platform

See more details at https://docs.slingdata.io/

  Usage:
    project [init|status|jobs|sync]

  Subcommands: 
    init     create or link to a sling project
    status   see project status
    jobs     manage project jobs
    sync     Push local file changes

  Flags: 
    --version   Displays the program version string.
    -h --help      Displays help with available flag, subcommand, and positional value parameters.

```

--------------------------------

### Update Replication Status Table

Source: https://docs.slingdata.io/concepts/hooks/query

This example demonstrates how to track the start of a replication process by updating a status table. The `pre` hook inserts a record into the `replication_status` table with the stream name, start time, and a 'RUNNING' status. It also shows an example of executing a SQL query from an external file using `file://path/to/file.sql`.

```yaml
hooks:
  pre:
    - type: query
      connection: target_db
      query: |
        INSERT INTO replication_status (
          stream_name, 
          start_time, 
          status
        ) VALUES (
          '{run.stream.name}',
          '{run.start_time}',
          'RUNNING'
        )

    - type: query
      connection: target_db
      query: file://path/to/file.sql
```

--------------------------------

### Sling Pipeline Execution with Dictionary Steps

Source: https://docs.slingdata.io/examples/sling-python

This example illustrates creating and running a Sling Pipeline using dictionaries to define the steps. This method offers flexibility and is suitable when Hook objects might be overkill or when dynamically building pipelines. It mirrors the functionality of using Hook objects but with a dictionary-based syntax.

```python
from sling import Pipeline

# Or programmatically using dictionaries
pipeline = Pipeline(
    steps=[
        {"type": "log", "message": "Hello world"},
        {"type": "copy", "from": "sftp//path/to/file", "to": "aws_s3/path/to/file"},
        {"type": "replication", "path": "path/to/replication.yaml"},
        {"type": "http", "url": "https://trigger.webhook.com"},
        {"type": "command", "command": ["ls", "-l"], "print": True}
    ],
    env={"MY_VAR": "value"}
)
pipeline.run()

```

--------------------------------

### Sling Python: Initialize Replication for File to Database

Source: https://docs.slingdata.io/examples/file-to-database

Shows how to use the Sling Python library to programmatically set up and run data replication from file sources to a database. It demonstrates setting environment variables and initializing the Replication object.

```python
from sling import Replication, ReplicationStream, SourceOptions, Mode, Format
import os

# Set environment variables
os.environ['MY_TARGET_DB'] = '...'
os.environ['SLING_SAMPLE_SIZE'] = '2000'  # increase the sample size to infer types
os.environ['SLING_THREADS'] = '3'  # run streams concurrently


```

--------------------------------

### Kerberos Authentication - Raw Credentials Configuration

Source: https://docs.slingdata.io/connections/database-connections/sqlserver

Configuration example for Kerberos authentication using raw credentials. Requires specifying the authenticator as 'krb5', realm, config file, username, and password (implied by context, not shown directly in snippet).

```yaml
authenticator=krb5
krb5_realm=comani.com
krb5_config_file=/etc/krb5.conf
```

--------------------------------

### Configure Elasticsearch via Environment Variables (Bash)

Source: https://docs.slingdata.io/connections/database-connections/elasticsearch

This section shows how to configure Elasticsearch connections using environment variables. It includes examples for a standard URL format (e.g., `elasticsearch://user:password@host.ip:9200`) and a JSON format for direct property assignment. It also provides an example for Elastic Cloud specific configuration using `ELASTICSEARCH_CLOUD` with `cloud_id` and `api_key`.

```bash
export ELASTICSEARCH='elasticsearch://user:password@host.ip:9200'
export ELASTICSEARCH='{ type: elasticsearch, user: "user", password: "password", host: "host.ip", port: 9200 }'

export ELASTICSEARCH_CLOUD="{ type: elasticsearch, cloud_id: '<cloud_id>', api_key: '<api_key>' }"
```

--------------------------------

### Set Cloudflare D1 Connection via Environment Variables

Source: https://docs.slingdata.io/connections/database-connections/d1

Configure Cloudflare D1 connection details using environment variables. Provides examples for Linux/macOS (bash) and Windows PowerShell, accepting either a URL format or a JSON object.

```bash
export D1='d1://any_user:<api_token>@<account_id>/<database>'
export D1='{ type: d1, api_token: "<api_token>", account_id: "<account_id>", database: "<database>" }'
```

```powershell
$env:D1="d1://any_user:<api_token>@<account_id>/<database>"
```

--------------------------------

### Set DuckDB Connection using Environment Variables (Bash/PowerShell)

Source: https://docs.slingdata.io/connections/database-connections/duckdb

Examples of setting the DuckDB connection string via environment variables. Supports both URL format and a JSON-like format for configuration. Includes syntax for both Linux/macOS Bash and Windows PowerShell.

```bash
export DUCKDB='duckdb:///path/to/file.db'
export DUCKDB='{ type: duckdb, instance: "/path/to/file.db" }'
```

```powershell
$env:DUCKDB="duckdb://C:/path/to/file.db" # for Windows PowerShell
```

--------------------------------

### Pipeline Configuration Example

Source: https://docs.slingdata.io/concepts/pipeline

Defines a Sling pipeline in YAML format. It includes a root 'steps' key containing an array of operations like logging, replication, and conditional querying. Environment variables can also be set.

```yaml
steps:
  - type: log
    message: "Starting pipeline execution"

  - type: replication
    path: path/to/replication.yaml
    id: my_replication

  - type: query
    if: state.my_replication.status == "success"
    connection: my_database
    query: "UPDATE status SET completed = true"

env:
  MY_KEY: VALUE
```

--------------------------------

### Replication with Python SDK

Source: https://docs.slingdata.io/examples/database-to-database

Orchestrate Sling data replication using the Python SDK. This example demonstrates setting up single and multiple streams with inline SQL and SQL file references, and running the replication process programmatically.

```python
from sling import Replication, ReplicationStream, Mode

# Single stream example with inline SQL
replication = Replication(
    source='MY_SOURCE_DB',
    target='MY_TARGET_DB',
    streams={ 
        'source_schema.source_table.1': ReplicationStream(
            sql="""
                select *
                from my_schema.my_table
                where col1 is not null
            """,
            object='target_schema.target_table',
            mode=Mode.FULL_REFRESH
        )
    }
)

replication.run()

# Multiple streams example with SQL file reference
replication = Replication(
    source='MY_SOURCE_DB',
    target='MY_TARGET_DB',
    defaults=ReplicationStream(
        mode=Mode.FULL_REFRESH
    ),
    streams={ 
        'source_schema.source_table.1': ReplicationStream(
            sql="""
                select *
                from my_schema.my_table
                where col1 is not null
            """,
            object='target_schema.target_table'
        ),
        'source_schema.source_table.2': ReplicationStream(
            sql='file:///path/to/query.sql',
            object='target_schema.target_table'
        )
    }
)

replication.run()
```

--------------------------------

### Run Replication from YAML File using Sling

Source: https://docs.slingdata.io/examples/sling-python

Demonstrates how to initialize and run a data replication process using a YAML configuration file with the Sling Python library. It shows loading configuration from a file path or directly from a file object.

```python
import yaml
from sling import Replication

# From a YAML file
replication = Replication(file_path="path/to/replication.yaml")
replication.run()

# Or load into object
with open('path/to/replication.yaml') as file:
  config = yaml.load(file, Loader=yaml.FullLoader)

replication = Replication(**config)

replication.run()
```

--------------------------------

### Set Backblaze B2 Connection using Environment Variable

Source: https://docs.slingdata.io/connections/file-connections/b2

This example demonstrates how to set up the Backblaze B2 connection details as an environment variable in JSON format. This is useful for applications or services that need to access Backblaze B2 credentials without hardcoding them.

```bash
export BACKBLAZE='{"type": "s3", "bucket": "sling-bucket", "access_key_id": "ACCESS_KEY_ID", "secret_access_key": "SECRET_ACCESS_KEY", "endpoint": "<endpoint>"}'
```

--------------------------------

### Sling Env File YAML for SQL Server Connections

Source: https://docs.slingdata.io/connections/database-connections/sqlserver

Provides examples of configuring SQL Server connections within a `env.yaml` file. Demonstrates standard connection parameters, URL-based connections, Kerberos authentication, and Azure Fabric specific settings.

```yaml
connections:
  MSSQL:
    type: sqlserver
    host: <host>
    user: <user>
    port: <port>
    instance: <instance>
    database: <database>
    schema: <schema>
    password: <password>
    encrypt: 'true'
    trust_server_certificate: 'true'

  MSSQL_URL:
    url: "sqlserver://myuser:mypass@host.ip:1433?database=mydatabase"

  MSSQL_KERBEROS:
    type: sqlserver
    host: <host>
    user: <user>
    port: <port>
    instance: <instance>
    database: <database>
    schema: <schema>
    password: <password>
    authenticator: 'krb5'
    krb5_config_file: /etc/krb5.conf
    krb5_realm: domain.com
    krb5_keytab_file: ~/MyUserName.keytab
    krb5_credcache_file: ~/MyUserNameCachedCreds
  
  MSSQL_FABRIC:
    type: sqlserver
    database: warehouse1
    host: xxx.datawarehouse.fabric.microsoft.com
    encrypt: true
    trust_server_certificate: false
    fedauth: ActiveDirectoryAzCli  # after running `az login` using Azure CLI Tool
    bcp_entra_auth: true  # in order to use BCP with fabric
```

--------------------------------

### CLI: Database to Local Parquet Export with Various Configurations

Source: https://docs.slingdata.io/examples/database-to-file

This collection of CLI commands demonstrates exporting data from a source database to local Parquet files. It includes examples for exporting to a single file, multiple files within a folder using a wildcard, and specifying target options like max rows and format. It also shows how to use Windows path formats.

```bash
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-object 'file:///tmp/my_file.parquet'

$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-object 'file:///tmp/my_parquet_folder/*.parquet'

$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-object 'file:///tmp/my_parquet_folder/' \
  --tgt-options '{file_max_rows: 4000000, format: parquet}'

# Windows Path format
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-object 'file://C:/Temp/my_parquet_folder/' \
  --tgt-options '{file_max_rows: 4000000, format: parquet}'
```

--------------------------------

### MongoDB Source/Target Synchronization Example

Source: https://docs.slingdata.io/connections/database-connections/mongodb

This YAML snippet illustrates a data synchronization configuration between MongoDB as a source and Oracle as a target. It defines default modes, specifies objects to stream, and shows how to apply filters, select specific columns, and configure incremental synchronization with primary and update keys.

```yaml
source: mongodb
target: oracle

defaults:
  mode: full-refresh
  object: user1.{stream_schema}_{stream_table}

streams:
  db1.collection1:
  db1.collection2:
    select: [col1, col2]
    where: '{"field": "value", "field2": {"$gt": 100}}'  # use "where" starting v1.3.6
  db2.collection3:
    where: '[{"or": [{"field1": "value1"}, {"field2": "value2"}]}]'  # use "where" starting v1.3.6
  db2.collection4:
    mode: incremental
    primary_key: [id]
    update_key: last_mod_at
  db2.collection5:
    mode: incremental
    primary_key: [id]
    where: '{"_id": {"$gte": "67859d8ee682ab32317abc6f", "$lte": "67859d8ee682ab32317abcc8"}}' # filter on ID (v1.4.21+)
    update_key: last_mod_at
```

--------------------------------

### List Available Connections (Sling CLI)

Source: https://docs.slingdata.io/examples/database-to-file

Lists all configured connections available within the Sling environment. This helps verify that database and storage connections are correctly set up and recognized by the Sling CLI.

```bash
$ sling conns list
+---------------+------------------+-----------------+
| CONN NAME     | CONN TYPE        | SOURCE          |
+---------------+------------------+-----------------+
| MY_S3_BUCKET  | FileSys - S3     | sling env yaml  |
| MY_SOURCE_DB  | DB - PostgreSQL  | env variable    |
| MY_GS_BUCKET  | FileSys - Google | sling env yaml  |
| MY_AZURE_CONT | FileSys - Azure  | sling env yaml  |
+---------------+------------------+-----------------+
```

```powershell
$ sling conns list
+---------------+------------------+-----------------+
| CONN NAME     | CONN TYPE        | SOURCE          |
+---------------+------------------+-----------------+
| MY_S3_BUCKET  | FileSys - S3     | sling env yaml  |
| MY_SOURCE_DB  | DB - PostgreSQL  | env variable    |
| MY_GS_BUCKET  | FileSys - Google | sling env yaml  |
| MY_AZURE_CONT | FileSys - Azure  | sling env yaml  |
+---------------+------------------+-----------------+
```

--------------------------------

### Set Proton Connection using sling conns

Source: https://docs.slingdata.io/connections/database-connections/proton

Demonstrates how to set a Proton connection named 'PROTON' using the `sling conns set` command. It shows examples using direct parameters and a connection URL. Requires `type=proton` and essential parameters like `host`, `user`, `database`, and `password`.

```bash
$ sling conns set PROTON type=proton host=<host> user=<user> database=<database> password=<password> port=<port>

# OR use url
$ sling conns set PROTON url="proton://myuser:mypass@host.ip:8463/mydatabase"
```

--------------------------------

### Advanced Sling Pipeline with Groups and Conditions

Source: https://docs.slingdata.io/concepts/pipeline/examples

This advanced pipeline demonstrates complex workflow management using nested groups and conditional logic in Sling. It includes steps for environment preparation, data replication, database optimization based on replication status, and error reporting via HTTP POST requests. The pipeline utilizes 'group', 'command', 'replication', 'query', 'log', and 'http' hooks.

```yaml
steps:
  - type: group
    id: preparation
    steps:
      - type: log
        message: 'Preparing environment'

      - type: command
        command: mkdir -p /tmp/processing
        print: true

  - type: replication
    path: replications/main.yaml
    id: main_rep

  - type: group
    if: state.main_rep.status == "success"
    steps:
      - type: query
        connection: target_db
        query: VACUUM ANALYZE {state.main_rep.object.full_name}

      - type: log
        message: 'Optimization complete'

  - type: group
    if: state.main_rep.status == "error"
    steps:
      - type: log
        message: 'Error occurred: {state.main_rep.error}'
        level: error

      - type: http
        url: https://errors.example.com/report
        method: POST
        payload: '{state.main_rep}'
```

--------------------------------

### Define Min.IO Connection in Sling Env File (YAML)

Source: https://docs.slingdata.io/connections/file-connections/minio

This YAML configuration demonstrates how to define Min.IO connection settings within a `env.yaml` file for Sling CLI. This method is useful for managing multiple connections and their configurations in a structured, declarative way. It mirrors the parameters required for direct command-line or environment variable configurations.

```yaml
connections:
  MINIO_BUCKET:
    type: s3
    bucket: <bucket>
    access_key_id: <access_key_id>
    secret_access_key: '<secret_access_key>'
    endpoint: '<endpoint>'
    url_style: path
```

--------------------------------

### Process Files in Directory with Slingdata Hooks (YAML)

Source: https://docs.slingdata.io/concepts/hooks/list

This YAML example demonstrates a Slingdata workflow that lists files recursively in an S3 directory and then processes each file using a group hook. It first uses the `list` hook to get all files and then iterates through the `result` of the `list` hook. Inside the group, it logs the name of each file that is identified as a file.

```yaml
hooks:
  pre:
    - type: list
      id: file_list
      location: "aws_s3/data/{run.stream.name}/"
      recursive: true

    - type: group
      loop: state.file_list.result
      steps:
        - type: log
          if: loop.value.is_file
          message: "Processing file: {loop.value.name}"
```

--------------------------------

### Set Min.IO Connection via Environment Variable (JSON/YAML)

Source: https://docs.slingdata.io/connections/file-connections/minio

This snippet shows how to configure Min.IO connection details as an environment variable. The configuration is presented in a format compatible with both JSON and YAML, allowing for flexible integration into CI/CD pipelines or system configurations. It includes the same essential parameters as the `sling conns` command.

```bash
export MINIO_BUCKET='{type: s3, bucket: sling-bucket, access_key_id: ACCESS_KEY_ID, secret_access_key: "SECRET_ACCESS_KEY", endpoint: <endpoint>, url_style: path}'
```

--------------------------------

### Install Sling Python Package

Source: https://docs.slingdata.io/sling-cli/getting-started

Installs the Sling Python library using pip. This allows programmatic control over Sling data replication tasks within Python applications.

```bash
pip install sling
```

--------------------------------

### Define Elasticsearch Connections in Sling Env File (YAML)

Source: https://docs.slingdata.io/connections/database-connections/elasticsearch

This example illustrates how to define Elasticsearch connections within a Sling `env.yaml` file. It shows configurations for standard connections including TLS settings, connections using multiple HTTP URLs, and connections for Elastic Cloud using `cloud_id` and `api_key`. This method is useful for managing multiple connections in a structured configuration file.

```yaml
connections:
  ELASTICSEARCH:
    type: elasticsearch
    host: host.ip
    port: 9200
    user: <user>
    password: <password>
    tls: true # if TLS connection needed

  ELASTICSEARCH_URL:
    type: elasticsearch
    http_url: http://host1:9200,http://host2:9200

  ELASTICSEARCH_CLOUD:
    type: elasticsearch
    cloud_id: deployment:id
    api_key: your_api_key
```

--------------------------------

### Slack Notification Example

Source: https://docs.slingdata.io/concepts/hooks/http

Example of sending a Slack notification after replication completes using an HTTP POST request.

```APIDOC
## Slack Notification

Send a notification to Slack after replication completes:

### Method

POST

### Endpoint

`https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK`

### Request Body

```yaml
{
  "text": "Replication Status Update",
  "blocks": [
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "*Stream:* {run.stream.name}\n*Status:* {run.status}\n*Rows Processed:* {run.total_rows}\n*Duration:* {run.duration} seconds"
      }
    }
  ]
}
```

### Example Usage in Hooks

```yaml
hooks:
  post:
    - type: http
      url: "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"
      method: POST
      payload: |
        {
          "text": "Replication Status Update",
          "blocks": [
            {
              "type": "section",
              "text": {
                "type": "mrkdwn",
                "text": "*Stream:* {run.stream.name}\n*Status:* {run.status}\n*Rows Processed:* {run.total_rows}\n*Duration:* {run.duration} seconds"
              }
            }
          ]
        }
      on_failure: warn
```
```

--------------------------------

### Microsoft Teams Alert Example

Source: https://docs.slingdata.io/concepts/hooks/http

Example of sending an alert to Microsoft Teams when replication fails using an HTTP POST request.

```APIDOC
## Microsoft Teams Alert

Send an alert to Microsoft Teams when replication fails:

### Method

POST

### Endpoint

`https://your-teams-webhook-url`

### Request Body

```yaml
{
  "@type": "MessageCard",
  "@context": "http://schema.org/extensions",
  "themeColor": "FF0000",
  "summary": "Replication Failed",
  "sections": [{
    "activityTitle": "⚠️ Replication Failed",
    "facts": [
      {
        "name": "Stream",
        "value": "{run.stream.name}"
      },
      {
        "name": "Target Table",
        "value": "{run.object.full_name}"
      },
      {
        "name": "Start Time",
        "value": "{run.start_time}"
      },
      {
        "name": "End Time",
        "value": "{run.end_time}"
      }
    ]
  }]
}
```

### Example Usage in Hooks

```yaml
hooks:
  post:
    - type: http
      if: run.status == "error"
      url: "https://your-teams-webhook-url"
      method: POST
      payload: |
        {
          "@type": "MessageCard",
          "@context": "http://schema.org/extensions",
          "themeColor": "FF0000",
          "summary": "Replication Failed",
          "sections": [{
            "activityTitle": "⚠️ Replication Failed",
            "facts": [
              {
                "name": "Stream",
                "value": "{run.stream.name}"
              },
              {
                "name": "Target Table",
                "value": "{run.object.full_name}"
              },
              {
                "name": "Start Time",
                "value": "{run.start_time}"
              },
              {
                "name": "End Time",
                "value": "{run.end_time}"
              }
            ]
          }]
        }
```
```

--------------------------------

### Example Replication with Bulk Loading YAML

Source: https://docs.slingdata.io/connections/database-connections/fabric

Demonstrates a sample Sling replication configuration utilizing bulk loading to Fabric. Specifies source and target systems, default modes, and stream details for replicating large tables.

```yaml
source: POSTGRES
target: FABRIC

defaults:
  mode: full-refresh

streams:
  public.large_table:
    object: dbo.large_table

```

--------------------------------

### Sling Data Usage Examples with Google Drive

Source: https://docs.slingdata.io/connections/file-connections/gdrive

Illustrates common operations using Google Drive connections with the `sling run` command, including listing files, copying data between local and cloud storage, and between Google Drive and databases.

```bash
# List files in Google Drive
 sling run --src-conn GDRIVE --src-stream "folder/file.csv" --stdout

# Copy from local to Google Drive
 sling run --src-stream file:///path/to/local.csv --tgt-conn GDRIVE --tgt-object "folder/remote.csv"

# Copy from Google Drive to database
 sling run --src-conn GDRIVE --src-stream "data/file.csv" --tgt-conn MYDB --tgt-object "schema.table"

# Use wildcard patterns
 sling run --src-conn GDRIVE --src-stream "reports/*.json" --stdout
```

--------------------------------

### YAML Query Hook Example with Result Storage

Source: https://docs.slingdata.io/concepts/hooks/query

An example demonstrating how to use a query hook to fetch metadata and store it in a variable. The stored results are then used in a subsequent log hook.

```yaml
hooks:
  pre:
    - type: query
      connection: source_db
      query: |
        SELECT 
          table_name,
          last_updated,
          row_count
        FROM metadata_table
        WHERE table_name = '{run.stream.name}'
      into: "table_metadata"
      
    - type: log
      message: "Last updated: {store.table_metadata[0].last_updated}, Row count: {store.table_metadata[0].row_count}"
```

--------------------------------

### Example: Using Variables in Stored Values

Source: https://docs.slingdata.io/concepts/hooks/store

This YAML example demonstrates how to store a complex value that incorporates variables from the runtime state, such as `run.stream.name` and `run.status`. It shows how to dynamically create data structures for later use.

```yaml
hooks:
  post:
    - type: store
      key: execution_summary
      value:
        stream: "{run.stream.name}"
        status: "{run.status}"
        processed_rows: "{run.total_rows}"
        timestamp: "{timestamp.datetime}"
    
    # Later, use the stored summary
    - type: log
      message: "Execution summary - Stream: {store.execution_summary.stream}, Status: {store.execution_summary.status}"
```

--------------------------------

### Create MariaDB User for Sling

Source: https://docs.slingdata.io/connections/database-connections/mariadb

SQL commands to create a database user named 'sling' and grant it necessary privileges for data loading and extraction. This involves creating the user, creating a schema if needed, and granting `SELECT`, `INSERT`, `UPDATE`, `DELETE`, `CREATE`, `DROP`, `ALTER`, `CREATE TEMPORARY TABLES`, and `CREATE VIEW` permissions.

```sql
CREATE USER 'sling'@'%' IDENTIFIED BY '<password>';

CREATE SCHEMA sling;
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, ALTER, CREATE TEMPORARY TABLES, CREATE VIEW ON sling.* TO 'sling'@'%';

-- To give read access to all tables in a specific schema
GRANT SELECT ON <schema_name>.* TO  'sling'@'%';
```

--------------------------------

### Query Data with Sling CLI (Bash)

Source: https://docs.slingdata.io/connections/datalake-connections/ducklake

This Bash command demonstrates querying data from a source connection ('DUCKLAKE') using the Sling CLI. It executes a SQL query and outputs the results to standard output.

```bash
sling run --src-conn DUCKLAKE --src-stream "SELECT * FROM sales.orders LIMIT 10" --stdout
```

--------------------------------

### Sling Pipeline Execution with Hook Objects

Source: https://docs.slingdata.io/examples/sling-python

This snippet shows how to construct and run a Sling Pipeline programmatically using specific Hook objects for type safety and clarity. It includes examples of logging, file copying, replication, HTTP requests, and command execution steps, along with environment variable configuration.

```python
from sling import Pipeline
from sling.hooks import StepLog, StepCopy, StepReplication, StepHTTP, StepCommand

# Or using Hook objects for type safety
pipeline = Pipeline(
    steps=[
        StepLog(message="Hello world"),
        StepCopy(from_="sftp//path/to/file", to="aws_s3/path/to/file"),
        StepReplication(path="path/to/replication.yaml"),
        StepHTTP(url="https://trigger.webhook.com"),
        StepCommand(command=["ls", "-l"], print_output=True)
    ],
    env={"MY_VAR": "value"}
)
pipeline.run()

```

--------------------------------

### Example: Archive Files Between Cloud Storage

Source: https://docs.slingdata.io/concepts/hooks/copy

Demonstrates how to archive files by copying them between different cloud storage providers, such as AWS S3 and Google Cloud Storage. This example utilizes dynamic paths based on run and timestamp information.

```yaml
hooks:
  post:
    - type: copy
      if: run.status == "success"
      from: "aws_s3/{run.object.file_path}"
      to: "gcs/archives/{target.name}/{timestamp.YYYY}/{timestamp.MM}/{run.object.file_path}"
      on_failure: warn
```

--------------------------------

### Generate Integer Range - SlingData Function

Source: https://docs.slingdata.io/concepts/functions

The `int_range` function generates an array of integers within a specified range. It takes `start` and `end` integers, and an optional `step` integer. The function returns an array containing the sequence of numbers from start up to (but not including) end, incrementing by step.

```SlingData
int_range(1, 5)
```

--------------------------------

### List Tables with Sling CLI (Bash)

Source: https://docs.slingdata.io/connections/datalake-connections/ducklake

This Bash command uses the Sling CLI to discover and list all available tables within a configured connection named 'DUCKLAKE'. It's a basic operation for exploring data sources.

```bash
sling conns discover DUCKLAKE
```

--------------------------------

### Set MariaDB Connection using sling conns CLI

Source: https://docs.slingdata.io/connections/database-connections/mariadb

Configure a MariaDB connection named MARIADB using the Sling CLI. You can specify connection details directly or use a URL string. Ensure `type=mariadb` is provided. This method is useful for quick setup and scripting.

```bash
$ sling conns set MARIADB type=mariadb host=<host> user=<user> database=<database> password=<password> port=<port> 

# OR use url
$ sling conns set MARIADB url="mariadb://myuser:mypass@host.ip:3306/mydatabase?tls=skip-verify"
```

--------------------------------

### File Storage Location Strings (Examples)

Source: https://docs.slingdata.io/sling-cli/environment

Illustrates the format for location strings when referencing files in local or remote storage. Supports relative and absolute paths across different cloud providers and SFTP.

```text
local/relative/path
local/../parent/relative/path
local//absolute/linux/path
local/C:/absolute/windows/path
aws_s3/path/to/folder
gcp/path/to/folder/file.parquet
azure/path/to/folder/file.log
sftp/relative/path/to/folder/file.log
sftp//absolute/path/to/folder/file.log
```
Tokens:
Raw