Soda

https://docs.soda.io
Admin
Soda is a data quality platform that combines automated observability and data testing to help...
Tokens:155,704
Snippets:1,370
Trust Score:9.7
Update:5 months ago
Show doc for...
Tokens:
Raw
### Dynamic Documentation Query Example

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/trino

Example of how to query documentation dynamically using an HTTP GET request with the 'ask' query parameter.

```http
GET https://docs.soda.io/reference/data-source-reference-for-soda-core/trino.md?ask=<question>
```

--------------------------------

### Install Soda Package (Enterprise)

Source: https://docs.soda.io/deployment-options/soda-python-libraries

Installs the 'soda' package (version 4 or higher) from the Enterprise private PyPI registry using uv. This command is an example and may need adjustment based on specific region or license.

```sh
uv pip install "soda>=4" --pre -i "https://${SODA_API_KEY_ID}:${SODA_API_KEY_SECRET}@enterprise.pypi.soda.io" --extra-index-url=https://pypi.cloud.soda.io
```

--------------------------------

### Install Soda Trino Package

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/trino

Install the necessary package for Soda to connect to Trino.

```bash
pip install soda-trino
```

--------------------------------

### Install Soda Package (Team US)

Source: https://docs.soda.io/deployment-options/soda-python-libraries

Installs the 'soda' package (version 4 or higher) from the Team US private PyPI registry using uv.

```bash
uv pip install "soda>=4" --pre -i "https://${SODA_API_KEY_ID}:${SODA_API_KEY_SECRET}@team.pypi.us.soda.io" --extra-index-url=https://pypi.cloud.soda.io
```

--------------------------------

### Install Soda Package (Enterprise US)

Source: https://docs.soda.io/deployment-options/soda-python-libraries

Installs the 'soda' package (version 4 or higher) from the Enterprise US private PyPI registry using uv.

```bash
uv pip install "soda>=4" --pre -i "https://${SODA_API_KEY_ID}:${SODA_API_KEY_SECRET}@enterprise.pypi.us.soda.io" --extra-index-url=https://pypi.cloud.soda.io
```

--------------------------------

### Install Soda Package (Team EU)

Source: https://docs.soda.io/deployment-options/soda-python-libraries

Installs the 'soda' package (version 4 or higher) from the Team EU private PyPI registry using uv.

```bash
uv pip install "soda>=4" --pre -i "https://${SODA_API_KEY_ID}:${SODA_API_KEY_SECRET}@team.pypi.cloud.soda.io" --extra-index-url=https://pypi.cloud.soda.io
```

--------------------------------

### Install Soda Package (Enterprise EU)

Source: https://docs.soda.io/deployment-options/soda-python-libraries

Installs the 'soda' package (version 4 or higher) from the Enterprise EU private PyPI registry using uv.

```bash
uv pip install "soda>=4" --pre -i "https://${SODA_API_KEY_ID}:${SODA_API_KEY_SECRET}@enterprise.pypi.cloud.soda.io" --extra-index-url=https://pypi.cloud.soda.io
```

--------------------------------

### Oracle Client Libraries Environment Variables

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/oracle

Example environment variable settings for making Oracle Client Libraries accessible to Soda Core. Ensure the Oracle Instant Client is installed and its libraries are in your system's PATH or DYLD_FALLBACK_LIBRARY_PATH.

```bash
export PATH=$PATH:/Users/adkinsty/Downloads/instantclient_23_3
export DYLD_FALLBACK_LIBRARY_PATH="/usr/local/lib:~/Downloads/instantclient_23_3:~/lib"
```

--------------------------------

### Install Soda Fabric Package

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/microsoft-fabric

Install the necessary package for Soda to connect to Microsoft Fabric.

```bash
pip install soda-fabric
```

--------------------------------

### Example Soda AI CLI Interaction

Source: https://docs.soda.io/reference/soda-ai/soda-ai-cli

This example demonstrates an interactive session with the Soda AI CLI, showing the initial prompt, suggested actions, and a user input for contract translation.

```bash
❯ soda ai

============================================================
Soda AI - AI Assistant for Data Contracts (experimental)
============================================================

Welcome back, you@company.com!

Suggested prompts:
  [1] Translate ODCS contract to Soda Data Contract
  [2] Translate Soda v3 checks file to a Soda Data Contract (v4)
  [3] Create a new data contract

Enter to submit | Shift+Enter for new line | /help for commands
============================================================

❯ translate my_contract.yml to v4

⏺ Here is your contract translated to Soda v4 format: ...
```

--------------------------------

### Install Soda Synapse Package

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/synapse

Install the necessary package for Soda to connect to Synapse. This is a prerequisite for configuration.

```bash
pip install soda-synapse
```

--------------------------------

### Query Documentation Dynamically

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/amazon-athena

Perform an HTTP GET request to query the documentation dynamically. Use the `ask` query parameter with a specific question to get direct answers and relevant excerpts.

```http
GET https://docs.soda.io/reference/data-source-reference-for-soda-core/amazon-athena.md?ask=<question>
```

--------------------------------

### Query Documentation Dynamically

Source: https://docs.soda.io/reference/rest-api/utility

To get more information not directly on the page, make an HTTP GET request with the 'ask' query parameter. The question should be specific and self-contained.

```http
GET https://docs.soda.io/reference/rest-api/utility.md?ask=<question>
```

--------------------------------

### Install Soda Core Library

Source: https://docs.soda.io/quickstart

Install the Soda Core library using pip. Ensure you use the specified index URL for development or specific versions.

```sh
pip install -i https://pypi.dev.sodadata.io/simple -U soda-core
```

--------------------------------

### Query Documentation with GET Request

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/databricks

Perform an HTTP GET request to a documentation page with the `ask` query parameter to retrieve dynamic answers and relevant excerpts. The question should be specific and self-contained.

```http
GET https://docs.soda.io/reference/data-source-reference-for-soda-core/databricks.md?ask=<question>
```

--------------------------------

### Install Soda BigQuery Package

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/bigquery

Install the necessary package for BigQuery integration with Soda Core.

```bash
pip install soda-bigquery
```

--------------------------------

### Install soda-snowflake Package

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/snowflake

Install the necessary package to enable Snowflake integration with Soda Core.

```bash
pip install soda-snowflake
```

--------------------------------

### Query Documentation via HTTP GET

Source: https://docs.soda.io/reference/soda-agent-basic-concepts

Perform an HTTP GET request to the current page URL with the 'ask' query parameter to dynamically query the documentation. The question should be specific and self-contained.

```http
GET https://docs.soda.io/reference/soda-agent-basic-concepts.md?ask=<question>
```

--------------------------------

### Query Documentation via HTTP GET

Source: https://docs.soda.io/data-observability

To get additional information not directly present on a page, perform an HTTP GET request to the page URL with the 'ask' query parameter. The question should be specific and in natural language.

```HTTP
GET https://docs.soda.io/data-observability.md?ask=<question>
```

--------------------------------

### Install soda-sparkdf package

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/spark-dataframe

Install the necessary package to use Soda with Spark DataFrames.

```bash
pip install soda-sparkdf
```

--------------------------------

### Install Soda Redshift Package

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/amazon-redshift

Install the required Python package for Soda to connect to Redshift.

```bash
pip install soda-redshift
```

--------------------------------

### Install Soda DuckDB Package

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/duckdb

Install the necessary package to use DuckDB with Soda Core.

```bash
pip install soda-duckdb
```

--------------------------------

### Install Soda Core PostgreSQL Package from Public PyPI

Source: https://docs.soda.io/deployment-options/soda-python-libraries

Install the Soda Core package for PostgreSQL from the public Soda PyPI registry. Use --pre for pre-release versions and -U to upgrade. Replace 'soda-postgres' with your specific data source package.

```bash
uv pip install -i https://pypi.cloud.soda.io/simple --pre -U "soda-postgres>4"
```

--------------------------------

### Install Soda SQL Server Package

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/sql-server

Install the necessary package for SQL Server integration with Soda.

```bash
pip install soda-sqlserver
```

--------------------------------

### Query Documentation Index

Source: https://docs.soda.io/api-docs/reporting-api-to-overview-dashboards.html

Use this GET request to ask a question about the documentation. The question should be specific and in natural language. The response includes an answer and relevant excerpts.

```http
GET https://docs.soda.io/reference/rest-api/discovered-datasets.md?ask=<question>
```

--------------------------------

### Query Documentation Dynamically

Source: https://docs.soda.io/deployment-options/soda-python-libraries

Perform an HTTP GET request on the current page URL with the `ask` query parameter to get answers to specific questions. The question should be specific, self-contained, and written in natural language. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.

```bash
GET https://docs.soda.io/deployment-options/soda-python-libraries.md?ask=<question>
```

--------------------------------

### Install soda-postgres Package

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/postgresql

Install the necessary package for PostgreSQL integration with Soda Core using pip.

```bash
pip install soda-postgres
```

--------------------------------

### Query Documentation Index

Source: https://docs.soda.io/assets/Data%20Ingestion%20Checks.ipynb

Use this GET request to ask a specific question about the documentation. The response includes direct answers and relevant excerpts.

```http
GET https://docs.soda.io/data-testing.md?ask=<question>
```

--------------------------------

### Query Documentation via HTTP GET

Source: https://docs.soda.io/soda/quick-start-sodacl.html

Use this method to ask specific questions about the documentation. The response includes a direct answer and relevant excerpts.

```http
GET https://docs.soda.io/quickstart.md?ask=<question>
```

--------------------------------

### Perform HTTP GET Request with `ask` Parameter

Source: https://docs.soda.io/reference/rest-api/secrets

Use this to query the documentation dynamically. The question should be specific and in natural language. The response will contain a direct answer and relevant excerpts.

```http
GET https://docs.soda.io/reference/rest-api/secrets.md?ask=<question>
```

--------------------------------

### Install Soda DuckDB Package

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/duckdb/duckdb-advanced-usage

Install the necessary Soda DuckDB package using pip. Ensure you are using a version greater than 4.

```bash
pip install -i https://pypi.cloud.soda.io/simple --pre -U "soda-duckdb>4"
```

--------------------------------

### Install databricks-sql-connector in Databricks Notebook

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/spark-dataframe

If you encounter an ImportError when trying to import from databricks.sql, install the databricks-sql-connector package. A library restart is necessary for the installation to be recognized.

```python
!pip install databricks-sql-connector
dbutils.library.restartPython()  # for pip installation to take effect
```

--------------------------------

### Query Documentation Dynamically

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core

To get more information not directly on this page, perform an HTTP GET request to the current URL with the 'ask' query parameter. The question should be specific and in natural language.

```HTTP
GET https://docs.soda.io/reference/data-source-reference-for-soda-core.md?ask=<question>
```

--------------------------------

### Ask a Question via HTTP GET

Source: https://docs.soda.io/soda-cl/metrics-and-checks.html

Use this method to get a direct answer to a specific question from the documentation. Append the 'ask' parameter with your question to the documentation index URL.

```http
GET https://docs.soda.io/reference/documentation-access-and-licensing.md?ask=<question>
```

--------------------------------

### Install Databricks Package

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/databricks

Install the necessary package for Databricks integration with Soda Core. This is a prerequisite for configuring the data source.

```bash
pip install soda-databricks
```

--------------------------------

### Query Documentation Dynamically

Source: https://docs.soda.io/reference/rest-api/incidents

Perform an HTTP GET request on the current page URL with the `ask` query parameter to get additional information not directly available on the page. The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

```HTTP
GET https://docs.soda.io/reference/rest-api/incidents.md?ask=<question>
```

--------------------------------

### Query Documentation Dynamically

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/polars

To get information not directly on this page, make an HTTP GET request to the page URL with the 'ask' query parameter. The question should be specific and self-contained. The response includes an answer and relevant excerpts.

```http
GET https://docs.soda.io/reference/data-source-reference-for-soda-core/polars.md?ask=<question>
```

--------------------------------

### Install soda-oracle Package

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/oracle

Install the soda-oracle package from Soda's private enterprise PyPI server using Soda Cloud API keys. This is required for connecting to Oracle with Soda Core, unless using a Runner.

```bash
pip install --pre -U "soda-oracle>4" -i "https://${SODA_CLOUD_API_KEY_ID}:${SODA_CLOUD_API_KEY_SECRET}@enterprise.pypi.us.soda.io/simple
```

--------------------------------

### Create and Activate Virtual Environment with uv

Source: https://docs.soda.io/deployment-options/soda-python-libraries

Use uv to create a virtual environment named .venv and activate it. This is the recommended method for installing Soda.

```bash
uv venv .venv
source .venv/bin/activate
```

--------------------------------

### Install Soda Athena Package

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/amazon-athena

Install the required package for Soda Athena integration using pip. Ensure you are using a version greater than 4.

```bash
pip install soda-athena>4
```

--------------------------------

### Install soda-dremio package

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/dremio

Install the soda-dremio package using pip. Ensure you are using the latest version of soda-core. Replace placeholders with your actual API Key ID and Secret.

```bash
pip install "soda-dremio" -i "https://${SODA_CLOUD_API_KEY_ID}:${SODA_CLOUD_API_KEY_SECRET}@enterprise.pypi.soda.io/simple"
```

--------------------------------

### Query Documentation Dynamically

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/snowflake

Perform an HTTP GET request on the current page URL with the `ask` query parameter to dynamically query documentation. The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

```bash
GET https://docs.soda.io/reference/data-source-reference-for-soda-core/snowflake.md?ask=<question>
```

--------------------------------

### Ask a Question via GET Request

Source: https://docs.soda.io/release-notes/soda-library.html

Use this method to query the documentation index for specific information. Append the 'ask' parameter with your question to the documentation URL.

```http
GET https://docs.soda.io/release-notes/soda-core.md?ask=<question>
```

--------------------------------

### Query Documentation Index via HTTP GET

Source: https://docs.soda.io/soda/integrate-atlan.html

Use this method to ask specific questions about the documentation. The response includes direct answers and relevant excerpts.

```http
GET https://docs.soda.io/integrations.md?ask=<question>
```

--------------------------------

### Querying Documentation Dynamically

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/microsoft-fabric

To get additional information not directly present on the page, perform an HTTP GET request to the current page URL with the 'ask' query parameter. The question should be specific and in natural language.

```http
GET https://docs.soda.io/reference/data-source-reference-for-soda-core/microsoft-fabric.md?ask=<question>
```

--------------------------------

### Ask a Question via API

Source: https://docs.soda.io/soda/connect-spark.html

Use this GET request to query the documentation index with a specific question. The response includes a direct answer and relevant excerpts.

```http
GET https://docs.soda.io/reference/contract-language-reference.md?ask=<question>
```

--------------------------------

### Query Documentation Dynamically

Source: https://docs.soda.io/deployment-options/soda-runner

Perform an HTTP GET request on the current page URL with the `ask` query parameter to dynamically query documentation. The question should be specific, self-contained, and written in natural language. The response will contain a direct answer and relevant excerpts.

```http
GET https://docs.soda.io/deployment-options/soda-runner.md?ask=<question>
```

--------------------------------

### Perform HTTP GET Request with 'ask' Parameter

Source: https://docs.soda.io/dataset-attributes-and-responsibilities

Use this method to query documentation dynamically when information is not directly available. The question should be specific and self-contained.

```HTTP
GET https://docs.soda.io/dataset-attributes-and-responsibilities.md?ask=<question>
```

--------------------------------

### Ask a Question via Documentation Index

Source: https://docs.soda.io/api-docs/public-cloud-api-v1.html

Use this GET request to query the documentation index with a natural language question. The response includes the answer and relevant excerpts.

```http
GET https://docs.soda.io/reference/python-api.md?ask=<question>
```

--------------------------------

### Ask a Question via Documentation API

Source: https://docs.soda.io/soda-cl/check-attributes.html

Use this GET request to query the documentation index with a specific question. The response will include a direct answer and relevant excerpts.

```http
GET https://docs.soda.io/reference/rest-api/attributes.md?ask=<question>
```

--------------------------------

### Ask a Question via HTTP GET Request

Source: https://docs.soda.io/soda-cl/check-template.html

Use this method to query the documentation index for specific information. Append the 'ask' parameter with your question to the documentation URL.

```http
GET https://docs.soda.io/reference/data-source-reference-for-soda-core/spark-dataframe.md?ask=<question>
```

--------------------------------

### Query Documentation via HTTP GET

Source: https://docs.soda.io/reference/soda-ai

Use this method to ask specific questions about the documentation when information is not explicitly present on the current page. The question should be self-contained and in natural language.

```http
GET https://docs.soda.io/reference/soda-ai.md?ask=<question>
```

--------------------------------

### Soda Contract YAML Example for DuckDB

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/duckdb

An example of a contract YAML file for defining checks on a DuckDB dataset. It includes column-level checks for missing values and invalid values, as well as schema and row count checks.

```yaml
dataset: datasource/main/adventureworks

columns:
  - name: id
    checks:
      - missing:
  - name: name
    checks:
      - missing:
          threshold:
            metric: percent
            must_be_less_than: 10
  - name: size
    checks:
      - invalid:
          valid_values: ['S', 'M', 'L']

checks:
  - schema:
  - row_count:
```

--------------------------------

### Query Documentation Dynamically

Source: https://docs.soda.io/diagnostics-warehouse

Perform an HTTP GET request with the 'ask' query parameter to retrieve specific information not explicitly present on the page. The question should be specific and in natural language.

```http
GET https://docs.soda.io/diagnostics-warehouse.md?ask=<question>
```

--------------------------------

### Ask a Question via HTTP GET

Source: https://docs.soda.io/release-notes

Use this method to query the documentation index with a natural language question. The response includes a direct answer and relevant excerpts.

```HTTP
GET https://docs.soda.io/reference.md?ask=<question>
```

--------------------------------

### Get Datasource Diagnostics Warehouse Configuration

Source: https://docs.soda.io/reference/rest-api/datasources

Retrieves the warehouse configuration for datasource diagnostics. This endpoint is useful for understanding the current diagnostic setup.

```json
{
  "summary": "Get datasource diagnostics warehouse configuration",
  "tags": ["Datasources"]
}
```

--------------------------------

### Python: Initialize Soda Cloud API connection

Source: https://docs.soda.io/reference/rest-api

Set up the necessary imports and authentication for connecting to the Soda Cloud API. Load credentials from environment variables and construct the authorization token.

```python
## 1. Create a working session
from dotenv import load_dotenv
import os
import json
import requests
import base64

# (Recommended) Load environment variables from a .env file
load_dotenv()
api_key_id = os.getenv("API_KEY_ID")
api_key_secret = os.getenv("API_KEY_SECRET")

# Create token from your credentials
TOKEN = base64.b64encode(f"{api_key_id}:{api_key_secret}".encode()).decode()

HOST = "cloud.us.soda.io"                  # Or "cloud.soda.io"
HEADERS = {"Authorization": f"Basic {TOKEN}", "Content-Type": "application/json"}

datasetId = "<YOUR_DATASET_ID>"            # Replace with your details
data = {"key": "value", "key2": "value2"}  # Replace with your details
```

--------------------------------

### Soda Cloud Config File (Environment Variables)

Source: https://docs.soda.io/reference/cli-reference

Example `sc_config.yml` file using environment variables for Soda Cloud host, API key ID, and API key secret. This is a more secure way to manage credentials.

```yaml
soda_cloud:
  host: cloud.soda.io
  api_key_id: ${env.SODA_API_KEY_ID}
  api_key_secret: ${env.SODA_API_KEY_SECRET}
```

--------------------------------

### Soda Cloud Config File (Hardcoded)

Source: https://docs.soda.io/reference/cli-reference

Example `sc_config.yml` file with hardcoded Soda Cloud host, API key ID, and API key secret. Ensure sensitive credentials are managed securely.

```yaml
soda_cloud:
  host: cloud.soda.io          # Use cloud.us.soda.io for US region
  api_key_id: YOUR_API_KEY_ID
  api_key_secret: YOUR_API_KEY_SECRET
```

--------------------------------

### Perform HTTP GET Request with `ask` Parameter

Source: https://docs.soda.io/reference/rest-api/contracts

Use this method to dynamically query documentation when information is not explicitly available. Append your natural language question to the URL as the `ask` query parameter.

```http
GET https://docs.soda.io/reference/rest-api/contracts.md?ask=<question>
```

--------------------------------

### Ask a Question via Documentation Index

Source: https://docs.soda.io/api-docs/reporting-api-v1.html

Use this method to perform an HTTP GET request on the documentation index with the 'ask' parameter to find specific information. The question should be specific, self-contained, and written in natural language.

```http
GET https://docs.soda.io/reference/rest-api.md?ask=<question>
```

--------------------------------

### Upgrade Pip

Source: https://docs.soda.io/deployment-options/soda-python-libraries

Ensure you have the latest version of pip before proceeding with private PyPI installations.

```sh
pip install --upgrade pip
```

--------------------------------

### Python: Send GET request to Soda Cloud API

Source: https://docs.soda.io/reference/rest-api

Send a GET request to retrieve check results or details from the Soda Cloud API. The optional 'checkIds' parameter can filter the results.

```python
## 3. Send a GET request to retrieve check results or details
response = requests.get(
    f"https://{HOST}/api/v1/checks",
    headers=HEADERS,
    params={"checkIds": ["abcd"]}          ## (optional) Replace with your details
)
print(response.json())
```

--------------------------------

### Start Soda AI CLI Interactive Session

Source: https://docs.soda.io/reference/soda-ai/soda-ai-cli

Run this command to launch the interactive Soda AI assistant in your terminal. It provides a welcome message and suggested prompts for common tasks.

```bash
soda ai
```

--------------------------------

### Dataset-Level Checks Example

Source: https://docs.soda.io/reference/contract-language-reference

Defines dataset-level checks for schema and row count. Use this for rules applying to the entire dataset.

```yaml
checks: 
  - schema: 
  - row_count:  
```

--------------------------------

### Python API Usage with Soda Cloud Config

Source: https://docs.soda.io/reference/cli-reference

Example of providing the `sc_config.yml` file path to the `verify_contracts_on_runner` Python API method. This allows the API to authenticate and connect to Soda Cloud.

```python
verify_contracts_on_runner(
  dataset_identifiers=["my_dataset"],
  soda_cloud_file_path="sc_config.yml",
)
```

--------------------------------

### Snowflake Data Source Configuration with Private Key

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/snowflake

Example data source configuration in YAML, specifying Snowflake connection details including private key authentication. Uses environment variables for sensitive information and specifies the path to the mounted private key.

```yaml
data_source ltsnowflakecustomer:
  type: snowflake
  account: ${env.SNOWFLAKE_ACCOUNT}
  database: PUBLISH_DEV
  warehouse: ${env.SNOWFLAKE_WAREHOUSE}
  role: ${env.SNOWFLAKE_ROLE}
  user: ${env.SNOWFLAKE_USER}
  authenticator: SNOWFLAKE_JWT
  private_key_passphrase: ${env.SNOWFLAKE_PASSPHRASE}
  private_key_path: /opt/soda/etc/snowflake-private-key.pk8
  client_session_keep_alive: true
  session_parameters:
    QUERY_TAG: soda-queries
    QUOTED_IDENTIFIERS_IGNORE_CASE: false
```

--------------------------------

### Get status for async dataset onboarding process

Source: https://docs.soda.io/reference/rest-api/datasources

Allows you to check the status of an asynchronous dataset onboarding operation using the operation ID.

```APIDOC
## GET /api/v1/datasources/{datasourceId}/onboardDatasets/{operationId}

### Description
This endpoint allows you to check the status of an asynchronous dataset onboarding operation.

Use the operation ID returned by the onboard datasets endpoint to poll for its current state.

### Method
GET

### Endpoint
/api/v1/datasources/{datasourceId}/onboardDatasets/{operationId}

### Parameters
#### Path Parameters
- **datasourceId** (string) - Required - The ID of the datasource.
- **operationId** (string) - Required - The ID of the asynchronous operation.

### Responses
#### Success Response (200)
- **id** (string) - The ID of the operation.
- **state** (object) - The current state of the onboarding process. See `PublicApiOnboardingStateDTO`.
- **started** (string) - The timestamp when the operation started.
- **ended** (string) - The timestamp when the operation ended.
- **message** (string) - A message providing details about the operation's status.

#### Error Response (400, 401, 403, 404, 429, 500)
- **code** (string) - An error code.
- **message** (string) - A description of the error.
```

--------------------------------

### Column-Level Checks Example

Source: https://docs.soda.io/reference/contract-language-reference

Demonstrates a Soda data contract with column-level checks, specifying data types and checks like missing and duplicate for individual columns.

```yaml
# Example of fully operational contract with only column-level checks

dataset: datasource/db/schema/dataset
columns:
  - name: id
    data_type: varchar
    checks:    # Column-level checks definition
      - missing:
      - duplicate:
  - name: name
  - name: code
```

--------------------------------

### Initialize Soda Spark DataFrame for Databricks

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/spark-dataframe

Initialize a Soda Spark DataFrame data source for use on Databricks. This example includes enabling verbose logging and assumes Unity Catalog tables are accessible.

```python
from soda_core.contracts import verify_contract_locally
from soda_sparkdf import SparkDataFrameDataSource
from soda_core import configure_logging

# Enable or disable verbose logging
configure_logging(verbose=True)

# unity catalog tables are available in the spark session
# dataset DQN in the contract should include the full path to the table, e.g.,
# dataset: soda_databricks_example/unity_catalog/tyler/obs_test_data_seasoned
spark_data_source = SparkDataFrameDataSource.from_existing_session(
    session=spark,
    name="soda_databricks_example"
)

result = verify_contract_locally(
    data_sources=[spark_data_source],
    contract_file_path="obs_test_data_seasoned.yml",
    soda_cloud_file_path="sc_config.yml",
    publish=False
)

if result.is_ok:
    print("✅ Contract verification passed.")
else:
    print("❌ Contract verification failed:")
    print(result.get_errors_str())
```

--------------------------------

### CLI Usage with Soda Cloud Config

Source: https://docs.soda.io/reference/cli-reference

Example of how to pass the `sc_config.yml` file to the `soda scan` command for authentication with Soda Cloud. This enables sending results to Soda Cloud.

```shellscript
soda scan -d my_datasource -c configuration.yml --soda-cloud sc_config.yml
```

--------------------------------

### Oracle Data Source YAML Configuration

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/oracle

Example YAML configuration for an Oracle data source. Ensure the 'type' is 'oracle' and provide valid connection details including user, password (using environment variables is recommended), host, port, and service name.

```yaml
type: oracle
name: my_oracle
connection:
  user: system
  password: ${env.ORACLE_PASSWORD}
  host: oracle.databases
  port: 1521
  service_name: SODA
```

--------------------------------

### GET Request to Checks

Source: https://docs.soda.io/reference/rest-api

Retrieve check results or details from Soda Cloud using a GET request.

```APIDOC
## GET /api/v1/checks

### Description
Retrieve check results or details from Soda Cloud.

### Method
GET

### Endpoint
/api/v1/checks

### Parameters
#### Query Parameters
- **checkIds** (string) - Optional - A comma-separated list of check IDs to retrieve. Example: "abcd"

### Response
#### Success Response (200)
- **response** (object) - The JSON response containing check information.

#### Response Example
```json
{
  "example": "response body"
}
```
```

--------------------------------

### Get User Groups - OpenAPI Specification

Source: https://docs.soda.io/reference/rest-api/users

This OpenAPI 3.1.0 specification defines the GET /api/v1/userGroups endpoint. It includes details on request parameters (page, search, size), authentication (API keys via Basic header), authorization (any Soda Cloud user), and possible responses (200 OK, 400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found, 429 Too Many Requests, 500 Internal Server Error).

```json
{"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Users Endpoints","name":"Users"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"}},"schemas":{"PublicApiUserGroupsResponse":{"type":"object","properties":{"content":{"type":"array","items":{"$ref":"#/components/schemas/UserGroupContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"UserGroupContentDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"},"users":{"$ref":"#/components/schemas/UserContentDTO"}},"required":["name","userGroupId","users"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/userGroups":{"get":{"description":"This endpoint enables you to gather information about the user groups in your organization's Soda Cloud account, including lists of members.\n\nThis GET is a paginated API that uses the following parameters to request specific details:\n\n- `size`: Supply an integer value between 10 and 1000, inclusive. The default value is 1000.\n\n- `page`: Supply an integer value. The default value is 0.\n\n- `search`: Optionally, use this parameter to perform a fuzzy search on the user group name. \n\nIf not specified, the query gathers information for all user groups in the account and sorts the results by user group name in ascending order.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint enforces authentication using the API keys you provide in the `Basic` authentication header.\n\n## Authorization\n\nAny Soda Cloud user in your organization may execute this query.\n\n## Tags\n\n`Users`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/userGroups","parameters":[{"in":"query","name":"page","schema":{"$ref":"#/components/schemas/PublicApiUserGroupsResponse/properties/number"}},{"in":"query","name":"search","schema":{"$ref":"#/components/schemas/PublicApiUserGroupsResponse/properties/content/items/properties/name"}},{"in":"query","name":"size","schema":{"$ref":"#/components/schemas/PublicApiUserGroupsResponse/properties/size"}}],"responses":{"200":{"description":"Successful response","content":{"application/json":{"$ref":"#/components/schemas/PublicApiUserGroupsResponse"}}},"400":{"$ref":"#/components/schemas/ErrorResponse"},"401":{"$ref":"#/components/schemas/ErrorResponse"},"403":{"$ref":"#/components/schemas/ErrorResponse"},"404":{"$ref":"#/components/schemas/ErrorResponse"},"429":{"$ref":"#/components/schemas/ErrorResponse"},"500":{"$ref":"#/components/schemas/ErrorResponse"}},"summary":"Get user groups","tags":["Users"]}}}}
```

--------------------------------

### Get users

Source: https://docs.soda.io/reference/rest-api/users

This endpoint enables you to gather information about the users in your organization's Soda Cloud account. It is a paginated API that supports filtering and searching.

```APIDOC
## GET /api/v1/users

### Description
This endpoint enables you to gather information about the users in your organization's Soda Cloud account.

This GET is a paginated API that uses the following parameters to request specific details:

- `size`: Supply an integer value between 1 and 1000, inclusive. The default value is 1000.

- `page`: Supply an integer value. The default value is 0.

- `search`: Optionally, use this parameter to perform a fuzzy search on the user first name, last name, or email address. 

If not specified, the query gathers information for all users in the account and sorts the results by user name in ascending order.

### Method
GET

### Endpoint
/api/v1/users

### Parameters
#### Query Parameters
- **page** (integer) - Optional - The page number to retrieve.
- **search** (string) - Optional - Fuzzy search on user first name, last name, or email address.
- **size** (integer) - Optional - The number of users to return per page (1-1000).

### Response
#### Success Response (200)
- **content** (array) - An array of user objects.
- **first** (boolean) - Indicates if this is the first page of results.
- **last** (boolean) - Indicates if this is the last page of results.
- **number** (integer) - The current page number.
- **size** (integer) - The number of items per page.
- **totalElements** (integer) - The total number of elements across all pages.
- **totalPages** (integer) - The total number of pages.

#### Response Example
```json
{
  "content": [
    {
      "email": "user@example.com",
      "firstName": "John",
      "fullName": "John Doe",
      "lastName": "Doe",
      "userId": "user-id-123"
    }
  ],
  "first": true,
  "last": false,
  "number": 0,
  "size": 1000,
  "totalElements": 50,
  "totalPages": 1
}
```

#### Error Response (400, 401, 403, 404, 429, 500)
- **code** (string) - The error code.
- **message** (string) - A description of the error.

#### Error Response Example
```json
{
  "code": "INVALID_PARAMETER",
  "message": "The provided page size is out of the allowed range."
}
```
```

--------------------------------

### Get Scan Status

Source: https://docs.soda.io/reference/rest-api/scans

This endpoint enables you to check on the state of a scan that you executed using the **Trigger a scan** endpoint. Call this endpoint to monitor the status of a scan during its execution.

If you wish to access the logs of a completed scan, use the **Get scan logs** endpoint.

This POST uses the following parameter to provide specific details:

- `scanId`: Use the value of `X-Soda-Scan-Id` returned as part of the `201` response when you called the **Trigger a scan** endpoint.

As a scan executes, you can call this endpoint to progressively collect values based on the state of the scan. Refer to the list below for the states that calls to this endpoint return.

- `queuing`: The scan is in the queue for execution, awaiting a pick-up from a Soda Agent.

- `executing`: A Soda Agent has picked up the scan and is executing.

- `cancelationRequested`: An entity requested cancelation of this scan and the request is awaiting pick-up from the Soda Agent responsible for the scan.

- `timeOutRequested`: A time out has been detected, and an automatic request to stop the scan execution is awaiting pick-up from the Soda Agent responsible for the scan.

- `canceled`: A Soda Agent confirmed that the scan has been cancelled. This is the final state of the scan.

- `timedOut`: A Soda Agent confirmed that the scan has timed-out. This is the final state of the scan.

- `failed`: The scan did not start, or it did not successfully complete because of an unexpected cause. This is the final state of the scan.

- `completedWithErrors`: The scan completed successfully, but there were errors involving some of the checks in the scan. This is the final state of the scan.

- `completedWithFailures`: The scan completed successfully and reveals failed results for some checks. This is the final state of the scan.

- `completedWithWarnings`: The scan completed successfully and reveals warning results for some checks. This is the final state of the scan.

- `completed`: The scan completed successfully and reveals passing results for all checks. This is a final state of a scan

To get the logs of the completed scan, please use API `/api/v1/scans/{scanId}/logs`.

```APIDOC
## Get scan status

### Description
This endpoint enables you to check on the state of a scan that you executed using the **Trigger a scan** endpoint. Call this endpoint to monitor the status of a scan during its execution.

If you wish to access the logs of a completed scan, use the **Get scan logs** endpoint.

This POST uses the following parameter to provide specific details:

- `scanId`: Use the value of `X-Soda-Scan-Id` returned as part of the `201` response when you called the **Trigger a scan** endpoint.

As a scan executes, you can call this endpoint to progressively collect values based on the state of the scan. Refer to the list below for the states that calls to this endpoint return.

- `queuing`: The scan is in the queue for execution, awaiting a pick-up from a Soda Agent.

- `executing`: A Soda Agent has picked up the scan and is executing.

- `cancelationRequested`: An entity requested cancelation of this scan and the request is awaiting pick-up from the Soda Agent responsible for the scan.

- `timeOutRequested`: A time out has been detected, and an automatic request to stop the scan execution is awaiting pick-up from the Soda Agent responsible for the scan.

- `canceled`: A Soda Agent confirmed that the scan has been cancelled. This is the final state of the scan.

- `timedOut`: A Soda Agent confirmed that the scan has timed-out. This is the final state of the scan.

- `failed`: The scan did not start, or it did not successfully complete because of an unexpected cause. This is the final state of the scan.

- `completedWithErrors`: The scan completed successfully, but there were errors involving some of the checks in the scan. This is the final state of the scan.

- `completedWithFailures`: The scan completed successfully and reveals failed results for some checks. This is the final state of the scan.

- `completedWithWarnings`: The scan completed successfully and reveals warning results for some checks. This is the final state of the scan.

- `completed`: The scan completed successfully and reveals passing results for all checks. This is a final state of a scan

To get the logs of the completed scan, please use API `/api/v1/scans/{scanId}/logs`.

### Method
GET

### Endpoint
/api/v1/scans/{scanId}

### Parameters
#### Path Parameters
- **scanId** (string) - Required - The ID of the scan to retrieve status for.

### Response
#### Success Response (200)
- **id** (string) - The unique identifier of the scan.
- **state** (object) - The current state of the scan. See the description for possible states.
- **cloudUrl** (string) - The URL of the scan in Soda Cloud.
- **created** (string) - The timestamp when the scan was created.
- **agentId** (string) - The ID of the agent that executed the scan.
- **checks** (array) - A list of checks associated with the scan.
- **contractDatasetCloudUrl** (string) - The URL of the contract dataset in Soda Cloud.
- **ended** (string) - The timestamp when the scan ended.
- **errors** (integer) - The number of errors encountered during the scan.
- **failures** (integer) - The number of failed checks in the scan.
- **postProcessingStages** (array) - Information about post-processing stages.
- **runnerId** (string) - The ID of the scan runner.
- **scanDefinition** (object) - Details about the scan definition.
- **scanTime** (string) - The timestamp when the scan was executed.
- **started** (string) - The timestamp when the scan started.
- **submitted** (string) - The timestamp when the scan was submitted.
- **warnings** (integer) - The number of warnings encountered during the scan.

#### Response Example
```json
{
  "id": "scan-12345",
  "state": "completed",
  "cloudUrl": "https://cloud.soda.io/scans/12345",
  "created": "2023-10-27T10:00:00Z",
  "agentId": "agent-abcde",
  "checks": [
    {
      "id": "check-67890",
      "evaluationStatus": "pass"
    }
  ],
  "contractDatasetCloudUrl": "https://cloud.soda.io/datasets/contract-xyz",
  "ended": "2023-10-27T10:05:00Z",
  "errors": 0,
  "failures": 0,
  "postProcessingStages": [],
  "runnerId": "runner-fghij",
  "scanDefinition": {
    "id": "scandef-klmno",
    "name": "My Scan Definition"
  },
  "scanTime": "2023-10-27T10:02:00Z",
  "started": "2023-10-27T10:01:00Z",
  "submitted": "2023-10-27T10:00:30Z",
  "warnings": 1
}
```
```

--------------------------------

### Get status for async dataset onboarding process

Source: https://docs.soda.io/reference/rest-api/datasources

Use the operation ID returned by the onboard datasets endpoint to poll for its current state. Authentication is required using API keys in the Basic authentication header. Any Soda Cloud user can execute this query. Rate limiting is 60 requests/60 seconds.

```json
{
  "openapi": "3.1.0",
  "info": {
    "title": "Soda Cloud API v4",
    "version": "v1"
  },
  "tags": [
    {
      "description": "Soda Cloud API Datasource Endpoints",
      "name": "Datasources"
    }
  ],
  "servers": [
    {
      "description": "Cloud EU",
      "url": "https://cloud.soda.io"
    },
    {
      "description": "Cloud US",
      "url": "https://cloud.us.soda.io"
    }
  ],
  "security": [
    {
      "basicAuthApiKey": []
    }
  ],
  "components": {
    "securitySchemes": {
      "basicAuthApiKey": {
        "scheme": "basic",
        "type": "http"
      }
    },
    "schemas": {
      "PublicApiOnboardDatasetsStatusResponse": {
        "type": "object",
        "properties": {
          "ended": {
            "type": "string",
            "format": "date-time"
          },
          "id": {
            "type": "string"
          },
          "message": {
            "type": "string"
          },
          "started": {
            "type": "string",
            "format": "date-time"
          },
          "state": {
            "type": "object",
            "$ref": "#/components/schemas/PublicApiOnboardingStateDTO"
          }
        },
        "required": [
          "id",
          "state"
        ]
      },
      "PublicApiOnboardingStateDTO": {
        "type": "string",
        "enum": [
          "queued",
          "processing",
          "completed",
          "failed",
          "cancelled"
        ]
      },
      "ErrorResponse": {
        "type": "object",
        "properties": {
          "code": {
            "type": "string"
          },
          "message": {
            "type": "string"
          }
        }
      }
    }
  },
  "paths": {
    "/api/v1/datasources/{datasourceId}/onboardDatasets/{operationId}": {
      "get": {
        "description": "This endpoint allows you to check the status of an asynchronous dataset onboarding operation.\n\nUse the operation ID returned by the onboard datasets endpoint to poll for its current state.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint enforces authentication using the API keys you provide in the `Basic` authentication header.\n\n## Authorization\n\nAny Soda Cloud user in your organization may execute this query.\n\n## Tags\n\n`Datasources`\n\n## Rate limiting\n\n60 requests/60 seconds",
        "operationId": "GET/api/v1/datasources/{datasourceId}/onboardDatasets/{operationId}",
        "parameters": [
          {
            "in": "path",
            "name": "datasourceId",
            "required": true,
            "schema": {
              "type": "string"
            }
          },
          {
            "in": "path",
            "name": "operationId",
            "required": true,
            "schema": {
              "type": "string"
            }
          }
        ],
        "responses": {
          "200": {
            "content": {
              "application/json": {
                "schema": {
                  "type": "object",
                  "$ref": "#/components/schemas/PublicApiOnboardDatasetsStatusResponse"
                }
              }
            },
            "description": "Successful response"
          },
          "400": {
            "content": {
              "application/json": {
                "schema": {
                  "type": "object",
                  "$ref": "#/components/schemas/ErrorResponse"
                }
              }
            },
            "description": "Bad request"
          },
          "401": {
            "content": {
              "application/json": {
                "schema": {
                  "type": "object",
                  "$ref": "#/components/schemas/ErrorResponse"
                }
              }
            },
            "description": "Unauthorized"
          },
          "403": {
            "content": {
              "application/json": {
                "schema": {
                  "type": "object",
                  "$ref": "#/components/schemas/ErrorResponse"
                }
              }
            },
            "description": "Forbidden"
          },
          "404": {
            "content": {
              "application/json": {
                "schema": {
                  "type": "object",
                  "$ref": "#/components/schemas/ErrorResponse"
                }
              }
            },
            "description": "Not found"
          },
          "429": {
            "content": {
              "application/json": {
                "schema": {
                  "type": "object",
                  "$ref": "#/components/schemas/ErrorResponse"
                }
              }
            },
            "description": "Too many requests"
          },
          "500": {
            "content": {
              "application/json": {
                "schema": {
                  "type": "object",
                  "$ref": "#/components/schemas/ErrorResponse"
                }
              }
            },
            "description": "Internal server error"
          }
        },
        "summary": "Get status for async dataset onboarding process",
        "tags": [
          "Datasources"
        ]
      }
    }
  }
}
```

--------------------------------

### PublicApiOnboardingStateDTO

Source: https://docs.soda.io/reference/rest-api/models

Represents the state of an onboarding operation.

```APIDOC
## PublicApiOnboardingStateDTO

### Description
Represents the state of an onboarding operation.

### Enum Values
- `queued`
- `processing`
- `completed`
- `failed`
- `cancelled`
```

--------------------------------

### Verify data contracts with wildcard filter for check types

Source: https://docs.soda.io/reference/cli-reference

This command verifies data contracts and filters checks where the 'type' field starts with a specific prefix (e.g., 'miss*'), using a wildcard for partial matching.

```bash
soda contract verify -ds ds_config.yml -c contract.yaml \
  -cf type=miss*
```

--------------------------------

### Create and Activate Virtual Environment with venv

Source: https://docs.soda.io/deployment-options/soda-python-libraries

Use the standard Python venv module to create a virtual environment named .venv and activate it. Replace 'python' with 'python3' or a specific version if needed.

```bash
python -m venv .venv
source .venv/bin/activate
```

--------------------------------

### Create Athena Data Source Configuration

Source: https://docs.soda.io/reference/data-source-reference-for-soda-core/amazon-athena

Use the Soda CLI to create a new data source configuration file for Athena. This command generates a template YAML file.

```shellscript
soda data-source create -f ds_config.yml
```
Tokens:
Raw