### Setup Python Environment for DLT-META

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/getting_started/app.md

Clone the repository, create a virtual environment, activate it, and install the necessary Python package. This prepares your local machine for development.

```commandline
git clone https://github.com/databrickslabs/dlt-meta.git
cd dlt-meta
python -m venv .venv
source .venv/bin/activate
pip install databricks-sdk
```

--------------------------------

### Install Setuptools and Wheel

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/contributing/onboarding/_index.md

Install setuptools and wheel if they are not already present. These are necessary for building Python packages.

```bash
pip install setuptools wheel
```

--------------------------------

### Install and Configure DLT-META CLI

Source: https://context7.com/databrickslabs/dlt-meta/llms.txt

Commands to install the Databricks CLI, authenticate to a workspace, install the DLT-META extension, clone the repository, set up a Python virtual environment, and install dependencies.

```bash
# Install Databricks CLI (macOS/Linux)
brew install databricks

# Authenticate to your workspace
databricks auth login --host https://<workspace>.azuredatabricks.net

# Install dlt-meta Databricks Labs extension
databricks labs install dlt-meta

# Clone the repository (needed for demo configs)
git clone https://github.com/databrickslabs/dlt-meta.git
cd dlt-meta

# Set up Python virtual environment
python -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install "PyYAML>=6.0" setuptools databricks-sdk
pip install delta-spark==3.0.0 pyspark==3.5.5 pytest>=7.0.0 coverage>=7.0.0

export PYTHONPATH=$(pwd)
```

--------------------------------

### Install Databricks SDK

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/additionals/integration_tests.md

Install the databricks-sdk package, which is required for interacting with Databricks services.

```commandline
pip install databricks-sdk
```

--------------------------------

### DLT-META Onboarding File Example (YAML)

Source: https://context7.com/databrickslabs/dlt-meta/llms.txt

An example of an onboarding YAML file (Dataflowspec) defining two data flows for CloudFiles CDC and one for Eventhub. It specifies source details, bronze and silver table configurations, data quality expectations, and CDC settings.

```yaml
# onboarding.yml — defines two flows in group A1 (CloudFiles CDC) and one in A2 (Eventhub)

- data_flow_id: "100"
  data_flow_group: "A1"
  source_system: "MYSQL"
  source_format: "cloudFiles"
  source_details:
    source_database: "APP"
    source_table: "CUSTOMERS"
    source_path_dev: "dbfs:/dlt-meta/cdc_raw/customers"
    source_metadata:
      include_autoloader_metadata_column: "True"
      autoloader_metadata_col_name: "source_metadata"
      select_metadata_cols:
        input_file_name: "_metadata.file_name"
        input_file_path: "_metadata.file_path"
  bronze_catalog_dev: "main"
  bronze_database_dev: "bronze"
  bronze_table: "customers_cdc"
  bronze_table_comment: "CDC source for customers"
  bronze_reader_options:
    cloudFiles.format: "json"
    cloudFiles.inferColumnTypes: "true"
    cloudFiles.rescuedDataColumn: "_rescued_data"
  bronze_cluster_by: ["id", "operation_date"]
  bronze_data_quality_expectations_json_dev: "dbfs:/dlt-meta/conf/dqe/customers/bronze_dqe.json"
  bronze_quarantine_table: "customers_cdc_quarantine"
  bronze_database_quarantine_dev: "bronze"
  silver_catalog_dev: "main"
  silver_database_dev: "silver"
  silver_table: "customers"
  silver_table_comment: "Cleaned customers"
  silver_cdc_apply_changes:
    keys: ["id"]
    sequence_by: "operation_date"
    scd_type: "1"
    apply_as_deletes: "operation = 'DELETE'"
    except_column_list: ["operation", "operation_date", "_rescued_data"]
  silver_transformation_json_dev: "dbfs:/dlt-meta/conf/silver_transformations.json"

- data_flow_id: "103"
  data_flow_group: "A2"
  source_format: "eventhub"
  source_details:
    source_schema_path: "dbfs:/dlt-meta/resources/iot_schema.ddl"
    eventhub.accessKeyName: "RootManageSharedAccessKey"
    eventhub.name: "iot-events"
    eventhub.accessKeySecretName: "eh-access-key"
    eventhub.secretsScopeName: "my-secrets-scope"
    kafka.sasl.mechanism: "PLAIN"
    kafka.security.protocol: "SASL_SSL"
    eventhub.namespace: "my-eh-namespace"
    eventhub.port: "9093"
  bronze_reader_options:
    maxOffsetsPerTrigger: "50000"
    startingOffsets: "latest"
    failOnDataLoss: "false"
  bronze_catalog_dev: "main"
  bronze_database_dev: "bronze"
  bronze_table: "iot_events"
  bronze_partition_columns: "date"
  bronze_data_quality_expectations_json_dev: "dbfs:/dlt-meta/conf/dqe/iot/bronze_dqe.json"
  bronze_quarantine_table: "iot_events_quarantine"
  bronze_database_quarantine_dev: "bronze"
```

--------------------------------

### Install DLT-META via Databricks CLI

Source: https://github.com/databrickslabs/dlt-meta/blob/main/README.md

Install the dlt-meta package using the Databricks CLI.

```commandline
databricks labs install dlt-meta
```

--------------------------------

### Manual Onboarding using Databricks Notebook

Source: https://context7.com/databrickslabs/dlt-meta/llms.txt

Trigger onboarding directly from a Databricks notebook by installing the `dlt-meta` package and calling the `OnboardDataflowspec` class with a parameter map.

```python
# %pip install dlt-meta

onboarding_params_map = {
    "database": "main.dlt_meta_dataflowspecs",
    "onboarding_file_path": "dbfs:/dlt-meta/conf/onboarding.json",
    "bronze_dataflowspec_table": "bronze_dataflowspec",
    "silver_dataflowspec_table": "silver_dataflowspec",
    "overwrite": "True",
    "env": "dev",
    "version": "v1",
    "import_author": "data_team"
}

from src.onboard_dataflowspec import OnboardDataflowspec
OnboardDataflowspec(spark, onboarding_params_map, uc_enabled=True).onboard_dataflow_specs()
```

--------------------------------

### Start DLT-META App Locally

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/getting_started/app.md

Execute the main Python script to start the DLT-META application on your local machine. The app will be accessible via http://127.0.0.1:5000.

```commandline
python App.py
```

--------------------------------

### Install dlt-meta Library

Source: https://github.com/databrickslabs/dlt-meta/blob/main/demo/dabs/notebooks/init_dlt_meta_pipeline.ipynb

Install the dlt-meta library using pip. This is a prerequisite for using the DLT meta pipeline functionalities.

```python
%pip install dlt-meta
```

--------------------------------

### Install Python Dependencies for Local Run

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/getting_started/app.md

Before running the app locally, install all required Python packages listed in the requirements.txt file.

```commandline
cd dlt-meta/lakehouse_app
pip install -r requirements.txt
```

--------------------------------

### Install Additional Dependencies

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/contributing/onboarding/_index.md

Install additional libraries required for DLT-META, such as pyspark, delta-spark, and pytest.

```bash
pip install pyspark
```

```bash
pip install delta-spark
```

```bash
Pip install pytest
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/contributing/onboarding/_index.md

Install the project's dependencies in editable mode using pip. This allows for development and testing of the project.

```bash
pip install -e .
```

--------------------------------

### Install DLT-META Core Requirements

Source: https://github.com/databrickslabs/dlt-meta/blob/main/README.md

Install the core Python packages required for dlt-meta, including PyYAML, setuptools, and databricks-sdk.

```commandline
pip install "PyYAML>=6.0" setuptools databricks-sdk
```

--------------------------------

### Successful Integration Test Output Example

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/additionals/integration_tests.md

This is an example of the expected output file content after a successful integration test run. It includes completion messages and validation results for table counts.

```text
,0
0,Completed Bronze Lakeflow Declarative Pipeline.
1,Completed Silver Lakeflow Declarative Pipeline.
2,Validating Lakeflow Declarative Pipeline Bronze and Silver Table Counts...
3,Validating Counts for Table bronze_7d1d3ccc9e144a85b07c23110ea50133.transactions.
4,Expected: 10002 Actual: 10002. Passed!
5,Validating Counts for Table bronze_7d1d3ccc9e144a85b07c23110ea50133.transactions_quarantine.
6,Expected: 7 Actual: 7. Passed!
7,Validating Counts for Table bronze_7d1d3ccc9e144a85b07c23110ea50133.customers.
8,Expected: 98928 Actual: 98928. Passed!
9,Validating Counts for Table bronze_7d1d3ccc9e144a85b07c23110ea50133.customers_quarantine.
10,Expected: 1077 Actual: 1077. Passed!
11,Validating Counts for Table silver_7d1d3ccc9e144a85b07c23110ea50133.transactions.
12,Expected: 8759 Actual: 8759. Passed!
13,Validating Counts for Table silver_7d1d3ccc9e144a85b07c23110ea50133.customers.
14,Expected: 87256 Actual: 87256. Passed!
```

--------------------------------

### Start DLT Command

Source: https://github.com/databrickslabs/dlt-meta/blob/main/lakehouse_app/templates/landingPage.html

Initiates a new command session by sending the command to the server. It displays a waiting dialog, starts polling for output, and appends the command to the terminal.

```javascript
async function startCommand(command) {
  try {
    showWaitingDialog();
    const response = await fetch('/start_command', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json'
      },
      body: JSON.stringify({command})
    });
    const data = await response.json();
    currentCommandId = data.command_id;
    // Start polling for output
    pollOutput();
    appendToTerminal(`$ ${command}`, 'input-line');
    inputField.value = '';
    hideWaitingDialog();
  } catch (error) {
    hideWaitingDialog();
    appendToTerminal(`Error: ${error.message}`, 'error-line');
  }
}
```

--------------------------------

### Manual Onboarding Parameters for Databricks Jobs (Python Wheel)

Source: https://context7.com/databrickslabs/dlt-meta/llms.txt

Configure Databricks Workflow tasks using JSON parameters for Python wheel onboarding. Examples are provided for both Unity Catalog enabled and disabled scenarios.

```json
{
    "onboard_layer": "bronze_silver",
    "database": "main.dlt_meta_dataflowspecs",
    "onboarding_file_path": "dbfs:/dlt-meta/conf/onboarding.json",
    "bronze_dataflowspec_table": "bronze_dataflowspec",
    "silver_dataflowspec_table": "silver_dataflowspec",
    "import_author": "data_team",
    "version": "v1",
    "uc_enabled": "True",
    "overwrite": "True",
    "env": "dev"
}
```

```json
{
    "onboard_layer": "bronze_silver",
    "database": "dlt_demo",
    "onboarding_file_path": "dbfs:/dlt-meta/conf/onboarding.json",
    "bronze_dataflowspec_table": "bronze_dataflowspec_table",
    "bronze_dataflowspec_path": "dbfs:/dlt-meta/bronze_specs",
    "silver_dataflowspec_table": "silver_dataflowspec_table",
    "silver_dataflowspec_path": "dbfs:/dlt-meta/silver_specs",
    "import_author": "data_team",
    "version": "v1",
    "uc_enabled": "False",
    "overwrite": "True",
    "env": "dev"
}
```

--------------------------------

### Install Python Package Requirements

Source: https://github.com/databrickslabs/dlt-meta/blob/main/demo/README.md

Installs necessary Python packages for DLT-META demos. Ensure PyYAML is version 6.0 or higher.

```commandline
pip install "PyYAML>=6.0" setuptools databricks-sdk
pip install delta-spark==3.0.0 pyspark==3.5.5
```

--------------------------------

### Install Python Packages

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/getting_started/dltmeta_cli.md

Install the necessary Python packages for dlt-meta, including core requirements, development tools, and integration test dependencies. Ensure your virtual environment is activated.

```bash
# Core requirements
pip install "PyYAML>=6.0" setuptools databricks-sdk

# Development requirements
pip install flake8==6.0 delta-spark==3.0.0 pytest>=7.0.0 coverage>=7.0.0 pyspark==3.5.5

# Integration test requirements
pip install "typer[all]==0.6.1"
```

--------------------------------

### Onboard Dataflowspec with Unity Catalog

Source: https://context7.com/databrickslabs/dlt-meta/llms.txt

Instantiates and uses the OnboardDataflowspec class to onboard dataflow specifications with Unity Catalog enabled. Requires installation of the dlt-meta package.

```python
# In a Databricks notebook cell:
# %pip install dlt-meta

# --- With Unity Catalog ---
onboarding_params_uc = {
    "onboarding_file_path": "/Volumes/main/dlt_meta/dlt_meta_conf/onboarding.json",
    "database": "main.dlt_meta_dataflowspecs",   # catalog.schema
    "bronze_dataflowspec_table": "bronze_dataflowspec",
    "silver_dataflowspec_table": "silver_dataflowspec",
    "overwrite": "True",
    "env": "dev",
    "version": "v1",
    "import_author": "data_team"
}

from src.onboard_dataflowspec import OnboardDataflowspec

OnboardDataflowspec(
    spark,
    onboarding_params_uc,
    uc_enabled=True
).onboard_dataflow_specs()
# => Writes main.dlt_meta_dataflowspecs.bronze_dataflowspec
# => Writes main.dlt_meta_dataflowspecs.silver_dataflowspec
```

--------------------------------

### Deploy and Run DLT Pipeline using DAB CLI

Source: https://context7.com/databrickslabs/dlt-meta/llms.txt

Use the Databricks Asset Bundles CLI to deploy and run your defined DLT pipelines. Ensure you have the DAB CLI installed and configured for your Databricks environment.

```bash
# Deploy and run via DAB CLI
databricks bundle deploy --target dev
databricks bundle run people_pipeline --target dev
```

--------------------------------

### DLT-META Onboarding with Databricks Notebook (With Unity Catalog)

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/getting_started/dltmeta_manual.md

Configure and run the DLT-META onboarding process using a Databricks notebook with Unity Catalog enabled. Set 'uc_enabled=True' and specify the database correctly.

```python
onboarding_params_map = {
		"database": "uc_name.dlt_demo",
		"onboarding_file_path": "dbfs:/dlt-meta/conf/onboarding.json",,
		"bronze_dataflowspec_table": "bronze_dataflowspec_table", 
		"silver_dataflowspec_table": "silver_dataflowspec_table",
		"overwrite": "True",
		"env": "dev",
		"version": "v1",
		"import_author": "Ravi"
		}

from src.onboard_dataflowspec import OnboardDataflowspec
OnboardDataflowspec(spark, onboarding_params_map, uc_enabled=True).onboard_dataflow_specs()
```

--------------------------------

### Install Python Package Requirements

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/demo/Apply_Changes_From_Snapshot.md

Install core and development Python packages required for the dlt-meta project. Ensure you have PyYAML, setuptools, and databricks-sdk installed.

```commandline
# Core requirements
pip install "PyYAML>=6.0" setuptools databricks-sdk

# Development requirements
pip install flake8==6.0 delta-spark==3.0.0 pytest>=7.0.0 coverage>=7.0.0 pyspark==3.5.5
```

--------------------------------

### DLT-META Onboarding with Databricks Notebook (No Unity Catalog)

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/getting_started/dltmeta_manual.md

Configure and run the DLT-META onboarding process using a Databricks notebook without Unity Catalog. Ensure 'onboarding_params_map' is correctly set.

```python
onboarding_params_map = {
		"database": "dlt_demo",
		"onboarding_file_path": "dbfs:/dlt-meta/conf/onboarding.json",
		"bronze_dataflowspec_table": "bronze_dataflowspec_table", 
		"bronze_dataflowspec_path": "dbfs:/onboarding_tables_cdc/bronze",                       
		"silver_dataflowspec_table": "silver_dataflowspec_table",
		"silver_dataflowspec_path": "dbfs:/onboarding_tables_cdc/silver",
		"overwrite": "True",
		"env": "dev",
		"version": "v1",
		"import_author": "Ravi"
		}

from src.onboard_dataflowspec import OnboardDataflowspec
OnboardDataflowspec(spark, onboarding_params_map).onboard_dataflow_specs()
```

--------------------------------

### Python Wheel / Notebook Manual Onboarding

Source: https://context7.com/databrickslabs/dlt-meta/llms.txt

Instructions for triggering DLT-META onboarding directly from a Databricks Workflow task (Python wheel) or a notebook.

```APIDOC
## Python Wheel / Notebook Manual Onboarding

For teams that prefer to trigger onboarding directly from a Databricks Workflow task (Python wheel) or a notebook without using the CLI.

### Option A: Python Wheel Job parameters (paste as JSON in the "Parameters" field)

#### With Unity Catalog:
```json
{
    "onboard_layer": "bronze_silver",
    "database": "main.dlt_meta_dataflowspecs",
    "onboarding_file_path": "dbfs:/dlt-meta/conf/onboarding.json",
    "bronze_dataflowspec_table": "bronze_dataflowspec",
    "silver_dataflowspec_table": "silver_dataflowspec",
    "import_author": "data_team",
    "version": "v1",
    "uc_enabled": "True",
    "overwrite": "True",
    "env": "dev"
}
```

#### Without Unity Catalog:
```json
{
    "onboard_layer": "bronze_silver",
    "database": "dlt_demo",
    "onboarding_file_path": "dbfs:/dlt-meta/conf/onboarding.json",
    "bronze_dataflowspec_table": "bronze_dataflowspec_table",
    "bronze_dataflowspec_path": "dbfs:/dlt-meta/bronze_specs",
    "silver_dataflowspec_table": "silver_dataflowspec_table",
    "silver_dataflowspec_path": "dbfs:/dlt-meta/silver_specs",
    "import_author": "data_team",
    "version": "v1",
    "uc_enabled": "False",
    "overwrite": "True",
    "env": "dev"
}
```

### Option B: Databricks Notebook
```python
# %pip install dlt-meta

onboarding_params_map = {
    "database": "main.dlt_meta_dataflowspecs",
    "onboarding_file_path": "dbfs:/dlt-meta/conf/onboarding.json",
    "bronze_dataflowspec_table": "bronze_dataflowspec",
    "silver_dataflowspec_table": "silver_dataflowspec",
    "overwrite": "True",
    "env": "dev",
    "version": "v1",
    "import_author": "data_team"
}

from src.onboard_dataflowspec import OnboardDataflowspec
OnboardDataflowspec(spark, onboarding_params_map, uc_enabled=True).onboard_dataflow_specs()
```

### Option C: Lakeflow Declarative Pipeline runner notebook
```python
# %pip install dlt-meta
layer = spark.conf.get("layer", None)   # set in pipeline configuration
from src.dataflow_pipeline import DataflowPipeline
DataflowPipeline.invoke_dlt_pipeline(spark, layer)
```
```

--------------------------------

### Launch DLT Sink Demo

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/demo/DLT_Sink.md

Execute the demo script to launch the DLT sink pipeline. Configure it with your Unity Catalog name, Kafka topics, server details, and profile.

```commandline
python demo/launch_dlt_sink_demo.py --uc_catalog_name=<<uc_catalog_name>> --source=kafka --kafka_source_topic=<<kafka source topic name>> --kafka_sink_topic=<<kafka sink topic name>> --kafka_source_servers_secrets_scope_name=<<kafka source servers secret name>> --kafka_source_servers_secrets_scope_key=<<kafka source server secret scope key name>> --kafka_sink_servers_secret_scope_name=<<kafka sink server secret scope key name>> --kafka_sink_servers_secret_scope_key=<<kafka sink servers secret scope key name>> --profile=<<DEFAULT>>
```

--------------------------------

### Install DLT-META Development Requirements

Source: https://github.com/databrickslabs/dlt-meta/blob/main/README.md

Install development-specific Python packages for dlt-meta, such as delta-spark, pyspark, pytest, and coverage.

```commandline
pip install delta-spark==3.0.0 pyspark==3.5.5 pytest>=7.0.0 coverage>=7.0.0
```

--------------------------------

### Install DLT-META Integration Test Requirements

Source: https://github.com/databrickslabs/dlt-meta/blob/main/README.md

Install Python packages needed for integration testing dlt-meta, including typer.

```commandline
pip install "typer[all]==0.6.1"
```

--------------------------------

### DLT-META Onboarding JSON Parameters (With Unity Catalog)

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/getting_started/dltmeta_manual.md

Use these JSON parameters for onboarding jobs with Unity Catalog enabled. The 'database' field should specify the catalog and schema.

```json
    {
        "onboard_layer": "bronze_silver",
        "database": "uc_name.dlt_demo",
        "onboarding_file_path": "dbfs:/dlt-meta/conf/onboarding.json",
        "silver_dataflowspec_table": "silver_dataflowspec_table",
        "bronze_dataflowspec_table": "bronze_dataflowspec_table",
        "import_author": "Ravi",
        "version": "v1",
        "uc_enabled": "True",
        "overwrite": "True",
        "env": "dev"
    }
```

--------------------------------

### Databricks Labs DLT-Meta CLI Onboarding

Source: https://context7.com/databrickslabs/dlt-meta/llms.txt

Use the `databricks labs dlt-meta onboard` CLI command for interactive onboarding. This process uploads config files, creates jobs, and populates DataflowSpec tables.

```bash
# Step 1: authenticate
databricks auth login --host https://adb-<workspace-id>.azuredatabricks.net

# Step 2: run interactive onboarding
databricks labs dlt-meta onboard
# Prompts (accept defaults for demo, or provide your own values):
#   Run onboarding with unity catalog enabled? → True / False
#   Provide unity catalog name → main
#   Run onboarding with serverless? → True / False
#   Provide onboarding file path → demo/conf/onboarding.template
#   Provide onboarding files local directory → /path/to/dlt-meta/demo/
#   Provide dlt meta schema name → dlt_meta_dataflowspecs_<uuid>
#   Provide dlt meta bronze layer schema name → dltmeta_bronze_<uuid>
#   Provide dlt meta silver layer schema name → dltmeta_silver_<uuid>
#   Provide dlt meta layer → bronze_silver
#   Provide bronze dataflow spec table name → bronze_dataflowspec
#   Provide silver dataflow spec table name → silver_dataflowspec
#   Overwrite dataflow spec? → True
#   Provide dataflow spec version → v1
#   Provide environment name → dev
```

--------------------------------

### DLT-META CLI Commands

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/_index.md

Commands for onboarding and deploying DLT-META pipelines using the CLI.

```bash
databricks labs dlt-meta onboard
```

```bash
databricks labs dlt-meta deploy
```

--------------------------------

### Build the Project

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/contributing/onboarding/_index.md

Build the source distribution (sdist) and wheel distribution (bdist_wheel) of the project. This is useful for packaging and distribution.

```bash
python setup.py sdist bdist_wheel
```

--------------------------------

### DLT-META CLI Onboarding Parameters for Silver Layer

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/faq/execution.md

Use these parameters with the DLT-META CLI for onboarding the silver layer. Ensure 'onboard_layer' is set to 'silver'.

```json
{
        "onboard_layer": "silver",
        "database": "dlt_demo",
        "onboarding_file_path": "dbfs:/dlt-meta/conf/onboarding.json",
        "silver_dataflowspec_table": "silver_dataflowspec_table",
        "import_author": "Ravi",
        "version": "v1",
        "uc_enabled": "True",
        "overwrite": "True",
        "env": "dev"
}
```

--------------------------------

### DLT-META CLI Onboarding Parameters for Bronze Layer

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/faq/execution.md

Use these parameters with the DLT-META CLI for onboarding the bronze layer. Ensure 'onboard_layer' is set to 'bronze'.

```json
{
        "onboard_layer": "bronze",
        "database": "dlt_demo",
        "onboarding_file_path": "dbfs:/dlt-meta/conf/onboarding.json",
        "bronze_dataflowspec_table": "bronze_dataflowspec_table",
        "import_author": "Ravi",
        "version": "v1",
        "uc_enabled": "True",
        "overwrite": "True",
        "env": "dev"
}
```

--------------------------------

### Run DLT-META Onboarding Command

Source: https://github.com/databrickslabs/dlt-meta/blob/main/README.md

Execute the dlt-meta onboard command to push code and data to your Databricks workspace and create an onboarding job. Accepts default values if the repository is cloned.

```commandline
databricks labs dlt-meta onboard
```

--------------------------------

### Configure and Onboard Bronze Dataflow Spec

Source: https://github.com/databrickslabs/dlt-meta/blob/main/examples/manual_onboard_bronze.ipynb

Configure parameters for onboarding and then initiate the bronze dataflow spec onboarding process. Ensure spark is initialized and parameters are correctly set.

```python
onboarding_params_map = {
	"database": "uc_name.dlt_demo",
	"onboarding_file_path": "dbfs:/onboarding_files/users_onboarding.json",
	"bronze_dataflowspec_table": "bronze_dataflowspec_table", 
	"overwrite": "True",
	"env": "dev",
	"version": "v1",
	"import_author": "Ravi"
}

from src.onboard_dataflowspec import OnboardDataflowspec
OnboardDataflowspec(spark, onboarding_params_map, uc_enabled=True).onboard_bronze_dataflow_spec()

```

--------------------------------

### DLT-META Onboarding JSON Parameters (No Unity Catalog)

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/getting_started/dltmeta_manual.md

Use these JSON parameters for onboarding jobs without Unity Catalog enabled. Ensure the 'database' field is correctly set.

```json
    {
        "onboard_layer": "bronze_silver",
        "database": "dlt_demo",
        "onboarding_file_path": "dbfs:/dlt-meta/conf/onboarding.json",
        "silver_dataflowspec_table": "silver_dataflowspec_table",
        "silver_dataflowspec_path": "dbfs:/onboarding_tables_cdc/silver",
        "bronze_dataflowspec_table": "bronze_dataflowspec_table",
        "import_author": "Ravi",
        "version": "v1",
        "bronze_dataflowspec_path": "dbfs:/onboarding_tables_cdc/bronze",
        "onboard_layer": "bronze_silver",
        "uc_enabled": "False",
        "overwrite": "True",
        "env": "dev"
    }
```

--------------------------------

### Launch DLT Pipeline with DataflowPipeline

Source: https://context7.com/databrickslabs/dlt-meta/llms.txt

Launches a Lakeflow Declarative Pipeline by invoking DataflowPipeline.invoke_dlt_pipeline. Requires the dlt-meta package to be installed and configured.

```python
# init_dlt_meta_pipeline.py — the pipeline runner notebook

# Cell 1 (install package on the pipeline cluster)
# %pip install dlt-meta==0.0.10
# dbutils.library.restartPython()

# Cell 2 — invoke pipeline
layer = spark.conf.get("layer", None)   # "bronze", "silver", or "bronze_silver"

from src.dataflow_pipeline import DataflowPipeline

DataflowPipeline.invoke_dlt_pipeline(spark, layer)
```

--------------------------------

### Navigate to DAB Directory

Source: https://github.com/databrickslabs/dlt-meta/blob/main/demo/README.md

Changes the current directory to the 'demo/dabs' folder, where DAB configuration files are located.

```commandline
cd demo/dabs
```

--------------------------------

### PipelineReaders Autoloader (CloudFiles) Reader

Source: https://context7.com/databrickslabs/dlt-meta/llms.txt

Configure and use PipelineReaders to read data using Autoloader (CloudFiles). This example shows how to include autoloader metadata columns.

```python
from src.pipeline_readers import PipelineReaders
from pyspark.sql.types import StructType
import json

# --- Autoloader (CloudFiles) reader ---
source_details = {
    "path": "dbfs:/raw/customers",
    "source_metadata": json.dumps({
        "include_autoloader_metadata_column": "True",
        "autoloader_metadata_col_name": "source_metadata",
        "select_metadata_cols": {"input_file_name": "_metadata.file_name"}
    })
}
reader_opts = {"cloudFiles.format": "json", "cloudFiles.inferColumnTypes": "true"}
schema_json = None  # or a StructType.jsonValue() dict

pr = PipelineReaders(spark, "cloudFiles", source_details, reader_opts, schema_json)
df = pr.read_dlt_cloud_files()
# df contains raw records + source_metadata column
```

--------------------------------

### Launch Apply Changes From Snapshot Demo

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/demo/Apply_Changes_From_Snapshot.md

Execute the demo script to showcase the apply changes from snapshot pattern. Provide your Unity Catalog name using the --uc_catalog_name argument. Optionally, specify a Databricks CLI profile.

```commandline
python demo/launch_acfs_demo.py --uc_catalog_name=<<uc catalog name>>
```

--------------------------------

### PipelineReaders Eventhub / Kafka Reader

Source: https://context7.com/databrickslabs/dlt-meta/llms.txt

Configure and use PipelineReaders to read from Eventhub or Kafka. This example demonstrates setting up connection details, reader options, and schema parsing.

```python
from src.pipeline_readers import PipelineReaders
from pyspark.sql.types import StructType
import json

# --- Eventhub / Kafka reader ---
eh_source = {
    "eventhub.namespace": "my-namespace",
    "eventhub.port": "9093",
    "eventhub.name": "iot-events",
    "eventhub.accessKeyName": "RootManageSharedAccessKey",
    "eventhub.accessKeySecretName": "eh-secret",
    "eventhub.secretsScopeName": "my-scope",
    "kafka.sasl.mechanism": "PLAIN",
    "kafka.security.protocol": "SASL_SSL"
}
eh_reader_opts = {"maxOffsetsPerTrigger": "50000", "startingOffsets": "latest"}
with open("iot_schema.ddl") as f:
    ddl = f.read()
from pyspark.sql.types import _parse_datatype_string
schema = json.loads(json.dumps(_parse_datatype_string(ddl).jsonValue()))

pr_eh = PipelineReaders(spark, "eventhub", eh_source, eh_reader_opts, schema)
raw_df = pr_eh.read_kafka()
# raw_df has parsed_records column containing deserialized JSON payload
```

--------------------------------

### Onboard Dataflow Specifications

Source: https://github.com/databrickslabs/dlt-meta/blob/main/examples/manual_onboard.ipynb

Configure and execute the onboarding process for dataflow specifications. Ensure the 'spark' session is available and 'onboarding_params_map' is correctly defined with your specific parameters.

```python
onboarding_params_map = {
    "database": "uc_name.dlt_demo",
    "onboarding_file_path": "dbfs:/onboarding_files/users_onboarding.json",
    "bronze_dataflowspec_table": "bronze_dataflowspec_table", 
    "silver_dataflowspec_table": "silver_dataflowspec_table",
    "overwrite": "True",
    "env": "dev",
    "version": "v1",
    "import_author": "Ravi"
}

from src.onboard_dataflowspec import OnboardDataflowspec
OnboardDataflowspec(spark, onboarding_params_map, uc_enabled=True).onboard_dataflow_specs()
```

--------------------------------

### Lakeflow Declarative Pipeline Runner Notebook

Source: https://context7.com/databrickslabs/dlt-meta/llms.txt

Use this snippet in a Databricks notebook to invoke a DLT pipeline. Ensure the `dlt-meta` package is installed and the `layer` configuration is set in the pipeline.

```python
# %pip install dlt-meta
layer = spark.conf.get("layer", None)   # set in pipeline configuration
from src.dataflow_pipeline import DataflowPipeline
DataflowPipeline.invoke_dlt_pipeline(spark, layer)
```

--------------------------------

### Sync and Deploy App Code to Databricks

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/getting_started/app.md

Navigate to the app's directory, sync your local code to a specified path in the Databricks workspace, and then deploy the application. Replace placeholder values with your actual workspace host and desired path.

```commandline
cd dlt-meta/lakehouse_app

# Replace testapp with your preferred folder name
databricks sync . /Workspace/Users/<user1.user2>@databricks.com/testapp

# Deploy the app
databricks apps deploy demo-dltmeta --source-code-path /Workspace/Users/<user1.user2>@databricks.com/testapp
```

--------------------------------

### DLT Sink API for External Tables and Kafka

Source: https://context7.com/databrickslabs/dlt-meta/llms.txt

Route pipeline output to external Delta tables and Kafka topics using `dlt.create_sink`. This example writes raw products data to an archive table and a Kafka topic, with filtering.

```yaml
# Write bronze output to an external Delta table AND a Kafka topic
- data_flow_id: "400"
  data_flow_group: "D1"
  source_format: "cloudFiles"
  source_details:
    source_path_dev: "dbfs:/raw/products"
  bronze_database_dev: "bronze"
  bronze_table: "products_raw"
  bronze_reader_options:
    cloudFiles.format: "csv"
    header: "true"
  bronze_sinks:
    - name: "products_delta_sink"
      format: "delta"
      options:
        tableName: "main.external.products_archive"
    - name: "products_kafka_sink"
      format: "kafka"
      options:
        kafka.bootstrap.servers: "broker1:9092,broker2:9092"
        subscribe: "products-topic"
      select_exp: ["id", "name", "price"]
      where_clause: "price > 0"
```

--------------------------------

### Event Listeners for Input and Enter Key

Source: https://github.com/databrickslabs/dlt-meta/blob/main/lakehouse_app/templates/landingPage.html

Attaches event listeners to the send button and the input field. The send button triggers sending input or starting a command. The Enter key in the input field also triggers the send button.

```javascript
sendBtn.addEventListener('click', () => {
  const input = inputField.value.trim();
  if (!input) return;
  if (!currentCommandId) {
    startCommand(input);
  } else {
    sendInput(input);
  }
});

inputField.addEventListener('keydown', (event) => {
  if (event.key === 'Enter') {
    sendBtn.click();
  }
});
```

--------------------------------

### Databricks Notebook Onboarding for Silver Layer

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/faq/execution.md

This Python code demonstrates how to onboard the silver layer using a Databricks notebook. It requires a spark session and a map of onboarding parameters.

```python
onboarding_params_map = {
                "database": "uc_name.dlt_demo",
                "onboarding_file_path": "dbfs:/dlt-meta/conf/onboarding.json",
                "silver_dataflowspec_table": "silver_dataflowspec_table", 
                "overwrite": "True",
                "env": "dev",
                "version": "v1",
                "import_author": "Ravi"
                }

        from src.onboard_dataflowspec import OnboardDataflowspec
        OnboardDataflowspec(spark, onboarding_params_map, uc_enabled=True).onboard_silver_dataflow_spec()
```

--------------------------------

### Programmatic Onboarding and Deployment with DLTMeta Python API

Source: https://context7.com/databrickslabs/dlt-meta/llms.txt

The `DLTMeta` class from the Databricks SDK allows programmatic onboarding and deployment. Use the `onboard()` and `deploy()` methods with `OnboardCommand` and `DeployCommand` objects for automated scripts and CI/CD.

```python
from databricks.sdk import WorkspaceClient
from src.cli import DLTMeta, OnboardCommand, DeployCommand

ws = WorkspaceClient(host="https://adb-xxx.azuredatabricks.net", token="dapiXXX")
dltmeta = DLTMeta(ws)

# --- Programmatic Onboard (Unity Catalog) ---
onboard_cmd = OnboardCommand(
    onboarding_file_path="demo/conf/onboarding.json",
    onboarding_files_dir_path="file:/path/to/dlt-meta/demo/",
    onboard_layer="bronze_silver",
    env="dev",
    import_author="ci_pipeline",
    version="v1",
    dlt_meta_schema="dlt_meta_dataflowspecs",
    uc_enabled=True,
    uc_catalog_name="main",
    serverless=True,
    bronze_dataflowspec_table="bronze_dataflowspec",
    silver_dataflowspec_table="silver_dataflowspec",
    overwrite=True,
)
dltmeta.onboard(onboard_cmd)
# Uploads configs to UC Volume, creates & runs onboarding job

# --- Programmatic Deploy ---
deploy_cmd = DeployCommand(
    layer="bronze_silver",
    pipeline_name="prod_dlt_meta_pipeline",
    dlt_target_schema="dlt_output",
    onboard_bronze_group="A1",
    onboard_silver_group="A1",
    dlt_meta_bronze_schema="dlt_meta_dataflowspecs",
    dlt_meta_silver_schema="dlt_meta_dataflowspecs",
    dataflowspec_bronze_table="bronze_dataflowspec",
    dataflowspec_silver_table="silver_dataflowspec",
    uc_catalog_name="main",
    uc_enabled=True,
    serverless=True,
)
dltmeta.deploy(deploy_cmd)
# Creates Lakeflow Declarative Pipeline and starts an update
```

--------------------------------

### Create Python Virtual Environment

Source: https://github.com/databrickslabs/dlt-meta/blob/main/README.md

Create a Python virtual environment for the dlt-meta project.

```commandline
python -m venv .venv
```

--------------------------------

### Run Append FLOW Autoloader Demo

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/demo/Append_FLOW_CF.md

Execute the demo script to launch the Append FLOW Autoloader demo. Configure parameters like cloud provider, Databricks Runtime version, DBFS path, and Unity Catalog name.

```command
python demo/launch_af_cloudfiles_demo.py --cloud_provider_name=aws --dbr_version=15.3.x-scala2.12 --dbfs_path=dbfs:/tmp/DLT-META/demo/ --uc_catalog_name=dlt_meta_uc
```

--------------------------------

### Launch DAIS Demo

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/demo/DAIS.md

Execute the Python script to launch the DAIS demo. Provide your Unity Catalog name and cloud provider. Optionally, specify a Databricks CLI profile.

```bash
python demo/launch_dais_demo.py --uc_catalog_name=<<uc catalog name>> --cloud_provider_name=<<>>
```

--------------------------------

### Launch Apply Changes From Snapshot Demo

Source: https://github.com/databrickslabs/dlt-meta/blob/main/demo/README.md

Executes the Apply Changes From Snapshot demo script. Requires specifying the Unity Catalog name and optionally a Databricks profile.

```commandline
python demo/launch_acfs_demo.py --uc_catalog_name=<<uc catalog name>> --profile=<<DEFAULT>>
```

--------------------------------

### Launch Silver Fanout Demo

Source: https://github.com/databrickslabs/dlt-meta/blob/main/demo/README.md

Executes the Silver Fanout demo script. Requires specifying the source type and optionally a Databricks profile.

```commandline
python demo/launch_silver_fanout_demo.py --source=cloudfiles --uc_catalog_name=<<uc catalog name>> --profile=<<DEFAULT>>
```

--------------------------------

### Initialize DLT CLI Interface

Source: https://github.com/databrickslabs/dlt-meta/blob/main/lakehouse_app/templates/landingPage.html

Sets up event listeners for command buttons, the send button, and the input field. Initializes the terminal with a welcome message and a prompt.

```javascript
document.getElementById("setupDlt").querySelectorAll('.command-button').forEach(button => {
    button.addEventListener('click', function () {
      const command = this.getAttribute('data-command');
      setCommand(command);
    });
  });
  document.getElementById("setupDlt").querySelectorAll('.command-button').forEach(button => {
    button.addEventListener('click', function () {
      const command = this.getAttribute('data-command');
      startCommand(command);
    });
  });
```

--------------------------------

### Databricks Notebook Onboarding for Bronze Layer

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/faq/execution.md

This Python code demonstrates how to onboard the bronze layer using a Databricks notebook. It requires a spark session and a map of onboarding parameters.

```python
onboarding_params_map = {
        "database": "uc_name.dlt_demo",
        "onboarding_file_path": "dbfs:/dlt-meta/conf/onboarding.json",
        "bronze_dataflowspec_table": "bronze_dataflowspec_table", 
        "overwrite": "True",
        "env": "dev",
        "version": "v1",
        "import_author": "Ravi"
        }

        from src.onboard_dataflowspec import OnboardDataflowspec
        OnboardDataflowspec(spark, onboarding_params_map, uc_enabled=True).onboard_bronze_dataflow_spec()
```

--------------------------------

### Launch Eventhub Append Flow Demo

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/demo/Append_FLOW_EH.md

Executes the Eventhub append flow demo script with specified configuration parameters. Ensure all required arguments, including cloud provider, Databricks Runtime version, Unity Catalog details, Eventhub specifics, and secret scope information, are correctly provided.

```commandline
python demo/launch_af_eventhub_demo.py --cloud_provider_name=aws --uc_catalog_name=dlt_meta_uc --eventhub_name=dltmeta_demo --eventhub_name_append_flow=dltmeta_demo_af --eventhub_secrets_scope_name=dltmeta_eventhub_creds --eventhub_namespace=dltmeta --eventhub_port=9093 --eventhub_producer_accesskey_name=RootManageSharedAccessKey --eventhub_consumer_accesskey_name=RootManageSharedAccessKey --eventhub_accesskey_secret_name=RootManageSharedAccessKey
```

--------------------------------

### Navigate to Project Directory

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/demo/Apply_Changes_From_Snapshot.md

Change the current directory to the root of the cloned dlt-meta project. This is necessary for subsequent commands.

```commandline
cd dlt-meta
```

--------------------------------

### Launch Append Flow Event Hubs Demo

Source: https://github.com/databrickslabs/dlt-meta/blob/main/demo/README.md

Executes the append flow Event Hubs demo script. Requires numerous arguments including Unity Catalog name, Event Hubs details, access key names, secret scope, and port.

```commandline
python3 demo/launch_af_eventhub_demo.py --uc_catalog_name=<<uc catalog name>> --eventhub_name=dltmeta_demo --eventhub_name_append_flow=dltmeta_demo_af --eventhub_secrets_scope_name=dltmeta_eventhub_creds --eventhub_namespace=dltmeta --eventhub_port=9093 --eventhub_producer_accesskey_name=RootManageSharedAccessKey --eventhub_consumer_accesskey_name=RootManageSharedAccessKey --eventhub_accesskey_secret_name=RootManageSharedAccessKey --profile=<<DEFAULT>>
```

--------------------------------

### Create and Activate Python Virtual Environment

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/contributing/onboarding/_index.md

Create and activate a Python virtual environment for the project. This isolates project dependencies.

```bash
python -m venv .venv
```

```bash
python3 -m venv .venv
```

```bash
source .venv/bin/activate
```

--------------------------------

### Launch Autoloader Cloudfiles Demo

Source: https://github.com/databrickslabs/dlt-meta/blob/main/demo/README.md

Executes the autoloader cloudfiles demo script. Requires specifying Unity Catalog name, source type (cloudfiles), and Databricks profile.

```commandline
python demo/launch_af_cloudfiles_demo.py --uc_catalog_name=<<uc catalog name>> --source=cloudfiles --profile=<<DEFAULT>>
```

--------------------------------

### Run Silver Fanout Demo Script

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/demo/Silver_Fanout.md

Execute the Python script to launch the silver fanout demo. Provide your Databricks Unity Catalog and cloud provider names as arguments. You can also specify a Databricks CLI profile.

```commandline
python demo/launch_silver_fanout_demo.py --uc_catalog_name=<<uc catalog name>> --cloud_provider_name=aws
```

--------------------------------

### Initialize Terminal Prompt

Source: https://github.com/databrickslabs/dlt-meta/blob/main/lakehouse_app/templates/landingPage.html

Appends initial welcome messages and the command prompt to the terminal when the interface loads.

```javascript
// Initialize with a prompt
appendToTerminal('Welcome to the Databricks DLT-META CLI Browser Interface', 'input-line');
appendToTerminal('Type a command or click one of the suggestions below', 'input-line');
appendToTerminal('$ ', 'prompt');
```

--------------------------------

### Run Integration Test for Snapshot

Source: https://github.com/databrickslabs/dlt-meta/blob/main/integration_tests/README.md

Execute integration tests for the snapshot source. Requires a Unity Catalog catalog name.

```commandline
python integration_tests/run_integration_tests.py --source=snapshot --uc_catalog_name=<<uc catalog name>> --profile=<<DEFAULT>>
```

--------------------------------

### Create a Databricks Custom App

Source: https://github.com/databrickslabs/dlt-meta/blob/main/docs/content/getting_started/app.md

This command initiates the creation of a custom application on your Databricks workspace. The process may take a few minutes to complete.

```commandline
databricks apps create demo-dltmeta
```