### SQL Splitter Example Command

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/sql_splitter.mdx

An example command to run the SQL Splitter. It specifies an input SQL file and an output directory that must exist prior to execution.

```bash
./sqlsplit -i /project1/inputs/all_objects.sql -o /path/to/output
```

--------------------------------

### Complete Custom Configuration File Example

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/transpile/pluggable_transpilers/bladebridge/bladebridge_configuration.mdx

This example demonstrates a complete custom configuration file ('my_custom_config.json') that includes inheritance, a target SQL file header, and schema replacements for transpilation.

```json
{
    "inherit_from": [
      "/Users/user.name/.databricks/labs/remorph-transpilers/bladebridge/lib/.venv/lib/python3.10/site-packages/databricks/labs/bladebridge/Converter/Configs/base_mssql2databricks_sql.json"
    ],
    "target_sql_file_header": "USE CATALOG catalog01;",
    "line_subst": [
      { "from": "\\bschema01\\b\\.", "to": "gold." },
      { "from": "\\bschema02\\b\\.", "to": "silver." }
    ]
}
```

--------------------------------

### Install Documentation Dependencies

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/dev/index.mdx

Installs the necessary dependencies for the Lakebridge documentation.

```bash
make docs-install
```

--------------------------------

### Install Transpile module

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/installation.mdx

Set up the transpilation component, optionally including the LLM-based transpiler.

```bash
databricks labs lakebridge install-transpile
```

```bash
databricks labs lakebridge install-transpile --include-llm-transpiler true
```

```bash
databricks labs lakebridge transpile --help
```

--------------------------------

### Manual Transpiler Version JSON

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/transpile/pluggable_transpilers/index.mdx

Example JSON content for the version.json file used in manual transpiler installations.

```json
{"version": "v1.3.7", "date": "2025-03-17-15:02:31Z"}
```

--------------------------------

### Install Switch LLM Transpiler

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/transpile/pluggable_transpilers/switch/index.mdx

Install the Switch transpiler using the `install-transpile` command with the `--include-llm-transpiler` option. This command uploads necessary notebooks and creates a Databricks Job for conversions.

```bash
databricks labs lakebridge install-transpile --include-llm-transpiler true --profile profile_name
```

--------------------------------

### JVM Proxy Configuration Example

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/dev/contributing.md

Example of the .databricks-login.json file required for the JVM proxy. Ensure this file is created with your workspace profile and cluster ID.

```json
{
  "workspace_profile": "labs-azure-tool",
  "cluster_id": "0708-200540-wcwi4i9e"
}
```

--------------------------------

### Install Yarn with Homebrew

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/dev/index.mdx

Installs Node.js and Yarn globally on macOS using Homebrew.

```bash
brew install node
npm install --global yarn
```

--------------------------------

### Install Lakebridge

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/installation.mdx

Install the Lakebridge extension for the Databricks CLI.

```bash
databricks labs install lakebridge
```

```bash
databricks labs install lakebridge --profile <profile_name>
```

```bash
databricks labs lakebridge --help
```

--------------------------------

### SamplingOptions Configuration Examples

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/reconcile/configuration.mdx

Various configurations for random and stratified sampling methods.

```json
"sampling_options": {
  "method": "random",
  "specifications": { "type": "count", "value": 500 }
}
```

```json
"sampling_options": { "specifications": { "value": 500 } }
```

```json
"sampling_options": {
  "method": "stratified",
  "specifications": { "type": "count", "value": 500 },
  "stratified_columns": ["ss_store_sk"],
  "stratified_buckets": 10
}
```

--------------------------------

### Transpile with Default Configuration

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/transpile/index.mdx

Run the transpile process using pre-configured inputs from the installation. This is a simplified command for environments where all settings are already defined.

```bash
databricks labs lakebridge transpile
```

--------------------------------

### Install uv using Homebrew

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/dev/contributing.md

Command to install the 'uv' package manager using Homebrew. This is a prerequisite for setting up the Python development environment.

```shell
brew install uv
```

--------------------------------

### Set up Development Environment

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/dev/contributing.md

Command to set up the development environment for the project. This command installs all necessary libraries and project dependencies into a virtual environment named .venv.

```shell
make dev
```

--------------------------------

### Install Databricks CLI on Linux

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/faq.mdx

Use this script to install the Databricks CLI on Linux systems without Homebrew.

```bash
#!/usr/bin/env bash
apt update && apt install -y curl sudo unzip
curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/v0.299.0/install.sh | sudo sh
```

--------------------------------

### Verify Python and Java installation

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/installation.mdx

Check that the required runtime environments are correctly installed and accessible in the terminal.

```console
python -V
java -version
```

--------------------------------

### Verify Lakebridge Analyzer Installation

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/assessment/analyzer/index.mdx

Execute this command to verify the Lakebridge Analyzer installation. Successful installation is confirmed when the output matches the expected help information.

```bash
databricks labs lakebridge analyze --help
```

```console
Analyze existing non-Databricks database or ETL sources

Usage:
    databricks labs lakebridge analyze [flags]

Flags:
    --generate-json true|false    (Optional) Generate JSON file alongside the Excel report
    -h, --help                help for analyze
    --report-file path        (Optional) Local filesystem path of the analysis report file to write
    --source-directory path   (Optional) Local filesystem path of a directory containing sources to analyze
    --source-tech string      (Optional) The technology/platform of the sources to analyze

Global Flags:
  --debug            enable debug logging
  -o, --output type      output type: text or json (default text)
  -p, --profile string   ~/.databrickscfg profile
  -t, --target string    bundle target to use (if applicable)
 
```

--------------------------------

### Install Lakebridge Reconcile using PyPI

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/reconcile/running.mdx

Install the Lakebridge library within a Databricks notebook environment using pip. Restart the Python environment after installation.

```python
%pip install databricks-labs-lakebridge
dbutils.library.restartPython()
```

--------------------------------

### Sample Perl Implementation for init_hooks

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/transpile/pluggable_transpilers/bladebridge/bladebridge_configuration.mdx

Example Perl subroutine to handle the initialization hook. It receives configuration entries and the converter instance, printing them for debugging.

```perl
sub init_hooks {
  my $param = shift;
  %CFG = %{$param->{CONFIG}};
  $CONVERTER = $param->{CONVERTER};
  print "INIT_HOOKS Called. config:\n" . Dumper(%CFG);
}
```

--------------------------------

### Install Databricks CLI

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/installation.mdx

Commands to install the Databricks CLI on various operating systems.

```shell
brew tap databricks/tap
brew install databricks
```

```shell
curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/v0.299.0/install.sh
```

```bash
#!/usr/bin/env bash
apt update && apt install -y curl sudo unzip
curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/v0.299.0/install.sh | sudo sh
```

--------------------------------

### CREATE TABLE LIKE - Redshift Syntax

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/transpile/source_systems/redshift.mdx

Example of creating a table like another table in Redshift SQL, including default values.

```sql
CREATE TABLE customers_backup (LIKE customers INCLUDING DEFAULTS);
```

--------------------------------

### Execute BigQuery Profiler

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/assessment/profiler/bigquery.mdx

Run this command to start the BigQuery profiler. It connects to BigQuery using the configured settings and generates analysis tables in a local DuckDB database.

```bash
databricks labs lakebridge execute-database-profiler --source-tech bigquery
```

--------------------------------

### Configure ReconcileConfig Properties

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/reconcile/running.mdx

Example configuration for ReconcileConfig, specifying source and target connection details, and metadata storage.

```python
reconcile_config = ReconcileConfig(
        report_type="all",
        source=SourceConnectionConfig(
            dialect="snowflake",
            catalog="source_sf_catalog",
            schema="source_sf_schema",
            uc_connection_name="source_connection_name"
        ),
        target=TargetConnectionConfig(
            catalog="target_databricks_catalog",
            schema="target_databricks_schema",
        ),
        metadata_config = ReconcileMetadataConfig(
              catalog = "lakebridge_metadata",
              schema= "reconcile"
         ),
    )
```

--------------------------------

### Configure BigQuery Connection

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/assessment/profiler/bigquery.mdx

Run this command to start the interactive configuration process for BigQuery. You will be prompted to enter connection details, lookback window, and other profiler settings. The configuration is saved to `~/.databricks/labs/lakebridge/.credentials.yml`.

```console
databricks labs lakebridge configure-database-profiler

Please select the source system you want to configure
[0] bigquery
[1] mssql
[2] synapse
Enter a number between 0 and 2: 0

(local | env)
local means values are read as plain text
env means values are read from environment variables, and fall back to plain text if not variable is not found

Enter secret vault type (local | env)
[0] env
[1] local
Enter a number between 0 and 1: 1
Please provide BigQuery connection settings:

Enter BigQuery project and region pairs (Format: comma-separated project.region. Example: my-proj-a.us, my-proj-b.eu-west-1): customer-prod-1.us, customer-admin.us
Enter lookback window in days to profile (default: 180):
Enter max parallel SQLs per (project, region) iteration (default: 8):
Please configure profiler settings:
Exclude reservations and commitments data? (default: no):
Exclude streaming and write API summary? (default: no):
```

--------------------------------

### Example Function Call with Arguments

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/transpile/pluggable_transpilers/bladebridge/bladebridge_configuration.mdx

Illustrates how arguments are represented within a function call like SUBSTR, showing positional mapping for 'arg $1', 'arg $2', and 'arg $3'.

```sql
SELECT SUBSTR(
	UPPER( first_name || ' ' last_name ), -- arg $1
	10, -- arg $2
	20 -- arg $3
	)
```

--------------------------------

### Run Tests

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/dev/contributing.md

Command to run the project's tests. This is used to verify the installation and the integrity of the codebase.

```shell
make test
```

--------------------------------

### Redshift Configuration Transcript

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/assessment/profiler/redshift.mdx

An illustrative example of the interactive prompts encountered during the Redshift configuration process.

```console
databricks labs lakebridge configure-database-profiler

Please select the source system you want to configure
[0] bigquery
[1] legacy_synapse
[2] mssql
[3] oracle
[4] redshift
[5] snowflake
[6] synapse
[7] teradata
Enter a number ...: 4

Redshift authentication: sql_authentication or iam.

Authentication type
[0] iam
[1] sql_authentication
Enter a number between 0 and 1: 1
Credential source (local | env | file)
[0] local
[1] env
[2] file
...

Enter the Redshift cluster endpoint (host): mycluster.abc123.us-east-1.redshift.amazonaws.com
Enter the port details (default: 5439): 5439
Enter the database name: dev
Enter the user details: profiler_reader
Enter the password details: ********

Do you want to test the connection to redshift? [y/n]: y
```

--------------------------------

### Successful Transpile Output Header

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/transpile/pluggable_transpilers/morpheus/index.mdx

Example of a header comment in a successfully transpiled SQL file, indicating the source file path.

```sql
/*
    Successfully transpiled from /path/to/input/my_proc.sql
*/
CREATE PROCEDURE ...
```

--------------------------------

### Configure Standard Reconciliation

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/reconcile/configuration.mdx

Example configuration for standard table reconciliation including join columns, transformations, and thresholds.

```json
{
  "tables": [
    {
      "source_name": "product_prod",
      "target_name": "product",
      "jdbc_reader_options": {
        "num_partitions": 10,
        "partition_column": "p_id",
        "lower_bound": "0",
        "upper_bound": "10000000"
      },
      "join_columns": ["p_id"],
      "drop_columns": ["comment"],
      "column_mapping": [
        { "source_name": "p_id", "target_name": "product_id" },
        { "source_name": "p_name", "target_name": "product_name" }
      ],
      "transformations": [
        {
          "column_name": "creation_date",
          "source": "creation_date",
          "target": "to_date(creation_date,'yyyy-mm-dd')"
        }
      ],
      "column_thresholds": [
        { "column_name": "price", "lower_bound": "-50", "upper_bound": "50", "type": "float" }
      ],
      "table_thresholds": [
        { "lower_bound": "0%", "upper_bound": "5%", "model": "mismatch" }
      ],
      "filters": {
        "source": "p_id > 0",
        "target": "product_id > 0"
      },
      "sampling_options": {
        "method": "random",
        "specifications": { "type": "count", "value": 100 }
      }
    }
  ]
}
```

--------------------------------

### Configure reconciliation for various data sources

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/reconcile/configuration.mdx

YAML configuration examples for different source dialects including Snowflake, Oracle, MS SQL Server, Teradata, BigQuery, and Databricks.

```yaml
source:
    dialect: snowflake
    catalog: sample_data
    schema: default
    uc_connection_name: example_connection_snowflake
target:
    catalog: migrated
    schema: migrated
report_type: all
```

```yaml
source:
    dialect: oracle
    uc_connection_name: example_connection_oracle
target:
    catalog: migrated
    schema: migrated
report_type: data
```

```yaml
source:
    dialect: mssql
    uc_connection_name: example_connection_mssql
target:
    catalog: migrated
    schema: migrated
report_type: data
```

```yaml
source:
    dialect: teradata
    catalog: DBC
    schema: lakebridge_test
    uc_connection_name: teradata_sandbox
target:
    catalog: migrated
    schema: migrated
report_type: all
```

```yaml
source:
    dialect: bigquery
    catalog: bq_project_name
    schema: sample_dataset
    uc_connection_name: example_connection_bigquery
target:
    catalog: migrated
    schema: migrated
report_type: all
```

```yaml
source:
    dialect: databricks
    catalog: hive_metastore
    schema: hr
target:
    catalog: migrated
    schema: migrated
  ...
report_type: data
```

--------------------------------

### Configure Aggregate Reconciliation

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/reconcile/configuration.mdx

Example configuration for reconciling aggregate metrics between source and target tables.

```json
{
  "tables": [
    {
      "source_name": "product_prod",
      "target_name": "product",
      "join_columns": ["p_id"],
      "aggregates": [
        { "type": "MIN", "agg_columns": ["discount"], "group_by_columns": ["p_id"] },
        { "type": "AVG", "agg_columns": ["discount"], "group_by_columns": ["p_id"] },
        { "type": "MAX", "agg_columns": ["p_id"], "group_by_columns": ["creation_date"] },
        { "type": "SUM", "agg_columns": ["p_id"] }
      ],
      "jdbc_reader_options": {
        "num_partitions": 10,
        "partition_column": "p_id",
        "lower_bound": "0",
        "upper_bound": "10000000"
      }
    }
  ]
}
```

--------------------------------

### CREATE VIEW - Redshift Syntax

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/transpile/source_systems/redshift.mdx

Example of a CREATE VIEW statement in Redshift SQL, demonstrating string concatenation and date difference calculation.

```sql
CREATE VIEW active_customers AS
SELECT
    customer_id,
    first_name || ' ' || last_name AS full_name,
    email,
    DATEDIFF(year, birth_date, CURRENT_DATE) AS age
FROM customers
WHERE status = 'active';
```

--------------------------------

### Transform Timestamp Columns

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/reconcile/configuration.mdx

Example of converting timestamp columns to Unix epoch strings for consistent cross-platform comparison.

```python
Transformation(
    column_name='UPDATE_TIMESTAMP',
    source="coalesce(cast(EXTRACT(epoch_millisecond FROM UPDATE_TIMESTAMP) as string), '_null_recon_')",
    target="coalesce(cast(unix_millis(UPDATE_TIMESTAMP) as string), '_null_recon_')"
)
```

--------------------------------

### Example Workflow JSON

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/transpile/source_systems/datastage.mdx

This JSON defines a sample Databricks workflow with three tasks: Extract_Data, Transform_Data, and Send_Notification, illustrating task dependencies and execution.

```json
{
  "name": "SEQX_ETL_DAILY_LOAD",
  "tasks": [
    {
      "task_key": "Extract_Data",
      "notebook_task": {
        "notebook_path": "/Workspace/Jobs/PXJ_EXTRACT_DATA",
        "source": "WORKSPACE"
      }
    },
    {
      "task_key": "Transform_Data",
      "depends_on": [
        { "task_key": "Extract_Data" }
      ],
      "run_if": "ALL_SUCCESS",
      "notebook_task": {
        "notebook_path": "/Workspace/Jobs/PXJ_TRANSFORM_DATA",
        "source": "WORKSPACE"
      }
    },
    {
      "task_key": "Send_Notification",
      "depends_on": [
        { "task_key": "Transform_Data" }
      ],
      "spark_python_task": {
        "python_file": "/Workspace/Jobs/SHELL_SEND_EMAIL.py"
      }
    }
  ]
}
```

--------------------------------

### Install Transpile with Custom Configuration

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/transpile/pluggable_transpilers/bladebridge/bladebridge_configuration.mdx

When running the Bladebridge converter from Lakebridge, a custom configuration file can be supplied to the converter during the install-transpile process. Specify the path to your custom configuration file when prompted.

```bash
databricks labs lakebridge install-transpile
Do you want to override the existing installation? (default: no): yes
Specify the config file to override the default[Bladebridge] config - press <enter> for none (default: <none>):
<local_full_path>/custom_<source>2databricks.json
```

--------------------------------

### CREATE TABLE - Redshift Syntax

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/transpile/source_systems/redshift.mdx

Example of a CREATE TABLE statement in Redshift SQL. Note the use of IDENTITY, PRIMARY KEY, UNIQUE, DEFAULT, DISTSTYLE, DISTKEY, and SORTKEY.

```sql
CREATE TABLE customers (
    customer_id INT IDENTITY(1,1) PRIMARY KEY,
    first_name VARCHAR(50) NOT NULL,
    last_name VARCHAR(100) NOT NULL,
    email VARCHAR(255) UNIQUE,
    birth_date DATE,
    registration_date TIMESTAMP DEFAULT GETDATE(),
    loyalty_points DECIMAL(10,2) DEFAULT 0,
    status VARCHAR(20)
)
DISTSTYLE KEY
DISTKEY (customer_id)
SORTKEY (registration_date, customer_id);
```

--------------------------------

### Switch LLM Transpile Command Output

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/transpile/pluggable_transpilers/switch/index.mdx

Example console output when running the `llm-transpile` command. It indicates file upload status, job triggering, and provides a URL to monitor the asynchronous Switch LLM transpilation job.

```console
INFO [d.l.l.transpiler.switch_runner] Uploading /local/path/to/input to your_volume
INFO [d.l.l.transpiler.switch_runner] Upload complete: your_volume/input-xyz
INFO [d.l.l.transpiler.switch_runner] Triggering Switch job with job_id: <switch_job_id>
INFO [d.l.l.transpiler.switch_runner] Switch LLM transpilation job started: https://workspace.databricks.com/jobs/switch_job_id/runs/run_id
```

--------------------------------

### BladeBridge Transpilation for ETL Sources with Target Technology

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/transpile/pluggable_transpilers/bladebridge/bladebridge_configuration.mdx

This example demonstrates transpiling Datastage ETL exports to Spark SQL. Specify the source dialect, input path, output folder, and the desired target technology.

```bash
databricks labs lakebridge transpile \
  --source-dialect datastage \
  --input-source /path/to/datastage/exports \
  --output-folder /path/to/output \
  --target-technology SPARKSQL
```

--------------------------------

### Custom Conversion Prompt YAML Structure

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/transpile/pluggable_transpilers/switch/customizing_switch.mdx

Define custom conversion prompts using a YAML file. Include system messages for conversion instructions and optional few-shot examples to guide the LLM. The system message can include placeholders for comment language and common guidelines.

```yaml
system_message: |
  Convert SQL code to Python code that runs on Databricks according to the following instructions:

  # Input and Output
  - Input: A single SQL file containing one or multiple T-SQL statements
  - Output: Python code with Python comments (in {comment_lang}) explaining the code

  ${common_python_instructions_and_guidelines}

  # Additional Instructions
  1. Convert SQL queries to spark.sql() format
  2. Add clear Python comments explaining the code
  3. Use DataFrame operations instead of loops when possible
  4. Handle errors using try-except blocks

few_shots:
- role: user
  content: |
    SELECT name, age
    FROM users
    WHERE active = 1;
- role: assistant
  content: |
    # Get names and ages of active users
    active_users = spark.sql("""
        SELECT name, age
        FROM users
        WHERE active = 1
    """)
    display(active_users)
```

--------------------------------

### Serve Documentation in Development Mode

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/dev/index.mdx

Runs the documentation development server, enabling live reloading for quick previews.

```bash
make docs-serve-dev
```

--------------------------------

### Review and Auto-configure Tables with Python

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/reconcile/running.mdx

This Python code demonstrates the recommended flow: first discover tables, then curate the discovered list in memory before applying auto-configuration.

```python
discovered = discover_tables(reconcile_config=reconcile_config, spark=spark)

# Review/curate in memory — or save the file, edit it in the workspace, and load it back.
curated = TableRecon(tables=[t for t in discovered.tables if t.source_name != "audit_log"])

table_recon = auto_configure_tables(curated, reconcile_config=reconcile_config, spark=spark)
```

--------------------------------

### Serve Documentation for Search Testing

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/dev/index.mdx

Runs the documentation server for testing search functionality. Note that local search is not available in the development server.

```bash
make docs-serve
```

--------------------------------

### Build Documentation

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/dev/index.mdx

Builds the static documentation site. This command also checks for unresolved links.

```bash
make docs-build
```

--------------------------------

### Lakebridge Script Debugging Output Example

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/dev/contributing.md

Example output from debugging a Lakebridge script, showing log levels and script debugging information.

```text
databricks labs lakebridge debug-script --name foo
21:57:42  INFO [databricks.sdk] Using Azure CLI authentication with AAD tokens
21:57:42  WARN [databricks.sdk] azure_workspace_resource_id field not provided. It is recommended to specify this field in the Databricks configuration to avoid authentication errors.
Debugging script...
Map(log_level -> disabled, name -> foo)
```

--------------------------------

### Profiler command help output

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/assessment/profiler/index.mdx

Displays the usage instructions and available flags for the execute-database-profiler command.

```console
Profile the source system database

Usage:
  databricks labs lakebridge execute-database-profiler [flags]

Flags:
  -h, --help                 help for execute-database-profiler
      --source-tech string   (Optional) The technology/platform of the sources to Profile

Global Flags:
      --debug            enable debug logging
  -o, --output type      output type: text or json (default text)
  -p, --profile string   ~/.databrickscfg profile
  -t, --target string    bundle target to use (if applicable)
```

--------------------------------

### BladeBridge line_subst Rule Example

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/transpile/pluggable_transpilers/bladebridge/bladebridge_configuration.mdx

This example demonstrates a line substitution rule in BladeBridge configuration. It captures a pattern enclosed in '#' and transforms it to a different format, upcasing the captured string.

```json
{"from": "#(\\w+)#", "to": "${$1}", "upcase_string": true}
```

--------------------------------

### Configure MSSQL Profiler Connection

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/assessment/profiler/mssql.mdx

Run this command to initiate the interactive configuration wizard for the MSSQL profiler.

```console
databricks labs lakebridge configure-database-profiler

Please select the source system you want to configure
[0] synapse
[1] mssql
Enter a number between 0 and 1: 1

Enter secret vault type (local | env)
[0] env
[1] local
Enter a number between 0 and 1: 1
Select authentication method
[0] SqlPassword
[1] DefaultAzureCredential
[2] ActiveDirectoryPassword
[3] ActiveDirectoryServicePrincipal
Enter a number between 0 and 3: 0
Enter the username: profiler_user
Enter the password:
Enter fetch size (default: 1000):
Enter login timeout (seconds) (default: 30):
Enter the fully-qualified server name: my-server.database.windows.net
Enter the port details: 1433
Enter the database name: MyAppDB
Trust server certificate (default: no):
Enter timezone (e.g. America/New_York) (default: UTC):
Do you want to test the connection to mssql? (yes/no): yes
```

--------------------------------

### SQL Splitter Command-line Usage

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/sql_splitter.mdx

Displays the available command-line options for the sql_splitter utility. Use this to understand how to configure input files/folders, output directories, and various processing flags.

```bash
Usage:
sql_splitter
    -h                          this message
    <OPTIONS>

    -i input file OR comma-separated list of files
       OR
    -d input folder

    -o output folder

    [-s]                        plug in newline after semicolon
    [-E extensions]             default is sql
    [-t]                        trim lines from both sides
    [-b]                        remove square brackets
    [-P]                        do not add package variables to procedures and functions
    [-G pattern]                custom object separator pattern
    [-r <reportfile><ext>]      generate <reportfile><ext> file and <reportfile>_duplicates<ext> file
                                 (or if `-r -` is specified, output to STDOUT)
    [-v]                        verbose mode
```

--------------------------------

### Configure the Snowflake Profiler

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/assessment/profiler/snowflake.mdx

Initializes the profiler configuration by selecting Snowflake and providing connection details.

```bash
databricks labs lakebridge configure-database-profiler
```

--------------------------------

### Configure Synapse Profiler via CLI

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/assessment/profiler/synapse.mdx

Interactive command to initiate the configuration wizard for the Synapse database profiler.

```console
databricks labs lakebridge configure-database-profiler

Please select the source system you want to configure
[0] synapse
Enter a number between 0 and 0: 0

(local | env)
local means values are read as plain text
env means values are read from environment variables, and fall back to plain text if not variable is not found

Enter secret vault type (local | env)
[0] env
[1] local
Enter a number between 0 and 1: 1
Select authentication method
[0] SqlPassword
[1] DefaultAzureCredential
[2] ActiveDirectoryPassword
[3] ActiveDirectoryServicePrincipal
Enter a number between 0 and 3: 0
Enter the username: user
Enter the password:
Please provide Synapse Workspace settings:
Enter Synapse workspace name: synapse
Enter development endpoint: synapse.endpoint
Enter fetch size (default: 1000):
Enter login timeout (seconds) (default: 30):
Enter timezone (e.g. America/New_York) (default: UTC):
Exclude serverless SQL pool from profiling? (default: no):
Exclude dedicated SQL pools from profiling? (default: no):
Exclude Spark pools from profiling? (default: no):
Exclude monitoring metrics from profiling? (default: no):
Redact SQL pools SQL text? (default: no):
```

--------------------------------

### Execute the database profiler

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/assessment/profiler/index.mdx

Use this command to run the profiler or view available flags and options.

```bash
databricks labs lakebridge execute-database-profiler --help
```

--------------------------------

### Sample Perl Implementation for prescan_code_oracle

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/transpile/pluggable_transpilers/bladebridge/bladebridge_configuration.mdx

Example Perl subroutine for the pre-scan hook. It receives the filename and configuration, printing a message to indicate its execution.

```perl
sub prescan_code_oracle {
  my $filename = shift;
  my $cf = shift;
  print "******** prescan_code_oracle $filename *********\n";
  # Open and analyze the file...
}
```

--------------------------------

### Configure Legacy Synapse Profiler Connection

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/assessment/profiler/legacy_synapse.mdx

Run this command in your terminal to initiate the interactive configuration wizard for the legacy Synapse profiler.

```console
databricks labs lakebridge configure-database-profiler

Please select the source system you want to configure
[0] synapse
[1] mssql
[2] legacy_synapse
Enter a number between 0 and 2: 2

(local | env)
local means values are read as plain text
env means values are read from environment variables, and fall back to plain text if not variable is not found

Enter secret vault type (local | env)
[0] env
[1] local
Enter a number between 0 and 1: 1
Select authentication method
[0] SqlPassword
[1] DefaultAzureCredential
[2] ActiveDirectoryPassword
[3] ActiveDirectoryServicePrincipal
Enter a number between 0 and 3: 0
Enter the username: profiler_user
Enter the password:
Enter fetch size (default: 1000):
Enter login timeout (seconds) (default: 30):
Enter the fully-qualified server name: my-dw-server.database.windows.net
Enter the port details: 1433
Enter the database name: my_pool
Trust server certificate (default: no):
Enter timezone (e.g. America/New_York) (default: UTC):
Enter the Azure subscription ID: 00000000-0000-0000-0000-000000000000
Enter the Azure resource group: rg-analytics
```

--------------------------------

### Transpile with Errors/Warnings Output Header

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/transpile/pluggable_transpilers/morpheus/index.mdx

Example of a header comment for a file that failed transpilation, listing specific errors found with line and column numbers.

```sql
/*
   Failed transpilation of /path/to/input/broken.sql

   The following errors were found while transpiling:
     - [3:5] Function GIBBERISH cannot be translated to Databricks SQL
*/
SELECT
  t1.name,
  GIBBERISH(t1.comment)
FROM t1;
```

--------------------------------

### Configure Git for GPG Signing

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/dev/contributing.md

Set up your local Git environment to automatically sign commits using your GPG key. Ensure your GPG key is listed and configured globally.

```shell
git config --global --unset gpg.format
gpg --list-secret-keys --keyid-format=long
git config --global user.signingkey <KEY.ID.FROM.ABOVE>
git config --global commit.gpgsign true
```

--------------------------------

### Configure Reconcile module

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/installation.mdx

Initialize the Reconcile module and configure workspace resources.

```bash
databricks labs lakebridge configure-reconcile
```

```text
[profile-name]
host         = <your-workspace-url>
warehouse_id = <your-warehouse-id>
cluster_id = <your-cluster-id>
```

```bash
databricks labs lakebridge reconcile --help
```

--------------------------------

### CREATE TABLE AS SELECT (CTAS) - Redshift Syntax

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/transpile/source_systems/redshift.mdx

Example of a CREATE TABLE AS SELECT (CTAS) statement in Redshift SQL for creating a summary table from order data.

```sql
CREATE TABLE customer_summary AS
SELECT
    customer_id,
    COUNT(*) AS order_count,
    SUM(total_amount) AS total_spent
FROM orders
GROUP BY customer_id;
```

--------------------------------

### Configure Teradata Database Profiler

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/assessment/profiler/teradata.mdx

Interactive command-line interface for setting up the Teradata database profiler connection.

```console
databricks labs lakebridge configure-database-profiler

Select the source technology
[0] bigquery
[1] legacy_synapse
[2] mssql
[3] oracle
[4] redshift
[5] snowflake
[6] synapse
[7] teradata
Enter a number between 0 and 7: 7

(local | env)
local means values are read as plain text
env means values are read from environment variables fall back to plain text if not variable is not found

Enter secret vault type (local | env)
[0] env
[1] local
Enter a number between 0 and 1: 1
Enter the Teradata server or host details: td-host.example.com
Enter the user details: dbc_user
Enter the password details:
Enter the default database name (default: DBC):
```

--------------------------------

### Configure Reconcile CLI

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/reconcile/index.mdx

Initializes Lakebridge workspace resources required for reconciliation.

```bash
databricks labs lakebridge configure-reconcile
```

--------------------------------

### Transpile with Full Arguments

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/transpile/index.mdx

Execute the transpile process by providing all necessary arguments directly in the command. This allows for granular control over the transpilation settings.

```bash
databricks labs lakebridge transpile --transpiler-config-path <absolute-path> --input-source <absolute-path> --source-dialect <snowflake> --output-folder <absolute-path> --skip-validation <True|False> --catalog-name <catalog name> --schema-name <schema name>
```

--------------------------------

### Run Switch LLM Transpile Command

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/transpile/pluggable_transpilers/switch/index.mdx

Execute the `llm-transpile` command to initiate Switch conversions. This command reads local input files, uploads them to Unity Catalog Volumes, and processes them using a Databricks Job. Specify source dialect, output workspace folder, and optional catalog, schema, volume, foundation model, and configuration path.

```bash
databricks labs lakebridge llm-transpile \
  --input-source /local/path/to/input \
  --output-ws-folder /Workspace/path/to/output \
  --source-dialect mysql \
  --accept-terms true \
  [--catalog-name your_catalog] \
  [--schema-name your_schema] \
  [--volume your_volume] \
  [--foundation-model your_foundation_model] \
  [--switch-config-path /Workspace/path/to/switch_config.yml] \
  [--profile profile_name]
```

--------------------------------

### PySpark Checksum and Deduplication

Source: https://github.com/databrickslabs/lakebridge/blob/main/docs/lakebridge/docs/transpile/source_systems/datastage.mdx

Converts DataStage checksum and deduplication nodes to PySpark using DataFrame API. This example includes source, checksum, unique row, merge, and transformation nodes.

```python
# Databricks notebook source
# Code converted on 2025-09-30 09:01:47
import os
from pyspark.sql import *
from pyspark.sql.functions import *

# COMMAND ----------
# Processing node DSLink4, type SOURCE
DSLink4 = spark.sql(f"SELECT SRKFIL, CODFIL, DESFIL, DATCADFIL FROM DIMFIL")

# COMMAND ----------
# Processing node DSLink5, type CHECKSUM
DSLink5 = DSLink4.select(
    DSLink4.SRKFIL.alias('SRKFIL'),
    DSLink4.CODFIL.alias('CODFIL'),
    md5(concat_ws('|', DSLink4.DESFIL, DSLink4.DATCADFIL)).alias('CurrentChecksum')
)

# COMMAND ----------
# Processing node DSLink3, type UNIQUE_ROW (Remove Duplicates)
DSLink3 = DSLink2.dropDuplicates(['CODFIL'])

# COMMAND ----------
# Processing node Matched, type MERGE
Matched = DSLink3.join(
    DSLink5, 
    [DSLink5.CODFIL == DSLink3.CODFIL],
    'LEFT_OUTER'
).select(
    DSLink3.CODFIL.alias('CODFIL'),
    DSLink3.DESFIL.alias('DESFIL'),
    DSLink5.SRKFIL.alias('SRKFIL'),
    DSLink5.CurrentChecksum.alias('CurrentChecksum')
)

# COMMAND ----------
# Processing node Updates, type TRANSFORMATION with filter
Updates = DSLink7.filter("CurrentChecksum <> UpdatesChecksum").select(
    DSLink7.SRKFIL.alias('SRKFIL'),
    DSLink7.CODFIL.alias('CODFIL'),
    DSLink7.DESFIL.alias('DESFIL')
)
```