### Widget Parameters for Setup Notebooks

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/DemoStructure.md

Example of how to define and get widget parameters in setup notebooks, specifically for resetting data.

```python
dbutils.widgets.dropdown("reset_all_data", "false", ["true", "false"])
reset_all_data = dbutils.widgets.get("reset_all_data") == "true"
```

--------------------------------

### Setup Pattern

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/CLAUDE.md

Example of how to run the setup notebook for a demo.

```python
%run ./_resources/00-setup $catalog=<catalog> $db=<schema>
```

--------------------------------

### Example Notebook Configurations

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/BundleConfiguration.md

Example list of notebook configurations for a demo.

```python
"notebooks": [
    {
        "path": "00-Introduction",
        "pre_run": False,
        "publish_on_website": True,
        "title": "Introduction & Architecture",
        "description": "Overview of the retail lakehouse solution"
    },
    {
        "path": "_resources/00-setup",
        "pre_run": True,
        "publish_on_website": False,
        "title": "Setup",
        "description": "Initialize demo environment"
    }
]
```

--------------------------------

### Volume Path Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/configuration.md

An example of a constructed volume path.

```bash
/Volumes/main__build/dbdemos_my_demo/raw_data/input_data
```

--------------------------------

### BundleConfiguration Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/BundleConfiguration.md

A basic example of BundleConfiguration.

```json
{
    "init_job": {},
    "cluster": {},
    "pipelines": []
}
```

--------------------------------

### Notebooks Configuration Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/configuration.md

Example configuration for notebooks within a bundle.

```json
"notebooks": [
    {
        "path": "01-Notebook-Name",
        "pre_run": False,
        "publish_on_website": True,
        "add_cluster_setup_cell": False,
        "title": "Human Readable Title",
        "description": "What this notebook does"
    }
]
```

--------------------------------

### Example Dashboard Configurations

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/BundleConfiguration.md

Example list of dashboard configurations for a demo.

```python
"dashboards": [
    {
        "name": "[dbdemos] Retail - Customer 360",
        "id": "retail-c360"
    },
    {
        "name": "[dbdemos] Retail - Inventory",
        "id": "retail-inventory"
    }
]
```

--------------------------------

### Notebook Configuration Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/types.md

An example of a notebook configuration object.

```json
{
    "path": "01-Data-Ingestion/01-Setup-SDP-pipeline",
    "pre_run": False,
    "publish_on_website": True,
    "add_cluster_setup_cell": False,
    "title": "Data Ingestion Pipeline",
    "description": "Set up Spark Declarative Pipeline for data ingestion"
}

```

--------------------------------

### Experiment Path Pattern Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/configuration.md

An example of how experiment paths are structured within dbdemos.

```text
/Shared/dbdemos/experiments/{demo_name}/{experiment_name}
```

--------------------------------

### config.py Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/DemoStructure.md

Example of the config.py file, defining Unity Catalog configuration for a demo.

```python
# Databricks notebook source
# MAGIC %md 
# MAGIC ## Configuration file

catalog = "main__build"
schema = dbName = db = "dbdemos_demo_name"
volume_name = "raw_data"
```

--------------------------------

### SQL Queries Configuration Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/BundleConfiguration.md

Example of grouping SQL queries for demo initialization across different phases.

```python
"sql_queries": [
    [  # Phase 1: Bronze tables
        """
        CREATE TABLE IF NOT EXISTS `{{CATALOG}}`.`{{SCHEMA}}`.customers_bronze
        COMMENT 'Raw customer data'
        AS SELECT * EXCEPT (_rescued_data)
        FROM read_files('/Volumes/{{CATALOG}}/{{SCHEMA}}/raw_data', format => 'parquet')
        """
    ],
    [  # Phase 2: Silver tables
        """
        CREATE TABLE IF NOT EXISTS `{{CATALOG}}`.`{{SCHEMA}}`.customers_silver
        COMMENT 'Cleaned customer data'
        AS SELECT DISTINCT * FROM `{{CATALOG}}`.`{{SCHEMA}}`.customers_bronze
        WHERE customer_id IS NOT NULL
        """
    ],
    [  # Phase 3: Load data
        """
        INSERT INTO `{{CATALOG}}`.`{{SCHEMA}}`.customers_silver
        SELECT * FROM ...
        """
    ],
    [  # Phase 4: Functions
        """
        CREATE OR REPLACE FUNCTION `{{CATALOG}}`.`{{SCHEMA}}`.calculate_score(
            score_input DOUBLE
        ) RETURNS DOUBLE
        RETURN score_input * 1.5
        """,
        """
        ALTER TABLE `{{CATALOG}}`.`{{SCHEMA}}`.customers_silver
        ADD CONSTRAINT pk_customer PRIMARY KEY (customer_id)
        """
    ],
    [  # Phase 5: Views/Metrics
        """
        CREATE OR REPLACE VIEW `{{CATALOG}}`.`{{SCHEMA}}`.top_customers
        WITH METRICS
        LANGUAGE YAML
        AS $$
        version: 0.1
        source: |
          SELECT * FROM {{CATALOG}}.{{SCHEMA}}.customers_silver
          WHERE lifetime_value > 10000
        measures:
          - name: Total Customers
            expr: COUNT(*)
        $$
        """
    ]
]
```

--------------------------------

### Initialize a New Demo

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/README.md

Setup script for initializing a new demo, including schema setup and data loading.

```python
# In _resources/00-setup.py
%run ../config
%run ../../../_resources/00-global-setup-v2

DBDemos.setup_schema(catalog, db, reset_all_data, volume_name)
```

--------------------------------

### Data Folders Configuration Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/configuration.md

Example configuration for data folders within a bundle.

```json
"data_folders": [
    {
        "source_folder": "path/in/repository",
        "source_format": "parquet",
        "target_volume_folder": "folder_in_volume",
        "target_format": "parquet"
    }
]
```

--------------------------------

### Dashboards Configuration Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/configuration.md

Example configuration for dashboards within a bundle.

```json
"dashboards": [
    {
        "name": "[dbdemos] Dashboard Name",
        "id": "dashboard-id"
    }
]
```

--------------------------------

### Pipelines Configuration Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/BundleConfiguration.md

Example of how to define Spark Declarative Pipeline (SDP) configurations.

```python
"pipelines": [
    {
        "path": "01-Data-Ingestion/01-Setup-SDP-pipeline",
        "name": "Retail Data Ingestion"
    }
]
```

--------------------------------

### Setup Notebook

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/demo-FSI/lakehouse-fsi-credit-decisioning/05-Generative-AI/05.1-AI-Functions-Creation.ipynb

This code snippet is a notebook magic command to run a setup script. It's used to initialize the environment for the demo.

```python
%run ../_resources/00-setup $reset_all_data=false
```

--------------------------------

### SQL Queries Configuration Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/configuration.md

Example configuration for SQL queries within a bundle, grouped by execution phase.

```json
"sql_queries": [
    [  # Phase 1: Create bronze tables
        "CREATE TABLE ... FROM read_files(...)"
    ],
    [  # Phase 2: Create silver tables
        "CREATE TABLE ... AS SELECT ..."
    ],
    [  # Phase 3: Insert data
        "INSERT INTO ... SELECT ..."
    ],
    [  # Phase 4: Create functions and constraints
        "CREATE FUNCTION ...",
        "ALTER TABLE ... ADD CONSTRAINT ..."
    ],
    [  # Phase 5: Create views and metrics
        "CREATE VIEW ... WITH METRICS ..."
    ]
]
```

--------------------------------

### ConfigVariables Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/types.md

An example of Python variables for configuration.

```python
catalog = "main__build"
schema = dbName = db = "dbdemos_retail_c360"
volume_name = "raw_data"
```

--------------------------------

### Reinstallation Workflow Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/configuration.md

Python code demonstrating the steps to re-install a demo with clean data.

```python
# Step 1: Set reset_all_data widget
dbutils.widgets.dropdown("reset_all_data", "true", ["true", "false"])

# Step 2: Run config
%run ./config

# Step 3: Run setup with reset
%run ./_resources/00-setup

# This executes:
# 1. DBDemos.setup_schema(..., reset_all_data=True, ...)
# 2. Drops existing volume and schema
# 3. Creates fresh catalog/schema/volume
# 4. Loads data from source files
# 5. Initializes all tables and pipelines
```

--------------------------------

### Genie Rooms Configuration Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/configuration.md

Example configuration for Genie rooms within a bundle.

```json
"genie_rooms": [
    {
        "id": "room-id",
        "display_name": "Room Display Name",
        "description": "Room description",
        "table_identifiers": [
            "{{CATALOG}}.{{SCHEMA}}.table_name"
        ],
        "sql_instructions": [...],
        "function_names": [
            "{{CATALOG}}.{{SCHEMA}}.function_name"
        ],
        "curated_questions": [...] 
    }
]
```

--------------------------------

### Data Folders Configuration Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/BundleConfiguration.md

Example of defining data folder mappings from source repository to target volume.

```python
"data_folders": [
    {
        "source_folder": "retail/c360/customers",
        "source_format": "parquet",
        "target_volume_folder": "customers",
        "target_format": "parquet"
    },
    {
        "source_folder": "retail/c360/transactions",
        "source_format": "csv",
        "target_volume_folder": "transactions",
        "target_format": "parquet"
    }
]
```

--------------------------------

### init_job Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/BundleConfiguration.md

An example of an initialization job configuration, specifying the job name, notebook path, cluster type, and Spark version.

```python
"init_job": {
    "name": "demo-init",
    "notebook_path": "_resources/00-setup",
    "cluster_type": "all-purpose",
    "spark_version": "14.3.x-scala2.12"
}
```

--------------------------------

### Data Folder Configuration Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/types.md

An example of a data folder configuration object.

```json
{
    "source_folder": "retail/c360/customers",
    "source_format": "parquet",
    "target_volume_folder": "customers",
    "target_format": "parquet"
}

```

--------------------------------

### Dashboard Configuration Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/types.md

An example of a dashboard configuration object.

```json
{
    "name": "[dbdemos] Retail - Customer 360",
    "id": "retail-c360"
}

```

--------------------------------

### Example Path and Name Configuration

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/types.md

An example of a JSON object defining path and name for a pipeline.

```json
{
    "path": "01-Data-Ingestion/01-Setup-SDP-pipeline",
    "name": "Retail Data Ingestion"
}
```

--------------------------------

### Configuration

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/README.md

Example of runtime configuration parameters for a demo.

```python
catalog = "main__build"              # Unity Catalog
schema = dbName = db = "dbdemos_xxx" # Schema name
volume_name = "raw_data"             # Data volume
```

--------------------------------

### genie_rooms Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/BundleConfiguration.md

An example of a Genie room configuration for customer analysis, including table identifiers, SQL instructions for top customers and LTV, a custom function, and curated questions.

```python
"genie_rooms": [
    {
        "id": "customer-analysis",
        "display_name": "Customer 360 Analysis",
        "description": "Analyze customer behavior and lifetime value",
        "table_identifiers": [
            "{{CATALOG}}.{{SCHEMA}}.customers",
            "{{CATALOG}}.{{SCHEMA}}.transactions",
            "{{CATALOG}}.{{SCHEMA}}.interactions"
        ],
        "sql_instructions": [
            {
                "title": "Top 10 Customers by Spend",
                "content": """
                    SELECT customer_id, SUM(transaction_amount) as total_spend
                    FROM {{CATALOG}}.{{SCHEMA}}.transactions
                    GROUP BY customer_id
                    ORDER BY total_spend DESC
                    LIMIT 10
                """
            },
            {
                "title": "Customer Lifetime Value by Segment",
                "content": """
                    SELECT segment, AVG(lifetime_value) as avg_ltv
                    FROM {{CATALOG}}.{{SCHEMA}}.customers
                    GROUP BY segment
                    ORDER BY avg_ltv DESC
                """
            }
        ],
        "function_names": [
            "{{CATALOG}}.{{SCHEMA}}.calculate_ltv"
        ],
        "instructions": "Help users analyze customer data to improve retention and increase lifetime value",
        "curated_questions": [
            "Top 10 Customers by Spend",
            "Customer Lifetime Value by Segment",
            "Which customer segments are most profitable?",
            "What is the retention rate by segment?"
        ]
    }
]
```

--------------------------------

### Complete Bundle Configuration Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/BundleConfiguration.md

A complete example of a Databricks bundle configuration, including notebook source and markdown for demo bundling.

```python
# Databricks notebook source
# MAGIC %md 
# MAGIC ## Demo bundle configuration
# MAGIC Please ignore / do not delete, only used to prep and bundle the demo

```

--------------------------------

### Parquet format usage example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/types.md

Example of how to specify Parquet format when reading files.

```sql
readFiles(..., format => 'parquet')
```

--------------------------------

### Bundle Configuration Structure

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/configuration.md

Example structure of a bundle configuration file.

```json
{
    "name": "demo-name",
    "category": "Category",
    "title": "Human-Readable Title",
    "description": "Description of the demo",
    "custom_schema_supported": True,
    "default_catalog": "main",
    "default_schema": "dbdemos_demo_name",
    "bundle": True,
    "serverless_supported": True,
    "notebooks": [...],
    "dashboards": [...],
    "pipelines": [...],
    "data_folders": [...],
    "sql_queries": [...],
    "genie_rooms": [...] 
}
```

--------------------------------

### create_mockup_automl_run() return type example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/types.md

Example structure for the return type of DBDemos.create_mockup_automl_run().

```python
AutoMLRun(
    best_trial=BestTrialMock(
        mlflow_run_id=str,
        metrics=dict,
        params=dict,
        load_model()  # Method returning sklearn model
    ),
    experiment=XPMock(
        experiment_id=str
    )
)
```

--------------------------------

### Genie Room Configuration Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/types.md

An example of a dictionary representing a Genie Room configuration.

```python
{
    "id": "customer-analysis",
    "display_name": "Customer 360 Analysis",
    "description": "Analyze customer behavior and lifetime value",
    "table_identifiers": [
        "{{CATALOG}}.{{SCHEMA}}.customers",
        "{{CATALOG}}.{{SCHEMA}}.transactions"
    ],
    "sql_instructions": [
        {
            "title": "Top 10 Customers",
            "content": "SELECT ... FROM {{CATALOG}}.{{SCHEMA}}.customers ..."
        }
    ],
    "function_names": [
        "{{CATALOG}}.{{SCHEMA}}.calculate_ltv"
    ],
    "instructions": "Help users analyze customer data and identify insights",
    "curated_questions": [
        "Top 10 Customers by Spend",
        "Customer Lifetime Value by Segment"
    ]
}
```

--------------------------------

### JSON format usage example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/types.md

Example of how to specify JSON format when reading files.

```sql
readFiles(..., format => 'json')
```

--------------------------------

### init_experiment_for_batch() return type example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/types.md

Example structure for the return type of DBDemos.init_experiment_for_batch().

```json
{
    "experiment_id": str,
    "name": str,
    "artifact_location": str
}
```

--------------------------------

### SQLInstruction Type Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/types.md

An example of a dictionary representing a SQL instruction with a title and content.

```python
{
    "title": "Top 10 Customers by Spend",
    "content": """
        SELECT customer_id, SUM(amount) as total_spend
        FROM {{CATALOG}}.{{SCHEMA}}.transactions
        GROUP BY customer_id
        ORDER BY total_spend DESC
        LIMIT 10
    """
}
```

--------------------------------

### Install Feature Engineering Library

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/product_demos/Data-Science/feature-store/04_declarative_feature_engineering(BETA).ipynb

Installs the necessary databricks-feature-engineering library and restarts the Python environment.

```python
# Need to install the feature engineering library
%pip install databricks-feature-engineering>=0.14.0
dbutils.library.restartPython()
```

--------------------------------

### CSV format usage example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/types.md

Example of how to specify CSV format when reading files.

```sql
readFiles(..., format => 'csv')
```

--------------------------------

### Table References

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/DemoStructure.md

Examples of how to reference tables in Databricks, both fully qualified and unqualified.

```python
# Fully qualified
f"`{catalog}`.`{db}`.`table_name`"

# Or unqualified (if in current context)
"table_name"
```

--------------------------------

### cluster Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/BundleConfiguration.md

An example of a cluster configuration, detailing Spark version, node type, number of workers, and instance pool ID.

```python
"cluster": {
    "spark_version": "14.3.x-scala2.12",
    "node_type_id": "i3.xlarge",
    "num_workers": 2,
    "instance_pool_id": "pool-123"
}
```

--------------------------------

### Safe Schema Setup

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/errors.md

Safely sets up a schema, handling potential assertion or general exceptions during the process.

```python
try:
    DBDemos.setup_schema(
        catalog=catalog,
        db=db,
        reset_all_data=reset_all_data,
        volume_name=volume_name
    )
except AssertionError as e:
    print(f"Configuration error: {e}")
    raise
except Exception as e:
    print(f"Setup failed: {e}")
    print("Please verify:")
    print("- Catalog exists and is accessible")
    print("- User has CREATE permission on catalog")
    print("- Unity Catalog is enabled in workspace")
    raise
```

--------------------------------

### MLflow Configuration

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/configuration.md

Example of setting an MLflow experiment path.

```python
# Experiment path pattern
experiment_path = f"/Shared/dbdemos/experiments/{demo_name}/{experiment_name}"

mlflow.set_experiment(experiment_path)
```

--------------------------------

### Experiment Path Reference

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/DemoStructure.md

Example of how to construct an experiment path for MLflow experiments.

```python
experiment_path = f"/Shared/dbdemos/experiments/{demo_name}/{experiment_name}"
```

--------------------------------

### Bundle Configuration

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/README.md

Example of bundle configuration metadata for demo packaging.

```json
{
    "name": "demo-name",
    "category": "Category",
    "title": "Human Title",
    "notebooks": [...],
    "dashboards": [...],
    "sql_queries": [...],
    ...
}
```

--------------------------------

### Resolution for Invalid Catalog Name

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/errors.md

Example of valid and invalid catalog names for setup_schema().

```python
# ❌ Invalid
catalog = "hive_metastore"
catalog = "spark_catalog"

# ✓ Valid
catalog = "main"
catalog = "main__build"
catalog = "dbdemos"
catalog = "my_custom_catalog"
```

--------------------------------

### Contribute to the repository

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/README.md

Example steps for contributing to the repository, including cloning, branching, committing, and pushing code.

```bash
git clone
git checkout main
git pull
git checkout -b FEATURENAME create your branch with the feature 
.. do the work 
run all your notebooks and make sure they work smoothly in a new environment
commit your work: `git commit -a.` Make sure to enter meaningful commit message title.
git push origin FEATURENAME
```

--------------------------------

### Feature Store Configuration

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/configuration.md

Example of creating a feature table using the Feature Store client.

```python
from databricks.feature_store import FeatureStoreClient

fs = FeatureStoreClient()

# Primary table pattern
fs.create_table(
    name=f"{catalog}.{schema}.feature_table",
    primary_keys=["id"],
    timestamp_keys=["created_at"],
    path=f"/Volumes/{catalog}/{schema}/feature_store"
)
```

--------------------------------

### Set Model Endpoint Permission

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/configuration.md

Example of how to set permissions for a model endpoint.

```python
DBDemos.set_model_endpoint_permission(
    endpoint_name="my-model-endpoint",
    permission="CAN_QUERY",
    group_name="account users"
)
```

--------------------------------

### Schema Override Widget

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/configuration.md

Text widget for schema override in legacy setup files.

```python
dbutils.widgets.text("db", "", "Database")
```

--------------------------------

### Configuration Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/DBDemos.md

Typical usage pattern for configuring DBDemos in a demo notebook.

```python
# In config.py
catalog = "main__build"
schema = "dbdemos_my_demo"
volume_name = "raw_data"

# In _resources/00-setup notebook
dbutils.widgets.dropdown("reset_all_data", "false", ["true", "false"])
reset_all_data = dbutils.widgets.get("reset_all_data") == "true"

# Import global setup
%run ../../../_resources/00-global-setup-v2

# Initialize schema
DBDemos.setup_schema(catalog, schema, reset_all_data, volume_name)

# Check if data exists
if DBDemos.is_folder_empty(f"/Volumes/{catalog}/{schema}/{volume_name}"):
    # Load data...
    pass

# Set permissions
DBDemos.set_model_permission(f"{catalog}.{schema}.my_function", "EXECUTE", "account users")
```

--------------------------------

### _resources/00-setup.py Typical Pattern

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/DemoStructure.md

Typical pattern for the _resources/00-setup.py initialization notebook.

```python
# Databricks notebook source
# MAGIC %run ../config

dbutils.widgets.dropdown("reset_all_data", "false", ["true", "false"])
reset_all_data = dbutils.widgets.get("reset_all_data") == "true"

# MAGIC %run ../../../_resources/00-global-setup-v2

# MAGIC %run ./01-load-data $reset_all_data=$reset_all_data

# Additional initialization code...
```

--------------------------------

### Error Handling Example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/DBDemos.md

Example of how to handle errors when using DBDemos methods, specifically for schema setup.

```python
try:
    DBDemos.setup_schema(catalog, db, reset_all_data=True)
except AssertionError as e:
    print(f"Invalid catalog name: {e}")
except Exception as e:
    print(f"Schema setup failed: {e}")
    raise
```

--------------------------------

### Example Usage of Credit Report Generator

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/demo-FSI/lakehouse-fsi-credit-decisioning/05-Generative-AI/05.1-AI-Functions-Creation.ipynb

An example of how to call the `credit_report_generator` SQL function with specific customer data to get a credit report.

```sql
SELECT credit_report_generator(
    93407,0,0,45,1,30,97,0,0,158211467.36,7853.23,866, 86628,969,1,0,1,0
  ) as credit_report
```

--------------------------------

### min_dbr_version Widget Usage

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/types.md

Example of how to define and get the value for the 'min_dbr_version' text widget.

```python
dbutils.widgets.text("min_dbr_version", "12.2", "Min required DBR version")
min_version = dbutils.widgets.get("min_dbr_version")
```

--------------------------------

### get_last_experiment() return type example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/types.md

Example structure for the return type of DBDemos.get_last_experiment().

```json
{
    "path": str,              # Full path like /Shared/dbdemos/...
    "object_type": str,       # "MLFLOW_EXPERIMENT"
    "object_id": str,         # Numeric ID
    "modified_at": int        # Timestamp
}
```

--------------------------------

### Generative AI with Vector Search and LLM

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/DemoStructure.md

Example demonstrating the initialization of Vector Search, querying similar documents, and using the results with a Langchain LLM via Databricks model serving.

```python
# Databricks notebook source
from databricks.vector_search.client import VectorSearchClient
from langchain.llms import Databricks

# Initialize Vector Search
client = VectorSearchClient()
index = client.get_index(
    endpoint_name="vector-search-endpoint",
    index_name=f"{catalog}.{schema}.vector_index"
)

# Query similar documents
results = index.similarity_search(query_text, columns=["content"])

# Use with LLM
llm = Databricks(
    host=host,
    api_token=token,
    endpoint_name="model-serving-endpoint"
)

response = llm.predict(prompt)
```

--------------------------------

### set_model_permission() return type example

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/types.md

Example structure for the return type of DBDemos.set_model_permission().

```json
{
    "add": [Privilege(permission=str)],
    "remove": [],
    "principal": str
}
```

--------------------------------

### Initialize a Demo

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/README.md

Python code to initialize a demo by setting up the schema.

```python
# In any demo notebook
%run ../config
%run ../../../_resources/00-global-setup-v2

DBDemos.setup_schema(catalog, db, reset_all_data, volume_name)
```

--------------------------------

### Schema Initialization Parameters

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/configuration.md

Parameters used when calling DBDemos.setup_schema().

```python
DBDemos.setup_schema(
    catalog="main__build",
    db="dbdemos_demo_name", 
    reset_all_data=False,
    volume_name="raw_data"
)
```

--------------------------------

### Introduction Notebook Pattern

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/DemoStructure.md

Basic pattern for an introduction notebook, including markdown for title and descriptive text.

```python
# Databricks notebook source
# MAGIC %
# MAGIC # Demo Title
# MAGIC This demo shows how to...
```

--------------------------------

### get_python_version_mlflow() return type examples

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/types.md

Example version strings for the return type of DBDemos.get_python_version_mlflow().

```python
"3.11.10"  # Version string
```

```python
"3.12.3"   # Version string
```

--------------------------------

### _resources/01-load-data.py Typical Pattern

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/DemoStructure.md

Typical pattern for the _resources/01-load-data.py notebook, responsible for downloading and preparing dataset files.

```python
# Databricks notebook source
dbutils.widgets.dropdown("reset_all_data", "false", ["true", "false"])
reset_all_data = dbutils.widgets.get("reset_all_data") == "true"

# MAGIC %run ../config
# MAGIC %run ../../../_resources/00-global-setup-v2

DBDemos.setup_schema(catalog, db, reset_all_data, volume_name)
folder = f"/Volumes/{catalog}/{db}/{volume_name}"

# Check if data already exists
data_missing = DBDemos.is_any_folder_empty([folder+"/dataset1", folder+"/dataset2"])

if reset_all_data or data_missing:
    # Download from GitHub
    DBDemos.download_file_from_git(folder+'/dataset1', "databricks-demos", "dbdemos-dataset", "/path/in/repo")
    
    # Transform to desired format
    spark.read.csv(folder+'/dataset1/file.csv', header=True) \
        .write.format('json').mode('overwrite') \
        .save(folder+'/dataset1')
```

--------------------------------

### Basic Configuration File

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/configuration.md

Defines the default catalog, schema, and volume settings for a demo.

```python
# Databricks notebook source

# Unity Catalog name (required)
# Note: Not recommended to change here - reinstall demo with catalog parameter instead
catalog = "main__build"

# Schema/database name within the catalog (required)
# Multiple aliases support for backwards compatibility
schema = dbName = db = "dbdemos_demo_name"

# Volume name for storing raw data files (optional)
volume_name = "raw_data"
```

--------------------------------

### Get MLflow Compatible Python Version

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/DBDemos.md

Gets the Python version string compatible with MLflow model logging.

```python
@staticmethod
def get_python_version_mlflow() -> str
```

```python
# Get compatible Python version
py_version = DBDemos.get_python_version_mlflow()
print(f"Using Python {py_version} for MLflow")
```

--------------------------------

### Install Python Library

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/errors.md

Installs a Python library (e.g., scikit-learn) using pip within a notebook and restarts the Python kernel.

```python
# Install via pip in notebook
%pip install scikit-learn

# Or install in cluster configuration
# Cluster → Libraries → Install Library → PyPI package

# Restart Python after installation
dbutils.library.restartPython()
```

--------------------------------

### Volume Paths for Data and Artifacts

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/DemoStructure.md

Examples of Unity Catalog Volume paths for storing raw ingestion data, processed data, and model artifacts.

```python
# Raw ingestion data
f"/Volumes/{catalog}/{db}/{volume_name}/raw_data"

# Processed data
f"/Volumes/{catalog}/{db}/{volume_name}/processed"

# Model artifacts
f"/Volumes/{catalog}/{db}/{volume_name}/models"
```

--------------------------------

### sql_instructions Format

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/BundleConfiguration.md

Specifies the format for SQL instructions within Genie rooms, including a title and the SQL query content.

```python
{
    "title": "Question Title",
    "content": "SELECT ... SQL query ..."
}
```

--------------------------------

### Catalog Requirements

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/README.md

Example of fixing an AssertionError related to catalog names.

```python
# Fix: Use Unity Catalog name
catalog = "main__build"  # Valid
catalog = "hive_metastore"  # Invalid
```

--------------------------------

### Volume Path Pattern

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/configuration.md

The pattern for constructing volume paths.

```bash
/Volumes/{catalog}/{schema}/{volume_name}/{relative_path}
```

--------------------------------

### Wait Timeout Fix

Source: https://github.com/databricks-demos/dbdemos-notebooks/blob/main/_autodocs/api-reference/README.md

Example of increasing the timeout duration for waiting for a table.

```python
# Fix: Check pipeline job logs, increase timeout
DBDemos.wait_for_table("table_name", timeout_duration=600)
```