### Run Quickstart Scripts

Source: https://docs.databricks.com/gcp/en/generative-ai/agent-framework/author-agent

Installs dependencies, sets up the environment, and starts the agent application locally.

```bash
uv run quickstart
```

```bash
uv run start-app
```

--------------------------------

### Example Maven Library Installation

Source: https://docs.databricks.com/gcp/en/reference/jobs-2.0-api

Configure a Maven library installation using Gradle-style coordinates. The 'repo' field is optional.

```json
{ "coordinates": "org.jsoup:jsoup:1.7.2" }
```

--------------------------------

### Example PyPI Library Installation

Source: https://docs.databricks.com/gcp/en/reference/jobs-2.0-api

Specify a PyPI library to install. The 'repo' field is optional; if omitted, the default pip index is used.

```json
{ "package": "simplejson", "repo": "https://my-repo.com" }
```

--------------------------------

### Example JAR Library Installation

Source: https://docs.databricks.com/gcp/en/reference/jobs-2.0-api

Use this to install a JAR library from DBFS or GCS. Ensure the cluster has read access for GCS URIs.

```json
{ "jar": "dbfs:/mnt/databricks/library.jar"}
```

```json
{ "jar": "gs://my-bucket/library.jar" }
```

--------------------------------

### Example Wheel Library Installation

Source: https://docs.databricks.com/gcp/en/reference/jobs-2.0-api

Install a wheel or zipped wheels from DBFS or GCS. Ensure cluster read access for GCS and correct file naming conventions.

```json
{ "whl": "dbfs:/my/whl" }
```

```json
{ "whl": "gs://my-bucket/whl" }
```

--------------------------------

### Example of GET command for Databricks Volumes

Source: https://docs.databricks.com/gcp/en/sql/language-manual/sql-ref-syntax-aux-connector-get

Example demonstrating how to fetch a CSV file from a specified volume path to a local file path.

```sql
> GET '/Volumes/test_catalog/test_schema/test_volume/2023/06/file1.csv' TO '/home/bob/data.csv'  
```

--------------------------------

### Example Egg Library Installation

Source: https://docs.databricks.com/gcp/en/reference/jobs-2.0-api

Install an egg library from DBFS or GCS. For GCS, the cluster must have read permissions.

```json
{ "egg": "dbfs:/my/egg" }
```

```json
{ "egg": "gs://my-bucket/egg" }
```

--------------------------------

### Start Airflow Webserver

Source: https://docs.databricks.com/gcp/en/jobs/how-to/use-airflow-with-jobs

Start the Airflow web server to access the Airflow UI. This command should be run in the Airflow installation directory.

```bash
pipenv shell
export AIRFLOW_HOME=$(pwd)
airflow webserver
```

--------------------------------

### Example Maven Library Installation with Repo and Exclusions

Source: https://docs.databricks.com/gcp/en/reference/jobs-2.0-api

Specify a Maven library with a custom repository and dependency exclusions.

```json
{ "coordinates": "org.jsoup:jsoup:1.7.2", "repo": "https://my-repo.com", "exclusions": ["slf4j:slf4j", "*:hadoop-client"] }
```

--------------------------------

### TensorFlow Keras Example Notebook

Source: https://docs.databricks.com/gcp/en/machine-learning/train-model/tensorflow

This notebook provides an example of training machine learning models on tabular data using TensorFlow Keras, including inline TensorBoard integration. It is suitable for getting started with TensorFlow Keras.

```python
The 10-minute tutorial notebook shows an example of training machine learning models on tabular data with TensorFlow Keras, including using inline TensorBoard.
```

--------------------------------

### Add File Examples

Source: https://docs.databricks.com/gcp/en/sql/language-manual/sql-ref-syntax-aux-resource-mgmt-add-file

Demonstrates adding files and directories using the ADD FILE command with different path formats, including paths with spaces.

```sql
> ADD FILE /tmp/test;
```

```sql
> ADD FILE "/path/to/file/abc.txt";
```

```sql
> ADD FILE '/another/test.txt';
```

```sql
> ADD FILE "/path with space/abc.txt";
```

```sql
> ADD FILE "/path/to/some/directory" "/path with space/abc.txt";
```

--------------------------------

### Example concurrent start script

Source: https://docs.databricks.com/gcp/en/dev-tools/databricks-apps/deploy

Use this script with `concurrently` to run both Node.js and Python processes simultaneously. Ensure `concurrently` is installed as a dev dependency.

```bash
concurrently "npm run start:node" "python my_app.py"
```

--------------------------------

### Create and Populate a Partitioned Table

Source: https://docs.databricks.com/gcp/en/sql/language-manual/sql-ref-syntax-aux-show-partitions

Demonstrates how to create a partitioned table and insert data into specific partitions. This setup is used for subsequent SHOW PARTITIONS examples.

```sql
-- create a partitioned table and insert a few rows.  
> USE salesdb;  
> CREATE TABLE customer(id INT, name STRING) PARTITIONED BY (state STRING, city STRING);  
> INSERT INTO customer PARTITION (state = 'CA', city = 'Fremont') VALUES (100, 'John');  
> INSERT INTO customer PARTITION (state = 'CA', city = 'San Jose') VALUES (200, 'Marry');  
> INSERT INTO customer PARTITION (state = 'AZ', city = 'Peoria') VALUES (300, 'Daniel');  
```

--------------------------------

### Get Enhanced Security Monitoring Setting

Source: https://docs.databricks.com/gcp/en/dev-tools/cli/reference/account-settings-commands

Retrieves the current enhanced security monitoring setting for the account. No specific setup is required beyond having the Databricks CLI installed and configured.

```bash
databricks account settings esm-enablement-account get  
```

--------------------------------

### Manage Active Streaming Queries

Source: https://docs.databricks.com/gcp/en/pyspark/reference/classes/streamingquerymanager

This example demonstrates how to start a streaming query, access its name, wait for termination, stop it, and reset the manager. Use `spark.streams.active` to get a list of active queries.

```python
sdf = spark.readStream.format("rate").load()
sq = sdf.writeStream.format('memory').queryName('this_query').start()
sqm = spark.streams
[q.name for q in sqm.active]
# ['this_query']
sqm.awaitAnyTermination(5)
# True
sq.stop()
sqm.resetTerminated()
```

--------------------------------

### SQL Examples for Partitioning

Source: https://docs.databricks.com/gcp/en/sql/language-manual/sql-ref-partition

Illustrates practical usage of PARTITIONED BY and PARTITION clauses in SQL for creating tables, inserting data, and managing partitions.

```APIDOC
## Examples
```sql
-- Use the PARTTIONED BY clause in a table definition
> CREATE TABLE student(university STRING, 
                       major      STRING, 
                       name       STRING)
         PARTITIONED BY(university, major) 

> CREATE TABLE professor(name STRING)
         PARTITIONED BY(university STRING, 
                        department STRING);

-- Use the PARTITION specification to INSERT into a table
> INSERT INTO student
         PARTITION(university= 'TU Kaiserslautern') (major, name)
         SELECT major, name FROM freshmen; 

-- Use the partition specification to add and drop a partition
> CREATE TABLE log(date DATE, id INT, event STRING)
     USING CSV
     PARTITIONED BY (date); 

> ALTER TABLE log ADD PARTITION(date = DATE'2021-09-10'); 

> ALTER TABLE log DROP PARTITION(date = DATE'2021-09-10'); 

-- Drop all partitions from the named university, independent of the major.
> ALTER TABLE student DROP PARTITION(university = 'TU Kaiserslautern');
```
```

--------------------------------

### Start Local Agent Server

Source: https://docs.databricks.com/gcp/en/generative-ai/agent-framework/multi-agent-apps

Use `uv run quickstart` to configure Databricks authentication and set up MLflow tracing, followed by `uv run start-app` to launch the agent server and chat UI locally.

```bash
uv run quickstart  
uv run start-app  

```

--------------------------------

### Install Faker Package for PySpark Data Source Example

Source: https://docs.databricks.com/gcp/en/pyspark/datasources

Install the 'faker' package, which is used in the example to generate fake data for the custom PySpark data source.

```Python
%pip install faker
```

--------------------------------

### Notebook Setup: Install SDK and Define Variables

Source: https://docs.databricks.com/gcp/en/data-governance/unity-catalog/data-quality-monitoring/data-profiling/create-monitor-api

Basic setup for a Databricks notebook, including installing the Python SDK and defining catalog, schema, and table names.

```python
%md
## Setup
* Install Python SDK
* Define catalog, schema and table names
```

--------------------------------

### Example: Get Current Database

Source: https://docs.databricks.com/gcp/en/pyspark/reference/classes/catalog

Example of retrieving the current default database.

```APIDOC
## Example: Get Current Database

### Description
Retrieves the name of the current default database in the Spark session.

### Method
GET (conceptual)

### Endpoint
spark.catalog.currentDatabase()

### Request Example
```python
spark.catalog.currentDatabase()
```

### Response Example
```
'default'
```
```

--------------------------------

### Create a database instance

Source: https://docs.databricks.com/gcp/en/dev-tools/cli/reference/database-commands

Create a new database instance with a specified capacity SKU. Use the `--json` option for advanced configurations not available as arguments.

```bash
databricks database create-database-instance my-instance --capacity CU_1  
```

--------------------------------

### Streaming Query Setup for Error Handling Examples

Source: https://docs.databricks.com/gcp/en/structured-streaming/foreach

This is the initial setup for a streaming query using the rate source, which is used in subsequent examples demonstrating error handling within foreachBatch.

```Python
from pyspark.sql.functions import expr  
  
stream = spark.readStream.format("rate").option("rowsPerSecond", "100").load()  

```

--------------------------------

### Get listing installable content metadata

Source: https://docs.databricks.com/gcp/en/dev-tools/cli/reference/consumer-fulfillments-commands

Retrieve high-level metadata for installable content of a specific listing. Use this to preview what can be installed. Supports pagination.

```bash
databricks consumer-fulfillments get LISTING_ID [flags]
```

```bash
databricks consumer-fulfillments get 12345
```

--------------------------------

### Example: Database Instance and Catalog Configuration

Source: https://docs.databricks.com/gcp/en/dev-tools/bundles/resources

An example demonstrating the configuration of a database instance and its associated database catalog.

```APIDOC
## Example: Database Instance and Catalog Configuration

### Description
This example shows how to define a database instance and a corresponding database catalog in a bundle.

### Method
Not Applicable (Configuration Example)

### Endpoint
Not Applicable (Configuration Example)

### Parameters
None

### Request Example
```yaml
resources:
  database_instances:
    my_instance:
      name: my-instance
      capacity: CU_1
  database_catalogs:
    my_catalog:
      database_instance_name: ${resources.database_instances.my_instance.name}
      name: example_catalog
      database_name: my_database
      create_database_if_not_exists: true
```

### Response
None

### Further Information
For an example bundle that demonstrates how to define a database instance and corresponding database catalog, see the [bundle-examples GitHub repository](https://github.com/databricks/bundle-examples).
```

--------------------------------

### Configure NewConnector with Advanced Options

Source: https://docs.databricks.com/gcp/en/dev-tools/go-sql-driver

This example demonstrates configuring the `NewConnector` with additional options such as initial namespace, maximum rows, session parameters, timeout, and user agent entry.

```Go
connector, err := dbsql.NewConnector(  
  dbsql.WithAccessToken(os.Getenv("DATABRICKS_ACCESS_TOKEN")),  
  dbsql.WithServerHostname(os.Getenv("DATABRICKS_HOST")),  
  dbsql.WithPort(443),  
  dbsql.WithHTTPPath(os.Getenv("DATABRICKS_HTTP_PATH")),  
  dbsql.WithInitialNamespace("samples", "nyctaxi"),  
  dbsql.WithMaxRows(100),  
  dbsql.SessionParams(map[string]string{"timezone": "America/Sao_Paulo", "ansi_mode": "true", "query_tags": "team:analytics,project:reporting"}),  
  dbsql.WithTimeout(time.Minute),  
  dbsql.WithUserAgentEntry("example-user"),  
)
```

--------------------------------

### Databricks SQL LIST FILE Examples

Source: https://docs.databricks.com/gcp/en/sql/language-manual/sql-ref-syntax-aux-resource-mgmt-list-file

Demonstrates how to use the LIST FILE command after adding files. The first example lists all added files, while the second lists specific files.

```sql
> ADD FILE /tmp/test /tmp/test_2;

> LIST FILE;
  file:/private/tmp/test
  file:/private/tmp/test_2
```

```sql
> LIST FILE /tmp/test /some/random/file /another/random/file
  file:/private/tmp/test
```

--------------------------------

### Example SPARK_STARTUP_FAILURE Error Message

Source: https://docs.databricks.com/gcp/en/compute/troubleshooting/cluster-error-codes

This is an example of an error message indicating that the Spark driver failed to start within the configured timeout.

```text
Cluster '[REDACTED]' was terminated. Reason: SPARK_STARTUP_FAILURE (SERVICE_FAULT). Parameters: databricks_error_message:Spark failed to start: DEADLINE_EXCEEDED.  
```

```text
Cluster '[REDACTED]' was terminated. Reason: SPARK_STARTUP_FAILURE (SERVICE_FAULT). Parameters: databricks_error_message:Spark failed to start: Timed out after 200 seconds.  
```

--------------------------------

### Verify sample table setup

Source: https://docs.databricks.com/gcp/en/transactions/tutorial

Queries the `sample_accounts` and `sample_transactions` tables to confirm the successful creation and insertion of sample data.

```SQL
SELECT * FROM sample_accounts;
SELECT * FROM sample_transactions;
```

--------------------------------

### Install a Package from GitHub using `renv`

Source: https://docs.databricks.com/gcp/en/sparkr/renv

Install a package directly from a GitHub repository, for example, 'eddelbuettel/digest', using `renv::install`.

```r
renv::install("eddelbuettel/digest")  

```

--------------------------------

### Basic OPTIMIZE examples

Source: https://docs.databricks.com/gcp/en/sql/language-manual/delta-optimize

Demonstrates basic usage of the OPTIMIZE command for different scenarios, including optimizing a whole table, a partitioned table, and a table with liquid clustering.

```sql
> OPTIMIZE events;  
  
> OPTIMIZE events FULL;  
  
> -- Partitioned table  
> OPTIMIZE events WHERE date >= '2017-01-01';  
  
> -- Table with liquid clustering enabled (DBR 18.1 and above)  
> OPTIMIZE events FULL WHERE date >= '2025-01-01';  
  
> OPTIMIZE events  
    WHERE date >= current_timestamp() - INTERVAL 1 day  
    ZORDER BY (eventType);  

```

--------------------------------

### Helper Functions for Marketplace Data Installation

Source: https://docs.databricks.com/gcp/en/notebooks/source/generative-ai/dspy/dspy-multiagent-genie.html

Provides utility functions to manage Databricks Marketplace data installations, including getting the current user, extracting terms of service versions, installing tables from a listing, and deleting installed tables.

```python
from databricks.sdk.service.marketplace import ConsumerTerms

def get_current_user()-> str:
  return (dbutils.notebook.entry_point
    .getDbutils()
    .notebook()
    .getContext()
    .userName()
    .get()
    .split("@")[0]
    .replace(".", "_"))

def get_terms_of_service_version(terms_of_service: str)->str:
  return terms_of_service.split("/files/")[1].split("/")[0]

def install_marketplace_tables(listing, catalog_suffix:str) -> None:
  consumer_terms_version = get_terms_of_service_version(listing.detail.terms_of_service)
  w.consumer_installations.create(
    listing_id=listing.id, 
    share_name=listing.summary.share.name, 
    catalog_name=f"{get_current_user()}_{catalog_suffix}",
    accepted_consumer_terms=ConsumerTerms(version=consumer_terms_version)
  )

def delete_marketplace_tables(listing_id:str, catalog_suffix:str) -> None:
  installation = [
      installation
      for installation in w.consumer_installations.list()
      if installation.catalog_name == f"{get_current_user()}_{catalog_suffix}"
  ][0]

  w.consumer_installations.delete(
      installation_id=installation.id,
      listing_id=listing_id,
  )
```

--------------------------------

### Example of setting and using query tags

Source: https://docs.databricks.com/gcp/en/sql/language-manual/sql-ref-syntax-aux-conf-mgmt-set-query-tags

This example demonstrates setting initial tags, executing a query, updating tags (including removing one), and then executing another query with the updated tags.

```sql
-- Set two key-value pairs to annotate subsequent statement executions in this session.  
> SET QUERY_TAGS['team'] = 'marketing', QUERY_TAGS['cost_center'] = '701';  
  
-- Execute a query with the tags attached  
> SELECT * FROM sales_data;  
  
-- Update tags: change team value, add new tag, and remove cost_center tag  
> SET QUERY_TAGS['team'] = 'engineering', QUERY_TAGS['env'] = 'prod', QUERY_TAGS['cost_center'] = UNSET;  
  
-- Display all tags currently set on the session  
> SET QUERY_TAGS;  
  key         value  
  ----------  -----------  
  team        engineering  
  env         prod  
  
-- Set a key-only tag (value is NULL)  
> SET QUERY_TAGS['experiment'] = NULL;  
  
-- Execute another query with the updated tags  
-- This statement execution will be tagged with team:engineering, env:prod, experiment  
> SELECT * FROM sales_data;
```

--------------------------------

### Create and Drop Catalog Examples

Source: https://docs.databricks.com/gcp/en/sql/language-manual/sql-ref-syntax-ddl-drop-catalog

Demonstrates creating a catalog and then dropping it using CASCADE to remove associated schemas, and subsequently dropping it using IF EXISTS and RESTRICT to ensure it's empty.

```sql
-- Create a `vaccine` catalog  
> CREATE CATALOG vaccine COMMENT 'This catalog is used to maintain information about vaccines';
```

```sql
-- Drop the catalog and its schemas  
> DROP CATALOG vaccine CASCADE;
```

```sql
-- Drop the catalog using IF EXISTS and only if it is empty.  
> DROP CATALOG IF EXISTS vaccine RESTRICT;
```

--------------------------------

### Example: Configure Postgres Project, Branch, and Endpoint

Source: https://docs.databricks.com/gcp/en/dev-tools/bundles/resources

This example demonstrates the complete configuration for a Postgres project, including its main branch and a primary read-write endpoint with autoscaling settings.

```yaml
resources:  
  postgres_projects:  
    my_db:  
      project_id: test-prod-app  
      display_name: 'Production Database'  
      pg_version: 17  
  
  postgres_branches:  
    main:  
      parent: ${resources.postgres_projects.my_db.id}  
      branch_id: main  
      is_protected: false  
      no_expiry: true  
  
  postgres_endpoints:  
    primary:  
      parent: ${resources.postgres_branches.main.id}  
      endpoint_id: primary  
      endpoint_type: ENDPOINT_TYPE_READ_WRITE  
      autoscaling_limit_min_cu: 0.5  
      autoscaling_limit_max_cu: 4  
```

--------------------------------

### Install a specific version of Databricks SDK for Python using pip

Source: https://docs.databricks.com/gcp/en/dev-tools/sdk-python

Install a specific version of the databricks-sdk package, for example, version 0.1.6.

```bash
pip3 install databricks-sdk==0.1.6  
```

--------------------------------

### Example of Showing Connections

Source: https://docs.databricks.com/gcp/en/sql/language-manual/sql-ref-syntax-aux-show-connections

Demonstrates the output format for the SHOW CONNECTIONS command, including connection name, type, creation timestamp, creator, and comment.

```sql
> SHOW CONNECTIONS; 
 name                connection_type created_at                   created_by    comment  
 ------------------- --------------- ---------------------------- ------------- ---------------------
 mysql_connection    mysql           2022-01-01T00:00:00.000+0000 alf@melmak.et mysql connection  
 postgres_connection postgresql      2022-06-12T13:30:00.000+0000 alf@melmak.et postgresql connection  
```

--------------------------------

### Get Schema of VARIANT Expression

Source: https://docs.databricks.com/gcp/en/sql/language-manual/functions/schema_of_variant

Use `schema_of_variant` to get the DDL schema of a VARIANT expression. This example shows a simple JSON object.

```sql
-- Simple example  
> SELECT schema_of_variant(parse_json('{"key": 123, "data": [4, 5]}'))  
  OBJECT<data: ARRAY<BIGINT>, key: BIGINT>
```

--------------------------------

### Install Libraries

Source: https://docs.databricks.com/gcp/en/notebooks/source/generative-ai/function-calling-examples.html

Installs the necessary Python libraries for the demo, including openai, tenacity, and tqdm. A restart of the Python environment is triggered.

```python
%pip install openai tenacity tqdm dbutils.library.restartPython()
```

--------------------------------

### Get H3 cell ID for a point

Source: https://docs.databricks.com/gcp/en/sql/language-manual/functions/h3_pointash3

Simple example of using h3_pointash3 to get the H3 cell ID for a given point and resolution.

```sql
-- Simple example.  
> SELECT h3_pointash3('POINT(100 45)', 6);
  604116085645508607
```

--------------------------------

### Databricks SQL Share Management Examples

Source: https://docs.databricks.com/gcp/en/sql/language-manual/sql-ref-syntax-aux-show-all-in-share

Demonstrates creating a share, adding tables with aliasing and partitioning, and then listing the share's content. Ensure the share name exists before performing operations.

```sql
-- Create share `customer_share` only if share with same name doesn't exist, with a comment.  
> CREATE SHARE IF NOT EXISTS customer_share COMMENT 'This is customer share';  
  
-- Add 2 tables to the share.  
-- Expose my_schema.tab1 a different name.  
-- Expose only two partitions of other_schema.tab2  
> ALTER SHARE customer_share ADD TABLE my_schema.tab1 AS their_schema.tab1;  
> ALTER SHARE customer_share ADD TABLE other_schema.tab2 PARTITION (c1 = 5), (c1 = 7);  
  
-- List the content of the share  
> SHOW ALL IN SHARE customer_share;  
  name              type  shared_object          added_at                     added_by                   comment partitions  
  ----------------- ----- ---------------------- ---------------------------- -------------------------- ------- ------------------  
  other_schema.tab2 TABLE main.other_schema.tab2 2022-01-01T00:00:01.000+0000 alwaysworks@databricks.com NULL  
  their_schema.tab1 TABLE main.myschema.tab2     2022-01-01T00:00:00.000+0000 alwaysworks@databricks.com NULL    (c1 = 5), (c1 = 7)  
```

--------------------------------

### Example response for GET RestrictWorkspaceAdmins

Source: https://docs.databricks.com/gcp/en/admin/workspace-settings/restrict-workspace-admins

This JSON structure represents the response from a `GET` request for the `RestrictWorkspaceAdmins` setting, including the `etag` needed for updates.

```json
{
  "etag": "<etag>",
  "restrict_workspace_admins": {
    "status": "ALLOW_ALL"
  },
  "setting_name": "default"
}

```

--------------------------------

### Create Sample Catalogs

Source: https://docs.databricks.com/gcp/en/sql/language-manual/sql-ref-syntax-aux-show-catalogs

Demonstrates the creation of two sample catalogs, 'payroll_cat' and 'payments_cat', which can then be used with the SHOW CATALOGS command for filtering examples.

```sql
CREATE CATALOG payroll_cat;
CREATE CATALOG payments_cat;
```

--------------------------------

### Get Spark Version Example

Source: https://docs.databricks.com/gcp/en/pyspark/reference/functions/version

This example demonstrates how to retrieve and display the Spark version using PySpark. Ensure Spark is initialized before running.

```python
from pyspark.sql import functions as sf  
spark.range(1).select(sf.version()).show(truncate=False)
```

--------------------------------

### Install Node.js, Create App Directory, and Install Express

Source: https://docs.databricks.com/gcp/en/dev-tools/databricks-apps/tutorial-node

Use these bash commands to install Node.js, create a new directory for your application, navigate into it, and install the Express framework.

```bash
brew install node  
mkdir my-node-app  
cd my-node-app  
npm install express  

```

--------------------------------

### Install MLflow and OpenAI for Databricks (MLflow 2.x)

Source: https://docs.databricks.com/gcp/en/mlflow3/genai/tracing/app-instrumentation/automatic

Install MLflow version 2.x with Databricks connectivity and the OpenAI SDK. This setup is for users not yet on MLflow 3. Restart the Python environment after installation.

```python
%pip install --upgrade "mlflow[databricks]>=2.15.0,<3.0.0" openai>=1.0.0  
# Also install libraries you want to trace (langchain, anthropic, etc.)  
  
dbutils.library.restartPython()  
```

--------------------------------

### Install Python Wheel and Setuptools

Source: https://docs.databricks.com/gcp/en/jobs/how-to/use-python-wheels-in-workflows

Install the necessary Python packages for building wheel files using pip.

```bash
pip install wheel setuptools  
```

--------------------------------

### Start Airflow Scheduler

Source: https://docs.databricks.com/gcp/en/jobs/how-to/use-airflow-with-jobs

Start the Airflow scheduler, which is responsible for scheduling DAGs. Run this command in a new terminal within the Airflow installation directory.

```bash
pipenv shell
export AIRFLOW_HOME=$(pwd)
airflow scheduler
```

--------------------------------

### Get Data Type of an Integer Expression

Source: https://docs.databricks.com/gcp/en/sql/language-manual/functions/typeof

Use the `typeof` function to get the DDL-formatted type string for an integer literal. This is a basic usage example.

```sql
SELECT typeof(1);
 int  
```

--------------------------------

### Databricks SQL Join Hints Examples

Source: https://docs.databricks.com/gcp/en/sql/language-manual/sql-ref-syntax-qry-select-hints

Provides SQL examples demonstrating the usage of various join hints.

```APIDOC
### Examples​

```sql
-- Join Hints for broadcast join  
> SELECT /*+ BROADCAST(t1) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;  
> SELECT /*+ BROADCASTJOIN (t1) */ * FROM t1 left JOIN t2 ON t1.key = t2.key;  
> SELECT /*+ MAPJOIN(t2) */ * FROM t1 right JOIN t2 ON t1.key = t2.key;  
  
-- Join Hints for shuffle sort merge join  
> SELECT /*+ SHUFFLE_MERGE(t1) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;  
> SELECT /*+ MERGEJOIN(t2) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;  
> SELECT /*+ MERGE(t1) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;  
  
-- Join Hints for shuffle hash join  
> SELECT /*+ SHUFFLE_HASH(t1) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;  
  
-- Join Hints for shuffle-and-replicate nested loop join  
> SELECT /*+ SHUFFLE_REPLICATE_NL(t1) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;  
  
-- When different join strategy hints are specified on both sides of a join, Databricks SQL  
-- prioritizes the BROADCAST hint over the MERGE hint over the SHUFFLE_HASH hint  
-- over the SHUFFLE_REPLICATE_NL hint.  
-- Databricks SQL will issue Warning in the following example  
-- org.apache.spark.sql.catalyst.analysis.HintErrorLogger: Hint (strategy=merge)  
-- is overridden by another hint and will not take effect.  
> SELECT /*+ BROADCAST(t1), MERGE(t1, t2) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;  
  
-- When a table name is occluded by an alias you must use the alias name in the hint  
> SELECT /*+ BROADCAST(t1), MERGE(s1, s2) */ * FROM t1 AS s1 INNER JOIN t2 AS s2 ON s1.key = s2.key;  
```
```

--------------------------------

### Install MLflow Tracing and Agno for Production

Source: https://docs.databricks.com/gcp/en/mlflow3/genai/tracing/integrations/agno

Install the optimized `mlflow-tracing` package and Agno for production deployments.

```bash
pip install --upgrade mlflow-tracing agno openai  
```

--------------------------------

### Get Substring with Column Arguments

Source: https://docs.databricks.com/gcp/en/pyspark/reference/classes/column/substr

Use this snippet to extract a substring from a column when the start position and length are defined by other columns. The start position is 1-based.

```Python
df = spark.createDataFrame(  
     [(3, 4, "Alice"), (2, 3, "Bob")], ["sidx", "eidx", "name"])  
df.select(df.name.substr(df.sidx, df.eidx).alias("col")).collect()  
```

--------------------------------

### Define Database Instance and Catalog

Source: https://docs.databricks.com/gcp/en/dev-tools/bundles/resources

This example shows how to define a database instance with a specific capacity and a corresponding database catalog, including conditional creation.

```yaml
resources:
  database_instances:
    my_instance:
      name: my-instance
      capacity: CU_1
  database_catalogs:
    my_catalog:
      database_instance_name: ${resources.database_instances.my_instance.name}
      name: example_catalog
      database_name: my_database
      create_database_if_not_exists: true

```

--------------------------------

### Get Substring with Integer Arguments

Source: https://docs.databricks.com/gcp/en/pyspark/reference/classes/column/substr

Use this snippet to extract a substring from a column when the start position and length are known integers. The start position is 1-based.

```Python
df = spark.createDataFrame(  
     [(2, "Alice"), (5, "Bob")], ["age", "name"])  
df.select(df.name.substr(1, 3).alias("col")).collect()  
```

--------------------------------

### Example Response for Get Setting

Source: https://docs.databricks.com/gcp/en/admin/workspace-settings/restrict-workspace-admins

This is an example JSON response when retrieving the `RestrictWorkspaceAdmins` setting. It includes the etag and the current status of governed tag creation.

```json
{
  "etag": "<etag>",
  "restrict_workspace_admins": {
    "status": "ALLOW_ALL",
    "disable_gov_tag_creation": false
  },
  "setting_name": "default"
}
```

--------------------------------

### Create and Drop Procedure Example

Source: https://docs.databricks.com/gcp/en/sql/language-manual/sql-ref-syntax-ddl-drop-procedure

Demonstrates creating a simple procedure and then dropping it. It also shows attempting to drop a non-existent procedure and using IF EXISTS for safe dropping.

```sql
-- Create a procedure `hello`  
> CREATE PROCEDURE hello() SQL SECURITY INVOKER LANGUAGE SQL  
  AS BEGIN  
    SELECT 'hello!';  
  END;  
  
-- Drop the procedure  
> DROP PROCEDURE hello;  
  
-- Try to drop a procedure which is not present  
> DROP PROCEDURE hello;  
Error: ROUTINE_NOT_FOUND  
  
-- Drop a procedure only if it exists  
> DROP PROCEDURE IF EXISTS hello;  
```

--------------------------------

### Start Chat Session (Commented Out)

Source: https://docs.databricks.com/gcp/en/mlflow3/genai/tracing/app-instrumentation/manual-tracing/span-tracing

An example of how to start a continuous chat session, allowing user input in a loop. This code is commented out and requires uncommenting to be active.

```python
# @mlflow.trace(span_type=SpanType.CHAIN)
# def start_session():
#     messages = [{"role": "system", "content": "You are a friendly chat bot"}]
#     while True:
#         user_input = input(">> ")
#         chat_iteration(messages, user_input)
# 
#         if user_input == "BYE":
#             break
# 
# 
# start_session()
```

--------------------------------

### Create and Drop Share Example

Source: https://docs.databricks.com/gcp/en/sql/language-manual/sql-ref-syntax-ddl-drop-share

Demonstrates how to create a share and then subsequently drop it using the DROP SHARE command. The IF EXISTS clause can be used to safely drop a share that may not exist.

```sql
-- Create `vaccine` share  
> CREATE SHARE vaccine COMMENT 'This share is used to share information about vaccines';  
  
-- Drop the share  
> DROP SHARE vaccine;  
  
-- Drop the share using IF EXISTS.  
> DROP SHARE IF EXISTS vaccine;  
```

--------------------------------

### Get Request ID from Start Update API

Source: https://docs.databricks.com/gcp/en/release-notes/dlt/2022/37

The start update API request now returns the `request_id` field in the response body. This ID is a stable identifier for the original request starting the update and is inherited by retried or restarted updates.

```json
{
  "update_id": "the ID of the update that was started",
  "request_id": "The ID of the request that started this update"
}
```

--------------------------------

### Databricks SQL substring function examples

Source: https://docs.databricks.com/gcp/en/sql/language-manual/sql-ref-datatype-rules

The substring function expects STRING for the string and INTEGER for start and length. Examples demonstrate implicit type casting and promotion.

```sql
-- Promotion of TINYINT to INTEGER  
> SELECT substring('hello', 1Y, 2);  
  he  
```

```sql
-- No casting  
> SELECT substring('hello', 1, 2);  
  he  
```

```sql
-- Casting of a literal string  
> SELECT substring('hello', '1', 2);  
  he  
```

```sql
-- Downcasting of a BIGINT to an INT  
> SELECT substring('hello', 1L, 2);  
  he  
```

```sql
-- Crosscasting from STRING to INTEGER  
> SELECT substring('hello', str, 2)  
  FROM VALUES(CAST('1' AS STRING)) AS T(str);  
  he  
```

```sql
-- Crosscasting from INTEGER to STRING  
> SELECT substring(12345, 2, 2);  
  23  
```

--------------------------------

### Databricks SQL Multiple Partitioning Hints Example

Source: https://docs.databricks.com/gcp/en/sql/language-manual/sql-ref-syntax-qry-select-hints

Demonstrates applying multiple partitioning hints (REPARTITION, COALESCE, REPARTITION_BY_RANGE) to a query. The optimizer picks the leftmost hint.

```sql
-- multiple partitioning hints  
> EXPLAIN EXTENDED SELECT /*+ REPARTITION(100), COALESCE(500), REPARTITION_BY_RANGE(3, c) */ * FROM t;  
```