### Run Quickstart and Start App

Source: https://docs.databricks.com/aws/en/agents/agent-framework/author-agent

Execute quickstart scripts to install dependencies and set up the environment, then start the application.

```bash
uv run quickstart
```

```bash
uv run start-app
```

--------------------------------

### Run Quickstart Scripts

Source: https://docs.databricks.com/aws/generative-ai/agent-framework/author-agent

Execute quickstart scripts to install dependencies, set up the environment, and start the agent application locally.

```Bash
nvm use 20  

```

```Bash
uv run quickstart  
uv run start-app  

```

--------------------------------

### Basic Example

Source: https://docs.databricks.com/aws/en/pyspark/reference/classes/datastreamwriter/start

A basic example demonstrating how to use the start method to write a stream to a memory table.

```APIDOC
## Basic example:
Python
```
df = spark.readStream.format("rate").load()
q = df.writeStream.format('memory').queryName('this_query').start()
q.isActive
# True
q.name
# 'this_query'
q.stop()
q.isActive
# False
```
```

--------------------------------

### Example with Trigger and Additional Parameters

Source: https://docs.databricks.com/aws/en/pyspark/reference/classes/datastreamwriter/start

An example showing the use of the start method with a trigger and other optional parameters like queryName, outputMode, and format.

```APIDOC
## With a trigger and additional parameters:
Python
```
q = df.writeStream.trigger(processingTime='5 seconds').start(
    queryName='that_query', outputMode="append", format='memory')
q.name
# 'that_query'
q.isActive
# True
q.stop()
```
```

--------------------------------

### Example Usage

Source: https://docs.databricks.com/aws/en/pyspark/reference/classes/streamingquerymanager

This example demonstrates how to start a streaming query, access active queries through the StreamingQueryManager, wait for termination, and reset terminated queries.

```APIDOC
## Examples

Python
```python
sdf = spark.readStream.format("rate").load()  
sq = sdf.writeStream.format('memory').queryName('this_query').start()  
sqm = spark.streams  
[q.name for q in sqm.active]  
# ['this_query']  
sqm.awaitAnyTermination(5)  
# True  
sq.stop()  
sqm.resetTerminated()  
```
```

--------------------------------

### Set up environment and reset demo for COPY INTO (Python)

Source: https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/copy-into/tutorial-notebook

Configure catalog, schema, and volume parameters, then reset the demo environment by dropping and creating a schema and volume. Replace `<catalog>` with your actual catalog name.

```python
# Set parameters and reset demo environment  
  
catalog = "<catalog>"  
  
username = spark.sql("SELECT regexp_replace(session_user(), '[^a-zA-Z0-9]', '_')").first()[0]  
schema = f"copyinto_{username}_db"  
volume = "copy_into_source"  
source = f"/Volumes/{catalog}/{schema}/{volume}"  
  
spark.sql(f"SET c.catalog={catalog}")  
spark.sql(f"SET c.schema={schema}")  
spark.sql(f"SET c.volume={volume}")  
  
spark.sql(f"DROP SCHEMA IF EXISTS {catalog}.{schema} CASCADE")  
spark.sql(f"CREATE SCHEMA {catalog}.{schema}")  
spark.sql(f"CREATE VOLUME {catalog}.{schema}.{volume}")  

```

--------------------------------

### Install MLflow Package in R

Source: https://docs.databricks.com/aws/en/notebooks/source/mlflow/mlflow-quick-start-r.html

Installs the latest version of the MLflow package for R. Use this command to get started with MLflow tracking.

```r
install.packages("mlflow")
```

--------------------------------

### Display Create Repo Help

Source: https://docs.databricks.com/aws/en/archive/dev-tools/cli/repos-cli

To see detailed usage instructions and options for creating a repository, run this command.

```bash
databricks repos create --help
```

--------------------------------

### Get input file block start offset

Source: https://docs.databricks.com/aws/en/pyspark/reference/functions/input_file_block_start

This example demonstrates how to use the input_file_block_start function to retrieve the starting offset of the input file block. It reads a text file and then selects the input_file_block_start for each row.

```Python
from pyspark.sql import functions as sf
df = spark.read.text("python/test_support/sql/ages_newlines.csv", lineSep=",")
df.select(sf.input_file_block_start()).show()
```

--------------------------------

### Create, Alter, and Show Share Example

Source: https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-aux-show-all-in-share

Demonstrates the creation of a share, adding tables to it (including aliasing and partitioning), and then listing the share's content. This covers the lifecycle of managing share contents.

```sql
-- Create share `customer_share` only if share with same name doesn't exist, with a comment.  
> CREATE SHARE IF NOT EXISTS customer_share COMMENT 'This is customer share';  
  
-- Add 2 tables to the share.  
-- Expose my_schema.tab1 a different name.  
-- Expose only two partitions of other_schema.tab2  
> ALTER SHARE customer_share ADD TABLE my_schema.tab1 AS their_schema.tab1;  
> ALTER SHARE customer_share ADD TABLE other_schema.tab2 PARTITION (c1 = 5), (c1 = 7);  
  
-- List the content of the share  
> SHOW ALL IN SHARE customer_share;  
  name              type  shared_object          added_at                     added_by                   comment partitions  
  ----------------- ----- ---------------------- ---------------------------- -------------------------- ------- ------------------  
  other_schema.tab2 TABLE main.other_schema.tab2 2022-01-01T00:00:01.000+0000 alwaysworks@databricks.com NULL  
  their_schema.tab1 TABLE main.myschema.tab2     2022-01-01T00:00:00.000+0000 alwaysworks@databricks.com NULL    (c1 = 5), (c1 = 7)  
```

--------------------------------

### Basic Structured Streaming DataFrame Display (Scala)

Source: https://docs.databricks.com/aws/en/visualizations/legacy-visualizations

Visualize a simple Structured Streaming DataFrame count in Scala. This is a basic example to get started with streaming visualizations.

```scala
val streaming_df = spark.readStream.format("rate").load()  
display(streaming_df.groupBy().count())  

```

--------------------------------

### Basic Structured Streaming DataFrame Display (Python)

Source: https://docs.databricks.com/aws/en/visualizations/legacy-visualizations

Visualize a simple Structured Streaming DataFrame count in Python. This is a basic example to get started with streaming visualizations.

```python
streaming_df = spark.readStream.format("rate").load()  
display(streaming_df.groupBy().count())  

```

--------------------------------

### Local Agent Setup and Launch

Source: https://docs.databricks.com/aws/en/agents/agent-framework/multi-agent-apps

Use `uv run quickstart` to configure Databricks authentication and set up an MLflow experiment. Then, use `uv run start-app` to launch the agent server and chat UI.

```bash
uv run quickstart

```

```bash
uv run start-app

```

--------------------------------

### Get Workspace API Example

Source: https://docs.databricks.com/aws/en/admin/workspace/create-workspace-api

Example of how to call the Get Workspace API to confirm the status of a newly created workspace.

```APIDOC
## Get Workspace API

To check the status of a newly created workspace, you can use the Get Workspace API. You will need the `workspace_id` obtained from the create workspace response.

### Method
GET

### Endpoint
`https://accounts.cloud.databricks.com/api/2.0/accounts/<databricks-account-id>/workspaces/<databricks-workspace-id>`

### Parameters
#### Path Parameters
- **databricks-account-id** (string) - Required - Your Databricks account ID.
- **databricks-workspace-id** (integer) - Required - The ID of the workspace to retrieve status for.

### Request Example
```bash
curl -X GET \
'https://accounts.cloud.databricks.com/api/2.0/accounts/<databricks-account-id>/workspaces/<databricks-workspace-id>' \
  --header 'Authorization: Bearer $OAUTH_TOKEN'
```

### Response
#### Success Response (200)
- **workspace_id** (integer) - The ID of the workspace.
- **workspace_name** (string) - The name of the workspace.
- **aws_region** (string) - The AWS region of the workspace.
- **creation_time** (integer) - The creation timestamp.
- **deployment_name** (string) - The deployment name.
- **workspace_status** (string) - The current status of the workspace (e.g., RUNNING, PROVISIONING, FAILED).
- **account_id** (string) - The Databricks account ID.
- **credentials_id** (string) - The ID of the AWS credentials.
- **storage_configuration_id** (string) - The ID of the storage configuration.
- **workspace_status_message** (string) - A message detailing the workspace status.
- **network_id** (string) - The network configuration ID.
- **managed_services_customer_managed_key_id** (string) - The customer-managed key ID for managed services.
- **storage_customer_managed_key_id** (string) - The customer-managed key ID for storage.
- **pricing_tier** (string) - The pricing tier of the workspace.

#### Response Example
```json
{
  "workspace_id": 123456789,
  "workspace_name": "my-company-example",
  "aws_region": "us-west-2",
  "creation_time": 1579768294842,
  "deployment_name": "my-company-example",
  "workspace_status": "RUNNING",
  "account_id": "<databricks-account-id>",
  "credentials_id": "<aws-credentials-id>",
  "storage_configuration_id": "<databricks-storage-config-id>",
  "workspace_status_message": "Workspace is running.",
  "network_id": "339f16b9-b8a3-4d50-9d1b-7e29e49448c3",
  "managed_services_customer_managed_key_id": "<aws-kms-managed-services-key-id>",
  "storage_customer_managed_key_id": "<aws-kms-notebook-workspace-storage-id>",
  "pricing_tier": "ENTERPRISE"
}
```

### Workspace Status Meanings
- **NOT_PROVISIONED**: Not yet provisioned.
- **PROVISIONING**: Still provisioning. Wait a few minutes and repeat this API request.
- **RUNNING**: Successful deployment and now running.
- **FAILED**: Failed deployment.
- **BANNED**: Banned.
- **CANCELLING**: In process of cancellation.
```

--------------------------------

### Example of SQL GET Statement

Source: https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-aux-connector-get

This example demonstrates how to use the GET statement to download a CSV file from a specified volume path to a local file path.

```sql
> GET '/Volumes/test_catalog/test_schema/test_volume/2023/06/file1.csv' TO '/home/bob/data.csv'  
```

--------------------------------

### Start Database Instance (API)

Source: https://docs.databricks.com/aws/en/oltp/create

This snippet demonstrates how to start a database instance using a direct API call.

```APIDOC
## Start Database Instance (API)

### Description
Starts a specified database instance using a direct API call.

### Method
PATCH

### Endpoint
`/api/2.0/database/instances/$INSTANCE_NAME`

### Headers
- **Authorization**: Bearer $DATABRICKS_TOKEN

### Request Body
```json
{
  "stopped": false
}
```

### Request Example
```bash
curl -X PATCH --header "Authorization: Bearer $DATABRICKS_TOKEN" https://$WORKSPACE/api/2.0/database/instances/$INSTANCE_NAME \
--data-binary @- << EOF
{
  "stopped": false
}
EOF
```
```

--------------------------------

### Get Registered Model (Example)

Source: https://docs.databricks.com/aws/en/dev-tools/cli/reference/registered-models-commands

Example of retrieving information for a specific registered model.

```bash
databricks registered-models get main.my_schema.my_model
```

--------------------------------

### Full Configuration Example

Source: https://docs.databricks.com/aws/en/repos/serverless-private-git

A comprehensive example demonstrating default settings and specific configurations for multiple Git remotes, including SSL verification, CA certificates, HTTP proxies, and custom ports.

```JSON
{
  "default": {
    "sslVerify": true,
    "caCertPath": "/Workspace/my_ca_cert.pem",
    "httpProxy": "https://git-proxy-server.company.com",
    "customHttpPort": "8080"
  },
  "remotes": [
    {
      "urlPrefix": "https://my-private-git.company.com/",
      "caCertPath": "/Workspace/my_ca_cert_2.pem"
    },
    {
      "urlPrefix": "https://another-git-server.com/project.git",
      "sslVerify": false
    }
  ]
}
```

--------------------------------

### Install Databricks Agents and Restart Python

Source: https://docs.databricks.com/aws/en/notebooks/source/generative-ai/agent-bricks-knowledge-assistant-evaluation.html

Installs the databricks-agents library and restarts the Python environment. This is a simplified setup compared to the full installation.

```python
%pip install databricks-agents==1.6.0
dbutils.library.restartPython()
```

--------------------------------

### Enter REPL Mode for Default Database Example

Source: https://docs.databricks.com/aws/en/dev-tools/databricks-sql-cli

Example of entering REPL mode scoped to the 'default' database.

```Bash
dbsqlcli default
```

--------------------------------

### PySpark overlay Function Examples

Source: https://docs.databricks.com/aws/en/pyspark/reference/functions/overlay

Demonstrates how to use the overlay function with different parameters. The first example replaces a portion of 'SPARK_SQL' with 'CORE' starting at position 7. The second example explicitly sets the length to 0, effectively inserting 'CORE' at position 7. The third example replaces 2 bytes starting at position 7 with 'y'.

```python
from pyspark.sql import functions as dbf  
df = spark.createDataFrame([("SPARK_SQL", "CORE")], ("x", "y"))  
df.select("*", dbf.overlay("x", df.y, 7)).show()  
```

```python
df.select("*", dbf.overlay("x", df.y, 7, 0)).show()  
```

```python
df.select("*", dbf.overlay("x", "y", 7, 2)).show()  
```

--------------------------------

### MLflow GenAI Prompt Optimization Quickstart Example

Source: https://docs.databricks.com/aws/en/mlflow3/genai/tutorials/examples/prompt-optimization-quickstart

This is a quickstart example for optimizing a prompt using MLflow GenAI Prompt Optimization with GEPA and GPT-OSS 20B. It demonstrates prompt optimization for classification tasks.

```python
%md
#Quickstart example

Below is a quickstart example that optimizes a simple prompt **classify this query**. We will use GPT-OSS 20B for this example.
```

--------------------------------

### Complete Gateway Configuration Example

Source: https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/gateway-event-logs

Example of a full gateway pipeline specification with progress events enabled and set to emit every five minutes.

```Python
gateway_pipeline_spec = {
   "pipeline_type": "INGESTION_GATEWAY",
   "name": "my_gateway_pipeline",
   "catalog": "main",
   "target": "my_schema",
   "continuous": True,
   "configuration": {
      "pipelines.gateway.progressEventsEnabled": "true",
      "pipelines.gateway.progressEventEmitFrequencySeconds": "300"
   },
   # ... rest of pipeline spec
}

```

--------------------------------

### SQL Examples for EXECUTE IMMEDIATE with Parameters

Source: https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-parameter-marker

Shows dynamic SQL execution using named parameter markers for DECIMAL precision/scale, view creation, and table creation with default values and comments.

```sql
> EXECUTE IMMEDIATE 'SELECT 1::DECIMAL(:precision, :scale)' USING 6 AS precision, 4 AS scale;  
  1.0000  
  
> EXECUTE IMMEDIATE 'CREATE VIEW v(c1 INT) AS SELECT :val AS c1' USING 10 AS val;  
> SELECT * FROM v;  
  10  
  
> EXECUTE IMMEDIATE 'CREATE TABLE T(c1 INT DEFAULT :def COMMENT \'This is a \' :com)' USING 17 AS def, 'comment' AS com;  
```

--------------------------------

### Setup Omnigent Credentials

Source: https://docs.databricks.com/aws/en/omnigent/quickstart

Run the 'omni setup' command to initiate the credential setup process. This command guides you through adding credentials for agent harnesses.

```bash
omni setup  

```

--------------------------------

### Example of DESCRIBE SHARE in SQL

Source: https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-aux-describe-share

This example demonstrates how to create a share and then use the DESCRIBE SHARE command to view its metadata. The output shows the share's name, creation timestamp, owner, and comment.

```sql
> CREATE SHARE vaccine COMMENT 'vaccine data to publish';
> DESCRIBE SHARE vaccine;
  name      created_at                   created_by                 comment  
  --------- ---------------------------- -------------------------- -----------------------
  vaccine   2022-01-01T00:00:00.000+0000 alwaysworks@databricks.com vaccine data to publish  

```

--------------------------------

### Get help for install command

Source: https://docs.databricks.com/aws/en/archive/dev-tools/dbutils-library

Displays help information specifically for the `install` command within `dbutils.library`.

```Python
dbutils.library.help("install")
```

--------------------------------

### Delta Live Tables Quickstart (SQL)

Source: https://docs.databricks.com/aws/en/ldp/hive-metastore

Import this notebook into a Databricks workspace to deploy an example Delta Live Tables pipeline using SQL. It covers reading raw JSON, cleaning data with expectations, and performing analysis.

```sql
%md # Delta Live Tables quickstart (SQL)

A notebook that provides an example Delta Live Tables pipeline to:
- Read raw JSON clickstream data into a table.
- Read records from the raw data table and use a Delta Live Tables query and expectations to create a new table with cleaned and prepared data.
- Perform an analysis on the prepared data with a Delta Live Tables query.
```

--------------------------------

### PySpark Shell Output Example

Source: https://docs.databricks.com/aws/en/dev-tools/databricks-connect-legacy

This is an example of the output you will see when starting the PySpark shell with Databricks Connect.

```text
Python 3... (v3...)  
[Clang 6... (clang-6...)] on darwin  
Type "help", "copyright", "credits" or "license" for more information.  
Setting default log level to "WARN".  
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).  
../../.. ..:..:.. WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable  
Welcome to  
       ____              __  
     / __/__  ___ _____/ /__  
    _	\ 	/ _ \/ _ `/ __/  '_/  
   /__ / .__/	,_	/_	/ /_/	\_	  version 3....  
      /_/  
  
Using Python version 3... (v3...)  
Spark context Web UI available at http://...:...  
Spark context available as 'sc' (master = local[*], app id = local-...).  
SparkSession available as 'spark'.  
>>>  

```

--------------------------------

### SQL Example: Create and Truncate Table with Partition

Source: https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-ddl-truncate-table

Demonstrates creating a partitioned table, inserting data, truncating a specific partition, and then truncating the entire table.

```sql
-- Create table Student with partition  
> CREATE TABLE Student (name STRING, rollno INT) PARTITIONED BY (age INT);  
  
> SELECT * FROM Student;  
 name rollno age  
 ---- ------ ---  
  ABC      1  10  
  DEF      2  10  
  XYZ      3  12  
  
-- Remove all rows from the table in the specified partition  
> TRUNCATE TABLE Student partition(age=10);  
  
-- After truncate execution, records belonging to partition age=10 are removed  
> SELECT * FROM Student;  
 name rollno age  
 ---- ------ ---  
  XYZ      3  12  
  
-- Remove all rows from the table from all partitions  
> TRUNCATE TABLE Student;  
  
> SELECT * FROM Student;  
 name rollno age  
 ---- ------ ---  


```

--------------------------------

### Delta Live Tables Quickstart (Python)

Source: https://docs.databricks.com/aws/en/ldp/hive-metastore

Import this notebook into a Databricks workspace to deploy an example Delta Live Tables pipeline. It demonstrates reading raw JSON, cleaning data with expectations, and performing analysis.

```python
# Delta Live Tables quickstart (Python)
A notebook that provides an example Delta Live Tables pipeline to:
  * Read raw JSON clickstream data into a table.
  * Read records from the raw data table and use a Delta Live Tables query and expectations to create a new table with cleaned and prepared data.
  * Perform an analysis on the prepared data with a Delta Live Tables query.
```

--------------------------------

### CrewAI Trip Planning Example

Source: https://docs.databricks.com/aws/en/mlflow3/genai/tracing/integrations/crewai

This Python code defines agents and tasks for a trip planning application using CrewAI, demonstrating how to structure complex agent interactions for tasks like city selection and local guide generation. It includes the setup for the Crew object and a sample execution.

```python
def identify_task(self, agent, origin, cities, interests, range):
    return Task(
        description=dedent(
            f"""
            Analyze and select the best city for the trip based
            on specific criteria such as weather patterns, seasonal
            events, and travel costs. This task involves comparing
            multiple cities, considering factors like current weather
            conditions, upcoming cultural or seasonal events, and
            overall travel expenses.
            Your final answer must be a detailed
            report on the chosen city, and everything you found out
            about it, including the actual flight costs, weather
            forecast and attractions.

            Traveling from: {origin}
            City Options: {cities}
            Trip Date: {range}
            Traveler Interests: {interests}
        """
        ),
        agent=agent,
        expected_output="Detailed report on the chosen city including flight costs, weather forecast, and attractions",
    )

def gather_task(self, agent, origin, interests, range):
    return Task(
        description=dedent(
            f"""
            As a local expert on this city you must compile an
            in-depth guide for someone traveling there and wanting
            to have THE BEST trip ever!
            Gather information about key attractions, local customs,
            special events, and daily activity recommendations.
            Find the best spots to go to, the kind of place only a
            local would know.
            This guide should provide a thorough overview of what
            the city has to offer, including hidden gems, cultural
            hotspots, must-visit landmarks, weather forecasts, and
            high level costs.
            The final answer must be a comprehensive city guide,
            rich in cultural insights and practical tips,
            tailored to enhance the travel experience.

            Trip Date: {range}
            Traveling from: {origin}
            Traveler Interests: {interests}
        """
        ),
        agent=agent,
        expected_output="Comprehensive city guide including hidden gems, cultural hotspots, and practical travel tips",
    )


class TripCrew:
    def __init__(self, origin, cities, date_range, interests):
        self.cities = cities
        self.origin = origin
        self.interests = interests
        self.date_range = date_range

    def run(self):
        agents = TripAgents()
        tasks = TripTasks()

        city_selector_agent = agents.city_selection_agent()
        local_expert_agent = agents.local_expert()

        identify_task = tasks.identify_task(
            city_selector_agent,
            self.origin,
            self.cities,
            self.interests,
            self.date_range,
        )
        gather_task = tasks.gather_task(
            local_expert_agent, self.origin, self.interests, self.date_range
        )

        crew = Crew(
            agents=[city_selector_agent, local_expert_agent],
            tasks=[identify_task, gather_task],
            verbose=True,
            memory=True,
            knowledge={
                "sources": [string_source],
                "metadata": {"preference": "personal"},
            },
        )

        result = crew.kickoff()
        return result


trip_crew = TripCrew("California", "Tokyo", "Dec 12 - Dec 20", "sports")
result = trip_crew.run()

```

--------------------------------

### Setup Ray Cluster with Spark

Source: https://docs.databricks.com/aws/en/machine-learning/ray/spark-ray-overview

Configure and start a Ray cluster within a Spark environment. This example demonstrates how to allocate resources for Ray workers, ensuring sufficient availability for both Ray and Spark operations. Adjust `min_worker_nodes` and `max_worker_nodes` to match your cluster's autoscaling configuration and resource needs.

```python
from ray.util.spark import setup_ray_cluster, shutdown_ray_cluster  

# For a Databricks cluster configured with autoscaling enabled,  
# The minimum worker nodes of 4 and maximum of 6 nodes.  
# 2 Spark-only nodes will launch when needed.  
# The Ray cluster will have 4 nodes allocated for its use.  
setup_ray_cluster(  
  min_worker_nodes=4,  
  max_worker_nodes=4,  
)  

# Pass any custom Ray configuration with ray.init  
ray.init()  

```

--------------------------------

### Set Up Virtual Environment and Install Dependencies

Source: https://docs.databricks.com/aws/en/ingestion/community-build

Create a Python virtual environment and install the necessary development dependencies for building community connectors. This ensures a consistent and isolated development environment.

```bash
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
```

--------------------------------

### Using Default Input Partition

Source: https://docs.databricks.com/aws/en/pyspark/reference/classes/inputpartition

Example demonstrating how to use the default InputPartition implementation within the partitions() method.

```APIDOC
## Examples
Use the default input partition implementation:
Python
```python
def partitions(self):
    return [InputPartition(1)]
```
```

--------------------------------

### Pydantic-AI Weather Agent Example

Source: https://docs.databricks.com/aws/en/mlflow3/genai/tracing/integrations/pydantic-ai

This Python script defines a Pydantic-AI agent that uses tools to get weather information for specified locations. It requires setting up dependencies like an AsyncClient and API keys for weather and geocoding services. The agent is configured with a system prompt to guide its tool usage.

```Python
import os
from dataclasses import dataclass
from typing import Any

from httpx import AsyncClient

from pydantic_ai import Agent, ModelRetry, RunContext


@dataclass
class Deps:
    client: AsyncClient
    weather_api_key: str | None
    geo_api_key: str | None


weather_agent = Agent(
    # Switch to your favorite LLM
    "google-gla:gemini-2.0-flash",
    # 'Be concise, reply with one sentence.' is enough for some models (like openai) to use
    # the below tools appropriately, but others like anthropic and gemini require a bit more direction.
    system_prompt=(
        "Be concise, reply with one sentence."
        "Use the `get_lat_lng` tool to get the latitude and longitude of the locations, "
        "then use the `get_weather` tool to get the weather."
    ),
    deps_type=Deps,
    retries=2,
    instrument=True,
)


@weather_agent.tool
async def get_lat_lng(
    ctx: RunContext[Deps], location_description: str
) -> dict[str, float]:
    """Get the latitude and longitude of a location.  

    Args:  
        ctx: The context.  
        location_description: A description of a location.  
    """
    if ctx.deps.geo_api_key is None:
        return {"lat": 51.1, "lng": -0.1}

    params = {
        "q": location_description,
        "api_key": ctx.deps.geo_api_key,
    }
    r = await ctx.deps.client.get("https://geocode.maps.co/search", params=params)
    r.raise_for_status()
    data = r.json()

    if data:
        return {"lat": data[0]["lat"], "lng": data[0]["lon"]}
    else:
        raise ModelRetry("Could not find the location")


@weather_agent.tool
async def get_weather(ctx: RunContext[Deps], lat: float, lng: float) -> dict[str, Any]:
    """Get the weather at a location.  

    Args:  
        ctx: The context.  
        lat: Latitude of the location.  
        lng: Longitude of the location.  
    """

    if ctx.deps.weather_api_key is None:
        return {"temperature": "21 °C", "description": "Sunny"}

    params = {
        "apikey": ctx.deps.weather_api_key,
        "location": f"{lat},{lng}",
        "units": "metric",
    }
    r = await ctx.deps.client.get(
        "https://api.tomorrow.io/v4/weather/realtime", params=params
    )
    r.raise_for_status()
    data = r.json()

    values = data["data"]["values"]
    # https://docs.tomorrow.io/reference/data-layers-weather-codes
    code_lookup = {
        1000: "Clear, Sunny",
        1100: "Mostly Clear",
        1101: "Partly Cloudy",
        1102: "Mostly Cloudy",
        1001: "Cloudy",
        2000: "Fog",
        2100: "Light Fog",
        4000: "Drizzle",
        4001: "Rain",
        4200: "Light Rain",
        4201: "Heavy Rain",
        5000: "Snow",
        5001: "Flurries",
        5100: "Light Snow",
        5101: "Heavy Snow",
        6000: "Freezing Drizzle",
        6001: "Freezing Rain",
        6200: "Light Freezing Rain",
        6201: "Heavy Freezing Rain",
        7000: "Ice Pellets",
        7101: "Heavy Ice Pellets",
        7102: "Light Ice Pellets",
        8000: "Thunderstorm",
    }
    return {
        "temperature": f'{values["temperatureApparent"]:0.0f}°C',
        "description": code_lookup.get(values["weatherCode"], "Unknown"),
    }


async def main():
    async with AsyncClient() as client:
        weather_api_key = os.getenv("WEATHER_API_KEY")
        geo_api_key = os.getenv("GEO_API_KEY")
        deps = Deps(
            client=client, weather_api_key=weather_api_key, geo_api_key=geo_api_key
        )
        result = await weather_agent.run(
            "What is the weather like in London and in Wiltshire?", deps=deps
        )
        print("Response:", result.output)


# If you are running this on a notebook
await main()

# Uncomment this is you are using an IDE or Python script.
# asyncio.run(main())

```

--------------------------------

### Create and Show Shares

Source: https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-aux-show-shares

Demonstrates creating two shares and then listing all shares in the metastore.

```sql
> CREATE SHARE vaccine COMMENT 'vaccine data to publish';
> CREATE SHARE meds COMMENT 'meds data to publish';
> SHOW SHARES;
  name      created_at                   created_by                 comment  
  ---------	----------------------------	--------------------------	-----------------------
  vaccine 	2022-01-01T00:00:00.000+0000	alwaysworks@databricks.com	vaccine data to publish  
  meds 	2022-01-01T00:00:01.000+0000	alwaysworks@databricks.com	meds data to publish  
  
```

--------------------------------

### Scala Spark Shell Output Example

Source: https://docs.databricks.com/aws/en/dev-tools/databricks-connect-legacy

This is an example of the output you will see when starting the Scala Spark shell with Databricks Connect.

```text
Setting default log level to "WARN".  
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).  
../../.. ..:..:.. WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable  
Spark context Web UI available at http://...  
Spark context available as 'sc' (master = local[*], app id = local-...).  
Spark session available as 'spark'.  
Welcome to  
      ____              __  
     / __/__  ___ _____/ /__  
    _	\ 	/ _ \/ _ `/ __/  '_/  
   /___/ .__/	,_	/_	/ /_/	\_	  version 3...  
      /_/  
  
Using Scala version 2... (OpenJDK 64-Bit Server VM, Java 1.8...)  
Type in expressions to have them evaluated.  
Type :help for more information.  
  
scala>  

```

--------------------------------

### Install Databricks Connect

Source: https://docs.databricks.com/aws/en/dev-tools/databricks-connect/python/tutorial-serverless

Install the Databricks Connect package using pip. Specify the desired version, for example, '17.3.*'.

```bash
pip install "databricks-connect==17.3.*"  
```

--------------------------------

### Start Database Instance (CLI)

Source: https://docs.databricks.com/aws/en/oltp/create

This snippet shows how to start a database instance using the Databricks CLI.

```APIDOC
## Start Database Instance (CLI)

### Description
Starts a specified database instance using the Databricks CLI.

### Command
`databricks database update-database-instance`

### Parameters
- **name** (string) - Required - The name of the database instance to start.
- **update_mask** (string) - Required - Specifies which fields to update. Use `"*"` to update all fields.
- **json** (string) - Required - A JSON string containing the instance update payload.
  - **stopped** (boolean) - Required - Set to `False` to start the instance.

### Request Example
```bash
databricks database update-database-instance my-database-instance '*' \
  --json '{  
    "stopped": false  
  }'
```
```

--------------------------------

### Use Catalog Example

Source: https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-names

Example of switching the current catalog to 'hive_metastore'.

```SQL
> USE CATALOG hive_metastore;
```

--------------------------------

### Install Dependencies and Run Node.js App

Source: https://docs.databricks.com/aws/en/dev-tools/databricks-apps/tutorial-node

Use npm commands to install project dependencies and start your Node.js application locally.

```bash
npm install
```

```bash
npm run start
```

--------------------------------

### Example Output of SHOW CONNECTIONS

Source: https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-aux-show-connections

This example displays the typical output format when running the SHOW CONNECTIONS command. It includes columns for connection name, type, creation timestamp, creator, and comment.

```sql
> SHOW CONNECTIONS;  
 name                connection_type created_at                   created_by    comment  
 ------------------- --------------- ---------------------------- ------------- ---------------------
 mysql_connection    mysql           2022-01-01T00:00:00.000+0000 alf@melmak.et mysql connection  
 postgres_connection postgresql      2022-06-12T13:30:00.000+0000 alf@melmak.et postgresql connection  

```

--------------------------------

### Install a specific version of Databricks SDK for Python with pip

Source: https://docs.databricks.com/aws/en/dev-tools/sdk-python

Install a specific version of the databricks-sdk package, for example, version 0.1.6.

```bash
pip3 install databricks-sdk==0.1.6  

```

--------------------------------

### Initialize Streaming Query

Source: https://docs.databricks.com/aws/en/pyspark/reference/classes/streamingquery/explain

Sets up a sample streaming query using the 'rate' source and 'memory' sink, which is necessary before calling explain().

```Python
sdf = spark.readStream.format("rate").load()
sq = sdf.writeStream.format('memory').queryName('query_explain').start()
sq.processAllAvailable()
```

--------------------------------

### Create DataFrame for nth_value Example

Source: https://docs.databricks.com/aws/en/pyspark/reference/functions/nth_value

Creates a sample DataFrame to demonstrate the nth_value function. This setup is common for window function examples.

```python
from pyspark.sql import functions as sf
from pyspark.sql import Window
df = spark.createDataFrame(
    [("a", 1), ("a", 2), ("a", 3), ("b", 8), ("b", 2)], ["c1", "c2"])
df.show()
```

--------------------------------

### Multiple Partitioning Hints Example

Source: https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-qry-select-hints

Illustrates the use of multiple partitioning hints in a single query. The optimizer picks the leftmost hint when multiple are specified.

```sql
-- multiple partitioning hints  
> EXPLAIN EXTENDED SELECT /*+ REPARTITION(100), COALESCE(500), REPARTITION_BY_RANGE(3, c) */ * FROM t;  
```

```text
== Parsed Logical Plan ==  
'UnresolvedHint REPARTITION, [100]  
+- 'UnresolvedHint COALESCE, [500]  
   +- 'UnresolvedHint REPARTITION_BY_RANGE, [3, 'c]  
      +- 'Project [*]  
         +- 'UnresolvedRelation [t]  
  
== Analyzed Logical Plan ==  
name: string, c: int  
Repartition 100, true  
+- Repartition 500, false  
   +- RepartitionByExpression [c#30 ASC NULLS FIRST], 3  
      +- Project [name#29, c#30]  
         +- SubqueryAlias spark_catalog.default.t  
            +- Relation[name#29,c#30] parquet  
  
== Optimized Logical Plan ==  
Repartition 100, true  
+- Relation[name#29,c#30] parquet  
  
== Physical Plan ==  
Exchange RoundRobinPartitioning(100), false, [id=#121]  
+- *(1) ColumnarToRow  
   +- FileScan parquet default.t[name#29,c#30] Batched: true, DataFilters: [], Format: Parquet,  
      Location: CatalogFileIndex[file:/spark/spark-warehouse/t], PartitionFilters: [],  
      PushedFilters: [], ReadSchema: struct<name:string>  
```

--------------------------------

### Install a private package from a repository with Databricks secrets

Source: https://docs.databricks.com/aws/en/libraries/notebooks-python-libraries

Install a package from a private repository using credentials managed by Databricks secrets. This example shows how to retrieve a token and use it in the installation command.

```python
token = dbutils.secrets.get(scope="scope", key="key")  
```

```bash
%pip install --index-url https://<user>:$token@<your-package-repository>.com/<path/to/repo> <package>==<version> --extra-index-url https://pypi.org/simple/  
```

--------------------------------

### Example: Create a Standard Catalog

Source: https://docs.databricks.com/aws/en/catalogs/create-catalog

This is an example of creating a catalog named 'example' using the standard catalog creation SQL command.

```sql
CREATE CATALOG IF NOT EXISTS example;  
```

--------------------------------

### Go Zerobus SDK Installation

Source: https://docs.databricks.com/aws/en/ingestion/zerobus-ingest

Install the Go Zerobus SDK using the go get command. Requires Go 1.21 or higher.

```Bash
go get github.com/databricks/zerobus-sdk/go@latest

```

--------------------------------

### Extracting a subset of an array

Source: https://docs.databricks.com/aws/en/sql/language-manual/functions/slice

Use `slice` to get a sub-array starting from a specified index with a given length. Array indices start at 1.

```sql
> SELECT slice(array(1, 2, 3, 4), 2, 2);
 [2,3]
```

```sql
> SELECT slice(array(1, 2, 3, 4), -2, 2);
 [3,4]
```

--------------------------------

### Set up Catalog and Schema

Source: https://docs.databricks.com/aws/en/notebooks/source/machine-learning/feature-store-with-uc-taxi-example.html

Use SQL to set the catalog and schema for storing feature tables. Ensure you have the necessary privileges.

```sql
USE CATALOG ml;

CREATE SCHEMA IF NOT EXISTS taxi_example;
USE SCHEMA taxi_example;
```

--------------------------------

### Create Example Catalog and Schema

Source: https://docs.databricks.com/aws/en/notebooks/source/unity-catalog-external-table-example-aws.html

Sets up a new catalog and schema in Unity Catalog to house the external table. Ensure you have the necessary permissions to create catalogs and schemas.

```sql
CREATE CATALOG IF NOT EXISTS example_catalog;
USE CATALOG example_catalog;
CREATE SCHEMA IF NOT EXISTS example_schema;
USE example_schema;
```

--------------------------------

### Install Databricks Feature Store

Source: https://docs.databricks.com/aws/en/notebooks/source/machine-learning/feature-store-taxi-example.html

Install the Databricks Feature Store library if not using Databricks Runtime ML. Run this at the start of the notebook.

```python
%pip install databricks-feature-store
```

--------------------------------

### Example: Database Instance with Catalog

Source: https://docs.databricks.com/aws/en/dev-tools/bundles/resources

This example demonstrates how to define a database instance and associate it with a database catalog. It shows the dependency where the catalog uses the instance's name.

```yaml
resources:
  database_instances:
    my_instance:
      name: my-instance
      capacity: CU_1
  database_catalogs:
    my_catalog:
      database_instance_name: ${resources.database_instances.my_instance.name}
      name: example_catalog
      database_name: my_database
      create_database_if_not_exists: true

```

--------------------------------

### Create and Populate a Partitioned Table

Source: https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-aux-show-partitions

Demonstrates creating a table partitioned by 'state' and 'city', and then inserting data into specific partitions.

```sql
USE salesdb;  
CREATE TABLE customer(id INT, name STRING) PARTITIONED BY (state STRING, city STRING);
INSERT INTO customer PARTITION (state = 'CA', city = 'Fremont') VALUES (100, 'John');  
INSERT INTO customer PARTITION (state = 'CA', city = 'San Jose') VALUES (200, 'Marry');  
INSERT INTO customer PARTITION (state = 'AZ', city = 'Peoria') VALUES (300, 'Daniel');  
```