### Configure Databricks CLI (Bash)

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/setup.md

Illustrates methods for configuring the Databricks CLI, including interactive token-based setup and manual configuration via the `.databrickscfg` file. Provides examples for different environments (default, dev, prod).

```bash
databricks configure --token

# Manual configuration:
# Create ~/.databrickscfg
# [DEFAULT]
# host = https://<workspace>.cloud.databricks.com
# token = dapi...
#
# [dev]
# host = https://dev-workspace.cloud.databricks.com
# token = dapi...
#
# [prod]
# host = https://prod-workspace.cloud.databricks.com
# token = dapi...

# Test CLI:
databricks workspace ls /
databricks clusters list
```

--------------------------------

### Jupyter Notebook Setup for Databricks Connect

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/databricks-connect.md

Provides a step-by-step guide for setting up a Jupyter Notebook to work with Databricks Connect. This includes installing the `databricks-connect` package, creating a Databricks session, testing the connection, and performing basic data analysis.

```python
# notebook_setup.ipynb

# Cell 1: Install and import
!pip install databricks-connect

from databricks.connect import DatabricksSession
from pyspark.sql.functions import *

# Cell 2: Create session
spark = DatabricksSession.builder.getOrCreate()

# Cell 3: Test connection
print(f"Spark version: {spark.version}")
spark.range(10).show()

# Cell 4: Your analysis
df = spark.table("main.sales.customers")
df.show()
```

--------------------------------

### Install Databricks CLI (Bash)

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/setup.md

Shows how to install the Databricks Command Line Interface (CLI) using pip or conda. Includes a command to verify the installation by checking the CLI version.

```bash
pip install databricks-cli

conda install -c databricks databricks-cli

databricks --version
```

--------------------------------

### Databricks Connect Configuration File Setup

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/databricks-connect.md

This example shows how to configure Databricks Connect using a configuration file (`~/.databrickscfg`). It demonstrates setting up a default profile and multiple environment-specific profiles (dev, prod) with their respective host, token, and cluster IDs. This method is an alternative to setting environment variables.

```ini
[DEFAULT]
host = https://your-workspace.cloud.databricks.com
token = dapiXXXXXXXXXXXXXXXXXXXXXXXXXXXX
cluster_id = 1234-567890-abc123

[dev]
host = https://dev-workspace.cloud.databricks.com
token = dapiYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
cluster_id = 5678-901234-def456

[prod]
host = https://prod-workspace.cloud.databricks.com
token = dapiZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ
cluster_id = 9012-345678-ghi789
```

--------------------------------

### VS Code Setup for Databricks Connect

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/databricks-connect.md

Provides instructions for setting up Visual Studio Code to work with Databricks Connect. This includes installing the Python extension, configuring the Python environment, setting environment variables via a `.env` file, and creating a launch configuration for debugging.

```bash
# Install VS Code Python extension
code --install-extension ms-python.python
```

```json
{
  "python.defaultInterpreterPath": "${workspaceFolder}/venv/bin/python",
  "python.envFile": "${workspaceFolder}/.env",
  "python.testing.pytestEnabled": true,
  "python.testing.pytestArgs": ["tests"]
}
```

```bash
DATABRICKS_HOST=https://your-workspace.cloud.databricks.com
DATABRICKS_TOKEN=dapiXXXXXXXXXXXXXXXXXXXXXXXXXXXX
DATABRICKS_CLUSTER_ID=1234-567890-abc123

```

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Python: Current File",
      "type": "python",
      "request": "launch",
      "program": "${file}",
      "console": "integratedTerminal",
      "envFile": "${workspaceFolder}/.env"
    }
  ]
}

```

--------------------------------

### Install and Verify Databricks Connect

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/databricks-connect.md

These bash commands demonstrate how to install the Databricks Connect client library using pip and verify the installation by checking its version. It shows installation for the latest version and for a specific Databricks Runtime version. Ensure you have Python and pip installed.

```bash
# Install Databricks Connect for DBR 13.0+
pip install databricks-connect

# Or install for specific Databricks Runtime version
pip install databricks-connect==13.3.*

# Verify installation
databricks-connect --version
```

--------------------------------

### Install Databricks Connect (Python)

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/setup.md

Shows how to install Databricks Connect using pip, emphasizing the need to match the installed version with the Databricks cluster version. This is crucial for enabling IDE connectivity.

```bash
# Install Databricks Connect (must match cluster version)
pip install databricks-connect==13.3.*
```

--------------------------------

### Complete SKILL.md Example

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/skills/SKILL-FORMAT-GUIDE.md

A comprehensive example of a correctly formatted SKILL.md file, demonstrating the YAML frontmatter and the main content structure.

```markdown
---
title: "AI & Agents Skills"
description: "Build intelligent applications with AI agents, vector search, and natural language interfaces"
category: "ai-agents"
tags:
  [
    "ai",
    "agents",
    "vector-search",
    "rag",
    "model-serving",
    "genie",
    "chatbots",
    "semantic-search",
    "llm",
  ]
priority: "high"
skills_count: 5
last_updated: 2026-02-27
version: "1.0.0"
status: "partial"
---

# AI & Agents Skills

This directory contains skills and patterns for building AI applications and agents on Databricks.

## 📚 Skills Overview

### High Priority Skills

#### 1. Databricks Agent Bricks

**Status**: ⚠️ Not yet documented
**Priority**: HIGH
**Description**: Build production-ready AI agents using Databricks Agent Bricks framework

**Key Topics**:

- Agent creation and deployment
- Tool integration and function calling
- Agent monitoring and debugging

... (rest of content)

```

--------------------------------

### PyCharm Setup for Databricks Connect

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/databricks-connect.md

Guides users through configuring PyCharm for Databricks Connect development. It covers setting up the Python interpreter, configuring environment variables for Databricks connection details, and adjusting run configurations.

```text
1. Go to `File` → `Settings` → `Project` → `Python Interpreter`
2. Add new environment or select existing
3. Install `databricks-connect` package

1. Go to `Run` → `Edit Configurations`
2. Add environment variables:
   - `DATABRICKS_HOST`
   - `DATABRICKS_TOKEN`
   - `DATABRICKS_CLUSTER_ID`

# Create run configuration with environment variables
# File → Settings → Build, Execution, Deployment → Console → Python Console

```

--------------------------------

### Initial Setup and Running Validation

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/tests/validation/QUICK-REFERENCE.md

This section covers the initial setup steps, including navigating to the validation directory, installing dependencies, and setting API keys. It also provides the easiest methods to run the validation scripts using provided shell scripts or directly via Python.

```bash
cd tests/validation
pip install -r requirements.txt
export ANTHROPIC_API_KEY="your-key-here"
```

```bash
bash validate-now.sh      # Linux/Mac
validate-now.bat          # Windows
```

```python
python agent_validator.py --provider anthropic --scope full --interactive
```

--------------------------------

### Install and Configure Databricks Connect for Jupyter

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/setup.md

Provides instructions for installing Jupyter and the Databricks Connect package, followed by code to establish a Spark session using Databricks Connect within a Jupyter notebook. This enables running Databricks code locally or in a notebook environment.

```bash
# Install Jupyter
pip install jupyter

# Install Databricks kernel
pip install databricks-connect

# Start Jupyter
jupyter notebook
```

```python
from databricks.connect import DatabricksSession

spark = DatabricksSession.builder.getOrCreate()
```

--------------------------------

### Pytest Command-Line Options for Test Execution

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/databricks-connect.md

Provides Bash commands for installing pytest, running all tests, running specific test files, executing tests with code coverage using pytest-cov, and enabling verbose output.

```bash
# Install pytest
pip install pytest

# Run all tests
pytest tests/

# Run specific test file
pytest tests/test_transformations.py

# Run with coverage
pip install pytest-cov
pytest --cov=src tests/

# Run with verbose output
pytest -v tests/

```

--------------------------------

### Setup Logging Configuration (Python)

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/databricks-connect.md

Configures basic logging for a Python application, directing output to both a file and the console. This setup allows for monitoring application events and errors. It uses Python's built-in `logging` module.

```python
import logging
from datetime import datetime

def setup_logging(log_file="app.log"):
    """Configure logging"""
    logging.basicConfig(
        level=logging.INFO,
        format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
        handlers=[
            logging.FileHandler(log_file),
            logging.StreamHandler()
        ]
    )

# Use in your application
setup_logging()
logger = logging.getLogger(__name__)

def process_data(spark, table_name):
    """Process data with logging"""
    logger.info(f"Starting data processing for {table_name}")
    start_time = datetime.now()

    try:
        df = spark.table(table_name)
        row_count = df.count()
        logger.info(f"Loaded {row_count} rows from {table_name}")

        # Process data...

        duration = (datetime.now() - start_time).total_seconds()
        logger.info(f"Processing completed in {duration:.2f} seconds")

    except Exception as e:
        logger.error(f"Error processing {table_name}: {str(e)}", exc_info=True)
        raise
```

--------------------------------

### Verify Dependency Installation

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/tests/validation/SETUP-CHECKLIST.md

This command lists installed Python packages and filters the output to show only 'anthropic' and 'openai'. It's used to verify that the required libraries have been successfully installed after a reinstallation or to check their versions.

```bash
pip list | grep -E "anthropic|openai"
```

--------------------------------

### Install Dependencies

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/tests/validation/SETUP-CHECKLIST.md

Installs necessary Python packages for the validation system from a requirements file. It then verifies the installation of either the Anthropic or OpenAI library.

```bash
cd tests/validation
pip install -r requirements.txt

python -c "import anthropic; print('✓ Anthropic installed')"
# OR
python -c "import openai; print('✓ OpenAI installed')"
```

--------------------------------

### AI Agent Tool Definition Examples (JSON)

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/ai-playground-tutorial.md

Provides JSON definitions for two AI agent tools: 'list_tables' to get tables in a schema and 'execute_query' to run SQL queries, including their parameters and descriptions.

```json
{
  "name": "list_tables",
  "description": "Lists all tables in a Unity Catalog schema",
  "parameters": {
    "schema_name": {
      "type": "string",
      "description": "Name of the schema (e.g., 'main.sales')"
    }
  }
}
```

```json
{
  "name": "execute_query",
  "description": "Executes a SQL query and returns results",
  "parameters": {
    "query": {
      "type": "string",
      "description": "SQL query to execute"
    },
    "limit": {
      "type": "integer",
      "description": "Maximum number of rows to return",
      "default": 100
    }
  }
}
```

--------------------------------

### Install Databricks SDK (Python)

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/setup.md

Provides commands to install the Databricks SDK for Python using pip. It includes options for installing with all optional dependencies or specific development dependencies.

```bash
pip install databricks-sdk

# With specific features:
# With all optional dependencies
pip install databricks-sdk[all]

# For development
pip install databricks-sdk[dev]
```

--------------------------------

### Tutorial Conversion Example

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/extraction-strategy.md

Example of converting tutorials into structured documentation, including step-by-step instructions and code examples.

```python
# Step 1 code
code_here()
```

```python
# Verification code
verify_here()
```

```python
# Full working example combining all steps
complete_code_here()
```

--------------------------------

### Install and Verify Package in Databricks Notebook

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/quickstart.md

Installs a Python package for the current notebook session using %pip and verifies the installation by printing the package version. This is useful for managing dependencies within a Databricks notebook environment.

```python
%pip install --quiet package-name

import package_name
print(package_name.__version__)
```

--------------------------------

### Few-Shot Learning Prompt Example

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/ai-playground-tutorial.md

Illustrates few-shot learning by providing examples of natural language to SQL query conversions, enabling the model to perform a similar conversion for a new input.

```text
Convert natural language to SQL queries:

Example 1:
Input: Show all customers from California
Output: SELECT * FROM customers WHERE state = 'CA';

Example 2:
Input: Count orders by product category
Output: SELECT category, COUNT(*) FROM orders GROUP BY category;

Now convert this:
Input: Find the top 5 customers by total purchase amount in the last year
Output:
```

--------------------------------

### Install Python Package using %pip

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/quickstart.md

This Databricks notebook magic command `%pip install package-name` is used to install a Python package directly onto the cluster. This is a common solution for resolving `ModuleNotFoundError` when importing libraries that are not pre-installed.

```python
# Install library on cluster
%pip install package-name
```

--------------------------------

### Agent Activation Process for New Users

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/AGENT-SYSTEM-DELIVERY.md

A step-by-step guide for new agents to activate and onboard. It covers repository familiarization, understanding current state, verifying sources, acknowledging core principles, setting up the validation system, and identifying next steps.

```plaintext
Phase 1: Repository Familiarization (15 min)
├─ Read PROJECT-STATUS.md
├─ Read docs/plan.md
├─ Review VALIDATION-SYSTEM-DELIVERY.md
├─ Read DATABRICKS-ACCURACY-AGENT.md
└─ Review AGENT-QUICK-REF.md

Phase 2: Understanding Current State (10 min)
├─ Review directory structure
├─ Identify completed documents (28 files)
├─ Identify remaining work (15%)
└─ Note quality metrics achieved

Phase 3: Official Sources Verification (5 min)
├─ Bookmark primary sources
├─ Bookmark secondary sources
└─ Test access to all sources

Phase 4: Core Principles Acknowledgment (5 min)
├─ Plan adherence
├─ Accuracy requirements
├─ Ask when unsure
├─ Never compromise quality
├─ Security first
├─ Test all code
├─ Document sources
└─ Maintain consistency

Phase 5: Validation System Setup (10 min)
├─ Locate validation scripts
├─ Understand validation modes
├─ Know validation commands
├─ Understand quality gates
└─ Know when to run validation

Phase 6: Next Steps Identification (5 min)
├─ Current phase from plan
├─ Next deliverable
├─ Required resources
├─ Success criteria
└─ Estimated effort

Final: Take Agent Oath
└─ Commit to 10-point oath
```

--------------------------------

### Databricks Cluster Policy Example

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/setup.md

An example JSON configuration for a Databricks cluster policy. Cluster policies are used to control and standardize cluster creation, helping manage costs and enforce organizational standards.

```json
{
  "spark_version": {
    "type": "fixed",
    "value": "13.3.x-scala2.12"
  },
  "node_type_id": {
    "type": "allowlist",
    "values": ["i3.xlarge", "i3.2xlarge"]
  },
  "autoscale": {
    "type": "fixed",
    "value": {
      "min_workers": 2,
      "max_workers": 8
    }
  },
  "autotermination_minutes": {
    "type": "fixed",
    "value": 30
  }
}
```

--------------------------------

### Setup and Run Validation Commands (Bash)

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/tests/VALIDATION-SYSTEM-SUMMARY.md

Provides essential bash commands for setting up the validation environment, running the validation process, and viewing the results. This includes installing dependencies, setting API keys, executing the main validation script, and displaying the latest report.

```bash
# Setup
cd tests/validation
pip install -r requirements.txt
export ANTHROPIC_API_KEY="your-key"

# Run validation
bash validate-now.sh

# View results
cat $(ls -t results/validation-report-*.md | head -1)

# Check accuracy
python -c "import json,glob; print(json.load(open(sorted(glob.glob('results/*.json'))[-1]))['total_accuracy'])"

# Full validation with API
python agent_validator.py --provider anthropic --scope full

# Generate manual request
python run_validation.py --scope full
```

--------------------------------

### Testing Strategy for Databricks Code Examples

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/NEXT-STEPS.md

This bash script outlines the steps for setting up a Databricks test environment and executing various code examples, including API calls, SQL queries, streaming, and ML workflows. It aims to validate the functionality of the documented code snippets.

```bash
# Create test environment
1. Provision test Databricks workspace
2. Set up authentication tokens
3. Create test databases and tables
4. Prepare sample datasets

# Execute tests
5. Run API examples using curl/Python
6. Execute SQL queries
7. Test streaming pipelines
8. Validate ML workflows
9. Document any failures
```

--------------------------------

### Databricks Workspace URLs

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/setup.md

Provides example URL formats for accessing Databricks workspaces across different cloud providers (AWS, Azure, GCP). These URLs are essential for connecting to your Databricks environment.

```text
AWS Databricks
URL Format: https://<workspace-id>.cloud.databricks.com
Example: https://dbc-abc12345-6789.cloud.databricks.com

Azure Databricks
URL Format: https://adb-<workspace-id>.<region>.azuredatabricks.net
Example: https://adb-1234567890123456.7.azuredatabricks.net

Google Cloud Platform Databricks
URL Format: https://<workspace-id>.gcp.databricks.com
Example: https://1234567890123456.7.gcp.databricks.com
```

--------------------------------

### Fix Module Import Errors for Databricks Connect (Python)

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/databricks-connect.md

Helps resolve 'No module named 'databricks.connect'' errors. It guides users to check their Python environment's executable path and install the `databricks-connect` package in the correct environment using pip. Includes a verification step.

```python
# Error: No module named 'databricks.connect'
# Solution: Install in correct Python environment

# Check Python path
python -c "import sys; print(sys.executable)"

# Install in correct environment
/path/to/python -m pip install databricks-connect

# Verify installation
python -c "from databricks.connect import DatabricksSession; print('OK')"
```

--------------------------------

### YAML Frontmatter Example (YAML)

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/skills/SKILL-QUICK-REF.md

An example of a complete frontmatter block for a SKILL.md file, demonstrating the correct formatting and inclusion of all required and recommended fields. This serves as a template for creating new skill documents.

```yaml
---
title: "AI & Agents Skills"
description: "Build intelligent applications with AI agents, vector search, and natural language interfaces"
category: "ai-agents"
tags:
  [
    "ai",
    "agents",
    "vector-search",
    "rag",
    "model-serving",
    "genie",
    "chatbots",
    "semantic-search",
    "llm",
  ]
priority: "high"
skills_count: 5
last_updated: 2026-02-27
version: "1.0.0"
status: "partial"
---

# AI & Agents Skills

Your content starts here...
```

--------------------------------

### Query and Visualize Data with SQL, Python, Scala, and R

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/GETTING-STARTED-UPDATE.md

This snippet demonstrates how to query data using Unity Catalog with SQL, Python, Scala, and R. It covers data exploration, analysis techniques, and the creation of built-in and custom visualizations using libraries like Matplotlib, Plotly, and Seaborn. It also touches upon dashboard creation and performance optimization.

```sql
-- SQL example for querying data
SELECT * FROM your_table LIMIT 10;
```

```python
# Python example for querying data
df = spark.sql("SELECT * FROM your_table LIMIT 10")
df.display()
```

```scala
// Scala example for querying data
val df = spark.sql("SELECT * FROM your_table LIMIT 10")
df.show()
```

```r
# R example for querying data
spark.sql("SELECT * FROM your_table LIMIT 10") %>% 
  spark_apply(function(x) head(x, 10)) %>% 
  print()
```

--------------------------------

### Resolve Databricks Version Mismatch (Bash)

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/databricks-connect.md

Addresses errors caused by version mismatches between Databricks Connect client and the Databricks cluster. It demonstrates how to check the cluster's Spark version and how to install a compatible version of `databricks-connect` using pip.

```bash
# Error: Version mismatch between client and cluster
# Solution: Install matching version

# Check cluster Spark version
databricks clusters get --cluster-id 1234-567890-abc123 | grep spark_version

# Install matching Databricks Connect version
pip install databricks-connect==13.3.*
```

--------------------------------

### Databricks Python SDK Example

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/CONTRIBUTING.md

Illustrates how to use the Databricks Python SDK to interact with Databricks workspaces. This example includes initializing a client, retrieving cluster information, and basic error handling.

```python
# Always include necessary imports
from databricks.sdk import WorkspaceClient
from databricks.sdk.service import compute

# Initialize client with clear comments
client = WorkspaceClient()

# Show error handling
try:
    # Example operation
    cluster = client.clusters.get(cluster_id="1234-567890-abc123")
    print(f"Cluster status: {cluster.state}")
except Exception as e:
    print(f"Error: {e}")
```

--------------------------------

### Databricks User-Defined Functions (UDFs)

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/databricks-connect.md

Illustrates how to define, register, and use User-Defined Functions (UDFs) with Databricks Connect. This example shows a Python UDF for categorizing transaction amounts, which can then be applied to a DataFrame or used within SQL queries. It requires the `databricks-connect` and `pyspark` libraries.

```python
# udfs.py
from databricks.connect import DatabricksSession
from pyspark.sql.functions import udf
from pyspark.sql.types import StringType, IntegerType

spark = DatabricksSession.builder.getOrCreate()

# Define UDF
@udf(returnType=StringType())
def categorize_amount(amount):
    """Categorize transaction amounts"""
    if amount < 100:
        return "Small"
    elif amount < 1000:
        return "Medium"
    elif amount < 10000:
        return "Large"
    else:
        return "Extra Large"

# Register UDF
spark.udf.register("categorize_amount", categorize_amount)

# Use UDF in DataFrame
df = spark.table("main.sales.transactions")
result = df.withColumn("category", categorize_amount(col("amount")))
result.show()

# Use UDF in SQL
spark.sql("""
    SELECT
        transaction_id,
        amount,
        categorize_amount(amount) as category
    FROM main.sales.transactions
""").show()
```

--------------------------------

### Databricks SDK Python Example (Python)

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/ZED-CONFIGURATION.md

A well-structured Python code example demonstrating how to use the Databricks SDK to list jobs. It includes necessary imports, secure initialization using environment variables, and basic error handling, adhering to the project's code example standards.

```python
# Import required libraries
from databricks.sdk import WorkspaceClient
import os

# Initialize with environment variables (SECURE)
w = WorkspaceClient(
    host=os.getenv('DATABRICKS_HOST'),
    token=os.getenv('DATABRICKS_TOKEN')
)

try:
    # Main operation
    result = w.jobs.list()
    for job in result:
        print(f"Job: {job.settings.name}")
except Exception as e:
    # Error handling included
    print(f"Error: {e}")
```

--------------------------------

### Common DBFS File System Operations

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/quickstart.md

This Python code provides examples of common Databricks File System (DBFS) operations using the `dbutils.fs` module. It covers listing files, copying, moving, removing files, reading file content, and writing content to files.

```python
# List files
dbutils.fs.ls("/path")

# Copy file
dbutils.fs.cp("/source/path", "/dest/path")

# Move file
dbutils.fs.mv("/source/path", "/dest/path")

# Remove file
dbutils.fs.rm("/path", recurse=True)

# Read file
content = dbutils.fs.head("/path/to/file.txt")

# Write file
dbutils.fs.put("/path/to/file.txt", "content", overwrite=True)
```

--------------------------------

### Example SKILL.md Frontmatter for Main Overview (YAML)

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/SKILL-FORMAT-IMPLEMENTATION.md

An example of the YAML frontmatter applied to the main 'Databricks Skills Reference' SKILL.md file. This demonstrates the structure and content for a comprehensive overview file.

```yaml
---
title: "Databricks Skills Reference"
description: "A comprehensive guide to skills and patterns for building on Databricks, organized by functional area and optimized for AI-assisted development with Context7"
category: "overview"
tags: ["skills", "patterns", "best-practices", "ai", "ml", "data-engineering", "analytics", "devops", "deployment", "automation", "dashboards", "workflows", "agents", "vector-search", "mlflow"]
skills_count: 24
last_updated: 2026-02-27
version: "1.0.0"
status: "active"
---

```

--------------------------------

### Install Project Dependencies

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/VALIDATION-SYSTEM-DELIVERY.md

Installs the necessary Python packages for the project from the requirements.txt file. This is a standard pip command and requires a Python environment with pip installed.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install Python Dependencies

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/tests/TESTING-GUIDE.md

Installs necessary Python packages for the validation system. Ensure Python 3.8+ is installed and navigate to the 'tests/validation' directory before running.

```bash
# Install Python 3.8+
python --version

# Navigate to validation directory
cd tests/validation

# Install dependencies
pip install -r requirements.txt
```

--------------------------------

### Configure Databricks SDK (Python)

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/setup.md

Demonstrates various ways to configure the Databricks SDK for Python, including using environment variables, direct parameter passing during client instantiation, and referencing profiles from the `.databrickscfg` file.

```python
import os
from databricks.sdk import WorkspaceClient

# Using environment variables:
# Set environment variables first
os.environ["DATABRICKS_HOST"] = "https://<workspace>.cloud.databricks.com"
os.environ["DATABRICKS_TOKEN"] = "dapi..."

# Create client (reads from environment)
client = WorkspaceClient()

# Using direct configuration:
client = WorkspaceClient(
    host="https://<workspace>.cloud.databricks.com",
    token="dapi..."
)

# Test connection
print(client.current_user.me())

# Using configuration file:
# Uses ~/.databrickscfg
client = WorkspaceClient(profile="dev")
```

--------------------------------

### Get Databricks Cluster Events using Databricks SDK

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/api/clusters.md

This example shows how to fetch recent cluster events using the Databricks SDK for Python. It filters events by type (STARTING, RUNNING, TERMINATING) within a 24-hour window. The SDK simplifies authentication and interaction with the Databricks API.

```python
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.compute import ClusterEventType
import time

client = WorkspaceClient()

# Get recent events
end_time = int(time.time() * 1000)
start_time = end_time - (24 * 60 * 60 * 1000)

events = client.clusters.events(
    cluster_id="1234-567890-abc123",
    start_time=start_time,
    end_time=end_time,
    event_types=[
        ClusterEventType.STARTING,
        ClusterEventType.RUNNING,
        ClusterEventType.TERMINATING
    ]
)

for event in events.events:
    print(f"Event: {event.type} at {event.timestamp}")
```

--------------------------------

### Run Example Tests Script

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/NEXT-STEPS.md

Bash commands to set up a directory for example tests and execute a Python script named 'run_example_tests.py' to validate Databricks examples.

```bash
# Create test script directory
mkdir -p tests/examples

# Copy examples and test
python tests/run_example_tests.py
```

--------------------------------

### Git Branching Example

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/CONTRIBUTING.md

Demonstrates how to create a new branch for feature development using Git. This is a standard practice for managing changes in a version-controlled repository.

```bash
git checkout -b feature/add-unity-catalog-examples
```

--------------------------------

### Read Data from Cloud Storage (AWS, Azure, GCS)

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/quickstart.md

This Python snippet provides examples for reading CSV data from different cloud storage providers: AWS S3, Azure Blob Storage, and Google Cloud Storage. Users need to replace placeholder bucket names, paths, and account details with their specific configurations.

```python
# Cell 17: Read from cloud storage
# AWS S3 example
# s3_df = spark.read.format("csv") \
#     .option("header", "true") \
#     .load("s3://bucket-name/path/to/file.csv")

# Azure Blob Storage example
# azure_df = spark.read.format("csv") \
#     .option("header", "true") \
#     .load("wasbs://container@account.blob.core.windows.net/path/file.csv")

# Google Cloud Storage example
# gcs_df = spark.read.format("csv") \
#     .option("header", "true") \
#     .load("gs://bucket-name/path/to/file.csv")

print("Update paths to read from your cloud storage")
```

--------------------------------

### Manage Databricks Authentication Token (Python/Bash)

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/databricks-connect.md

Provides guidance on how to handle Databricks authentication errors, typically related to invalid tokens. It shows how to check the token configuration in `.databrickscfg` or environment variables and advises on generating a new token via the Databricks UI.

```python
# Error: Invalid token
# Solution: Generate new token and update configuration

# Verify token in .databrickscfg
cat ~/.databrickscfg

# Or check environment variable
echo $DATABRICKS_TOKEN

# Generate new token from Databricks UI:
# User Settings → Access Tokens → Generate New Token
```

--------------------------------

### Databricks Python SDK: Create and Run Job

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/skills/development/SKILL.md

Demonstrates basic usage of the Databricks Python SDK to initialize a workspace client, list workspaces, create a new job with a notebook task and cluster configuration, run the job, and retrieve its status. Requires the 'databricks-sdk' package.

```python
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.jobs import Task, NotebookTask
from databricks.sdk.service.compute import ClusterSpec

# Initialize client (uses default auth from .databrickscfg)
w = WorkspaceClient()

# List workspaces
for workspace in w.workspaces.list():
    print(f"Workspace: {workspace.workspace_name}")

# Create a simple job
job = w.jobs.create(
    name="my-python-job",
    tasks=[
        Task(
            task_key="main_task",
            description="Process daily data",
            notebook_task=NotebookTask(
                notebook_path="/Users/user@company.com/my_notebook"
            ),
            new_cluster=ClusterSpec(
                spark_version="13.3.x-scala2.12",
                node_type_id="i3.xlarge",
                num_workers=2
            )
        )
    ]
)

print(f"Created job: {job.job_id}")

# Run the job
run = w.jobs.run_now(job_id=job.job_id)
print(f"Started run: {run.run_id}")

# Get run status
run_info = w.jobs.get_run(run_id=run.run_id)
print(f"Run state: {run_info.state.life_cycle_state}")
```

--------------------------------

### Get Correct Date Format (Bash)

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/skills/SKILL-FORMAT-GUIDE.md

Bash commands to get the current date in the required YYYY-MM-DD format and echo it, useful for updating the `last_updated` field.

```bash
# Get correct format
TODAY=$(date +%Y-%m-%d)
echo $TODAY

# Update file with correct date

```

--------------------------------

### Configure Databricks Connect Environment Variables

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/databricks-connect.md

This section provides examples for setting environment variables required for Databricks Connect. It includes configurations for Linux/macOS (bash), Windows PowerShell, and Windows CMD. These variables define the Databricks workspace host, authentication token, and the target cluster ID. Ensure you replace placeholder values with your actual credentials and cluster ID.

```bash
# Linux/macOS
export DATABRICKS_HOST="https://your-workspace.cloud.databricks.com"
export DATABRICKS_TOKEN="dapiXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
export DATABRICKS_CLUSTER_ID="1234-567890-abc123"

# Windows (PowerShell)
$env:DATABRICKS_HOST="https://your-workspace.cloud.databricks.com"
$env:DATABRICKS_TOKEN="dapiXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
$env:DATABRICKS_CLUSTER_ID="1234-567890-abc123"

# Windows (CMD)
set DATABRICKS_HOST=https://your-workspace.cloud.databricks.com
set DATABRICKS_TOKEN=dapiXXXXXXXXXXXXXXXXXXXXXXXXXXXX
set DATABRICKS_CLUSTER_ID=1234-567890-abc123
```

--------------------------------

### AI Agent Configuration Example (YAML)

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/getting-started/ai-playground-tutorial.md

Defines the configuration for an AI agent, including its name, description, system prompt, and available tools for interacting with Unity Catalog.

```yaml
Name: Data Analysis Assistant
Description: Helps users analyze data in Unity Catalog

System Prompt:
You are a data analysis assistant. You help users query and analyze data
stored in Unity Catalog tables. Always verify table existence before querying.

Available Tools:
- list_tables: Get list of available tables in a schema
- execute_query: Run SQL queries on Unity Catalog tables
- describe_table: Get schema information for a table
- generate_visualization: Create charts from query results
```

--------------------------------

### Install Dependencies (Bash)

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/tests/TESTING-GUIDE.md

This bash command installs the necessary Python dependencies listed in the requirements.txt file. It is a common step to ensure all required libraries, such as 'anthropic', are available for the project.

```bash
pip install -r requirements.txt
# Or specifically:
pip install anthropic
```

--------------------------------

### Create Database in Databricks SQL

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/docs/examples/sql.md

Demonstrates how to create a new database in Databricks SQL, including setting a comment and specifying a location. It also shows how to use the created database and list all available databases.

```sql
-- Create database
CREATE DATABASE IF NOT EXISTS my_database
COMMENT 'Database for analytics'
LOCATION '/mnt/data/my_database';

-- Use database
USE my_database;

-- Show databases
SHOW DATABASES;
```

--------------------------------

### Databricks Validation Configuration Example

Source: https://github.com/thegrowthexponent/c7-databricks/blob/main/tests/TESTING-GUIDE.md

An example JSON configuration file for the Databricks documentation validation system. It allows customization of validation rules, severity mapping, and quality gates.

```json
{
  "validation_rules": {
    "api_documentation": {
      "enabled": true,
      "severity_mapping": {
        "endpoint_mismatch": "critical",
        "required_parameter_missing": "critical"
      }
    }
  },
  "quality_gates": {
    "minimum_accuracy_score": 85,
    "maximum_critical_issues": 0,
    "maximum_high_issues": 5
  }
}
```