### Initialize Project Environment Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/README-DEV.md Clones the repository and executes the setup script to install dependencies and configure the environment. ```bash git clone https://github.com/databricks-solutions/judge-builder.git cd judge-builder ./setup.sh ``` -------------------------------- ### Install test dependencies Source: https://github.com/databrickslabs/sandbox/blob/main/feature-registry-app/tests/README.md Install the necessary packages required to run the test suite. ```bash pip install -r ../test-requirements.txt ``` -------------------------------- ### Clone the Repository Source: https://github.com/databrickslabs/sandbox/blob/main/uc-quickstart/README.md Initial step to download the project files and navigate to the quickstart directory. ```bash git clone https://github.com/databrickslabs/sandbox.git cd sandbox/uc-quickstart/ ``` -------------------------------- ### Install SQL Migration Assistant via Databricks CLI Source: https://github.com/databrickslabs/sandbox/blob/main/sql_migration_assistant/README.md Run this command to install the Databricks Labs Sandbox and the SQL Migration Assistant after configuring the Databricks CLI. ```bash databricks labs install sandbox && databricks labs sandbox sql-migration-assistant ``` -------------------------------- ### Install Dependencies Source: https://github.com/databrickslabs/sandbox/blob/main/dbsql/dbr_eos/dbr-eos-monitor-dabs/resources/jobs/DBR_Lookup_Table.ipynb Installs the beautifulsoup4 library and restarts the Python kernel to apply changes. ```python %pip install beautifulsoup4 ``` ```python dbutils.library.restartPython() ``` -------------------------------- ### Install Required Python Libraries Source: https://github.com/databrickslabs/sandbox/blob/main/uc-catalog-cloning/README.md Install the necessary dependencies for the cloning script on Databricks or a local development environment. ```bash databricks-sdk==0.14.0 termcolor ``` ```bash databricks-connect==13.3.3 ``` -------------------------------- ### Install Databricks Labs Sandbox Source: https://github.com/databrickslabs/sandbox/blob/main/README.md Executes the installation of the sandbox project via the Databricks CLI. Requires CLI version 0.210.1 or higher. ```bash databricks labs install sandbox ``` -------------------------------- ### Define notebook widgets Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/scan_dbfs_nb.ipynb Sets up input widgets for the starting directory and the output directory path. ```python dbutils.widgets.text("start_dir", "/", "Directory to scan") dbutils.widgets.text("output_dir", "", "Output directory (UC Volume, etc.)") ``` -------------------------------- ### Run Development Mode Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/README-DEV.md Starts the FastAPI backend and React frontend concurrently for local development. ```bash ./dev/watch.sh ``` -------------------------------- ### Install TACO Toolkit Source: https://github.com/databrickslabs/sandbox/blob/main/tableau-delta-sharing-connector-oauth/README.md Install the required Tableau TACO toolkit globally via npm. ```bash npm install -g @tableau/taco-toolkit@2.0.0 ``` -------------------------------- ### Installation via pyproject.toml Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/docs/databricks_apis/databricks_sdk.md Add the Databricks SDK dependency to your project configuration. ```toml # pyproject.toml databricks-sdk==0.59.0 ``` -------------------------------- ### Copy Terraform configuration template Source: https://github.com/databrickslabs/sandbox/blob/main/uc-quickstart/aws/README.md Initialize the configuration by copying the example template to the required terraform.tfvars file. ```bash cp template.tfvars.example terraform.tfvars ``` -------------------------------- ### Authenticate with Azure CLI Source: https://github.com/databrickslabs/sandbox/blob/main/uc-quickstart/azure/README.md Use these commands to install the Azure CLI and authenticate your session. ```bash # Install Azure CLI brew install azure-cli # Login and set subscription az login az account set --subscription "your-subscription-id" ``` -------------------------------- ### Initialize Application and Fetch Data Source: https://github.com/databrickslabs/sandbox/blob/main/agent_genie/templates/index.html Sets up the initial chat interface, triggers schema retrieval, and starts periodic statistics updates. ```javascript function initializeApp() { // Show welcome message const welcomeMessage = document.createElement('div'); welcomeMessage.className = 'chat-message bot-message'; welcomeMessage.textContent = 'Welcome to Databricks Genie++ Omni-Analytics Platform! The system is configured and ready to use.'; chatBox.appendChild(welcomeMessage); // Fetch schema information fetchSchema(); // Fetch statistics fetchStatistics(); // Refresh statistics every 5 minutes setInterval(fetchStatistics, 300000); // Focus input field userInput.focus(); } ``` -------------------------------- ### Go Code Coverage Profile Source: https://github.com/databrickslabs/sandbox/blob/main/acceptance/README.md Example content of a go-coverprofile file. ```text mode: set github.com/databrickslabs/sandbox/go-libs/env/context.go:13.53,15.22 2 0 github.com/databrickslabs/sandbox/go-libs/env/context.go:15.22,17.3 1 0 github.com/databrickslabs/sandbox/go-libs/env/context.go:18.2,18.12 1 0 github.com/databrickslabs/sandbox/go-libs/env/context.go:21.52,22.16 1 0 github.com/databrickslabs/sandbox/go-libs/env/context.go:22.16,24.3 1 0 github.com/databrickslabs/sandbox/go-libs/env/context.go:25.2,26.9 2 0 github.com/databrickslabs/sandbox/go-libs/env/context.go:26.9,28.3 1 0 ``` -------------------------------- ### Go Test JSON Output Source: https://github.com/databrickslabs/sandbox/blob/main/acceptance/README.md Example of standard output from the go test -json command. ```json {"Time":"2024-02-03T15:28:06.204157427Z","Action":"start","Package":"github.com/databrickslabs/sandbox/go-libs"} {"Time":"2024-02-03T15:28:06.204217579Z","Action":"output","Package":"github.com/databrickslabs/sandbox/go-libs","Output":"? \tgithub.com/databrickslabs/sandbox/go-libs\t[no test files]\n"} ... ``` -------------------------------- ### Manage Clusters Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/docs/databricks_apis/databricks_sdk.md List, retrieve details, start, and create Databricks clusters. ```python # List clusters clusters = client.clusters.list() # Get cluster details cluster = client.clusters.get('cluster-id') # Start a cluster client.clusters.start('cluster-id') # Create a cluster cluster_spec = { 'cluster_name': 'my-cluster', 'spark_version': '13.3.x-scala2.12', 'node_type_id': 'i3.xlarge', 'num_workers': 2 } cluster_id = client.clusters.create(**cluster_spec).cluster_id ``` -------------------------------- ### Configure GitHub Actions for Artifact Upload Source: https://github.com/databrickslabs/sandbox/blob/main/acceptance/README.md Example workflow configuration for uploading artifacts to GitHub Actions during the acceptance testing process. ```yaml name: acceptance on: pull_request: types: [opened, synchronize] paths: ['go-libs/**'] permissions: id-token: write contents: read pull-requests: write jobs: acceptance: if: github.event_name == 'pull_request' runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v4 - name: Setup Go uses: actions/setup-go@v5 with: go-version: 1.21 - name: Acceptance uses: databrickslabs/sandbox/acceptance@actions/artifact with: directory: go-libs env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} ``` -------------------------------- ### Query Traditional ML Model Serving Endpoints Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/docs/databricks_apis/model_serving.md Demonstrates how to query traditional ML models using the inputs parameter, including examples for sklearn, chat, and dataframe-based inputs. ```python # Query a traditional ML model serving endpoint def query_ml_endpoint(endpoint_name: str, input_data: dict): try: # For traditional ML models, use inputs parameter response = client.serving_endpoints.query( name=endpoint_name, inputs=input_data ) return response except DatabricksError as e: print(f"Error querying endpoint: {e}") return None # Example usage for different model types # For a simple sklearn model sklearn_input = { "inputs": [[1.0, 2.0, 3.0, 4.0]] } # For a chat model chat_input = { "messages": [ {"role": "user", "content": "What is Databricks?"} ], "max_tokens": 100 } # For a custom model with dataframe input df_input = { "dataframe_split": { "columns": ["feature1", "feature2", "feature3"], "data": [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]] } } # Query the endpoint result = query_endpoint("my-model-endpoint", sklearn_input) print(f"Result: {result}") ``` -------------------------------- ### Example Analyzer Output Source: https://github.com/databrickslabs/sandbox/blob/main/ip_access_list_analyzer/README.md Sample output showing the results of an analysis, including identified duplicates and proposed modifications. ```text INFO:root:There are duplicates in the IP Access lists! len(all_ips)=241, len(uniq_ips)=237 INFO:root:Going to remove list 'list1' (0f209622-ca20-455a-bdc4-4de3bed8a1ed) as it's empty INFO:root:Going to modify list 'list1 dup' (1f209622-ca20-455a-bdc4-4de3bed8a1ed). Entries to remove: ['52.55.144.63'] INFO:root:Going to modify list 'list2' (1f209623-ca20-455a-bdc4-4de3bed8a1ed). Entries to remove: ['10.1.2.0/24', '192.168.10.11', '52.55.144.63', '10.0.1.0'] INFO:root:List 'github_actions' (d798c5f5-3b53-4dc7-85b7-75dd67056512) isn't modified or not enabled INFO:root:List 'Disabled list' (fc594781-60cb-4b46-b0f7-ee9d951e3c3f) isn't modified or not enabled ``` -------------------------------- ### Configure Resource Mapping Flags Source: https://github.com/databrickslabs/sandbox/blob/main/supervisor-agent-export-import/README.md Examples of mapping flags used to rename connections, apps, and endpoints during the import process. ```bash --connection-map "old_mcp=new_mcp,sandbox_mcp=prod_mcp" --app-map "old_app_name=new_app_name" --endpoint-map "old_endpoint=new_endpoint" ``` -------------------------------- ### Example compute_scan_results.json structure Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/README.md Represents the configuration hierarchy for clusters, policies, jobs, and DLT pipelines. ```json { "clusters": { "0313-121649-cxqqcvz0": { "cluster_log_conf": "dbfs:/mnt/cluster-logs", "cluster_name": "EvHub" "libraries": [ "dbfs:/FileStore/whl/test.whl", "dbfs:/FileStore/jars/test.jar" ], } }, "cluster_policies": { "30640179C40010EF": { "cluster_log_conf": "dbfs:/mnt/cluster-logs", "policy_name": "minimum_global" }, "0005268696213EA3": { "libraries": [ "dbfs:/FileStore/jars/jar-tests-0.0.1.jar" ], "policy_name": "Test Libraries" } }, "jobs": { "75390238197996": { "tasks": { "python_script_on_dbfs": { "python_file_on_dbfs": "dbfs:/1.py" }, "abc": { "libraries": [ "dbfs:/1.jar", "dbfs:/1.whl" ] } } } }, "dlt": { "713294fd-9e53-4f2b-b8fe-1d924fb4905c": { "libraries": { "/Users/user@domain.com/Test/test": { "dbfs_file_refs": [ "dbfs:/FileStore/temp/test_data/input" ] } }, "storage": "dbfs:/pipelines/713294fd-9e53-4f2b-b8fe-1d924fb4905c", "name": "dlt_dabs_pipeline" } } } ``` -------------------------------- ### Get Serving Endpoint Details Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/docs/databricks_apis/model_serving.md Fetches detailed configuration, traffic routing, and model metadata for a specific endpoint. ```python # Get detailed information about a specific endpoint def get_endpoint_details(endpoint_name: str): try: endpoint = client.serving_endpoints.get(endpoint_name) print(f"Endpoint: {endpoint.name}") print(f"State: {endpoint.state}") print(f"URL: {endpoint.serving_endpoint_url}") # Get endpoint configuration if endpoint.config: print(f"Traffic Config:") for route in endpoint.config.traffic_config.routes: print(f" Route: {route.served_model_name} -> {route.traffic_percentage}%") print(f"Served Models:") for model in endpoint.config.served_models: print(f" Model: {model.name}") print(f" Version: {model.model_version}") print(f" Workload Size: {model.workload_size}") print(f" Scale to Zero: {model.scale_to_zero_enabled}") if model.model_name: print(f" Model Name: {model.model_name}") # Get endpoint tags if endpoint.tags: print(f"Tags: {dict(endpoint.tags)}") return endpoint except DatabricksError as e: print(f"Error getting endpoint details: {e}") return None # Usage endpoint_details = get_endpoint_details("my-model-endpoint") ``` -------------------------------- ### Optimize Prompts Automatically Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/docs/databricks_apis/mlflow_genai.md Leverage automated optimization to refine prompt templates based on provided examples and a target model. ```python # Optimize prompts automatically optimized_prompt = mlflow.genai.optimize_prompt( prompt="Answer the question: {question}", examples=[ {"question": "What is Databricks?", "expected_answer": "Databricks is..."}, {"question": "How to create clusters?", "expected_answer": "To create..."} ], model="gpt-4" ) print(f"Optimized prompt: {optimized_prompt.template}") ``` -------------------------------- ### Run DBFS Content Scanner via CLI Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/README.md Executes the DBFS scanner to classify content and estimate size, optionally starting from a specific directory. ```bash python scan_dbfs.py [start_directory] ``` -------------------------------- ### Initialize and run tests in Go Source: https://github.com/databrickslabs/sandbox/blob/main/acceptance/README.md Use this pattern to initialize the test environment with Azure CLI credentials and execute all tests within the current working directory. ```go testEnv := testenv.NewWithAzureCLI(vaultURI) loaded, err := testEnv.Load(ctx) if err != nil { return fmt.Errorf("load: %w", err) } ctx, stop, err := loaded.Start(ctx) if err != nil { return fmt.Errorf("start: %w", err) } defer stop() cwd, err := os.Getwd() if err != nil { return fmt.Errorf("cwd: %w", err) } // detect and run all tests report, err := ecosystem.RunAll(ctx, loaded.Redaction(), cwd) if err != nil { return fmt.Errorf("test: %w", err) } return fmt.Errorf("failed: %s", report.String()) ``` -------------------------------- ### Initialize Database Objects Source: https://github.com/databrickslabs/sandbox/blob/main/dbsql/dbr_eos/dbr-eos-monitor-dabs/resources/jobs/DBR_Lookup_Table.ipynb Creates the catalog and schema if they do not already exist. ```python # Check if the catalog already exists. More elaborate way of checking to prevent error if max catalogs have already been created existing_catalogs = spark.sql(f"SHOW CATALOGS LIKE '{wn_catalog}'") if existing_catalogs.count() == 0: create_catalog_sql = f"CREATE CATALOG IF NOT EXISTS {wn_catalog}" spark.sql(create_catalog_sql) ``` ```python create_schema_sql = f"CREATE SCHEMA IF NOT EXISTS {wn_catalog}.{wn_schema}" spark.sql(create_schema_sql) ``` ```python full_table_name = f"{wn_catalog}.{wn_schema}.{wn_table}" print(full_table_name) ``` -------------------------------- ### GET /serving-endpoints Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/docs/databricks_apis/model_serving.md Lists all available serving endpoints. ```APIDOC ## GET /serving-endpoints ### Description List all available serving endpoints. ### Method GET ### Endpoint /serving-endpoints ### Response #### Success Response (200) - **name** (string) - The name of the endpoint - **state** (string) - The current state of the endpoint - **url** (string) - The serving endpoint URL - **creation_time** (string) - The timestamp when the endpoint was created - **creator** (string) - The user who created the endpoint ``` -------------------------------- ### Build and Run Connector Commands Source: https://github.com/databrickslabs/sandbox/blob/main/tableau-delta-sharing-connector-oauth/README.md Commands to compile, package, and execute the connector for local testing. ```bash taco build ``` ```bash taco pack ``` ```bash taco run Desktop ``` -------------------------------- ### Initialize WorkspaceClient with Custom Configuration Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/docs/databricks_apis/databricks_sdk.md Instantiate a WorkspaceClient using a custom Config object to specify host, token, and timeout settings. ```python from databricks.sdk.config import Config # Custom configuration config = Config( host='https://your-workspace.cloud.databricks.com', token='your-token', retry_timeout_seconds=300 ) client = WorkspaceClient(config=config) ``` -------------------------------- ### GET /serving-endpoints/{endpoint_name} Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/docs/databricks_apis/model_serving.md Retrieves detailed information about a specific serving endpoint. ```APIDOC ## GET /serving-endpoints/{endpoint_name} ### Description Get detailed information about a serving endpoint. ### Method GET ### Endpoint /serving-endpoints/{endpoint_name} ### Parameters #### Path Parameters - **endpoint_name** (string) - Required - The name of the serving endpoint ### Response #### Success Response (200) - **name** (string) - The name of the endpoint - **state** (string) - The current state of the endpoint - **url** (string) - The serving endpoint URL - **config** (object) - Configuration details including served models - **tags** (object) - Tags associated with the endpoint ``` -------------------------------- ### Get File Information Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/docs/databricks_apis/workspace_apis.md Fetches metadata for a specific file or directory path. ```python # Get detailed file information try: file_info = client.workspace.get_status('/path/to/file') print(f"Path: {file_info.path}") print(f"Object Type: {file_info.object_type}") print(f"Language: {file_info.language}") print(f"Size: {file_info.size}") print(f"Modified: {file_info.modified_at}") except DatabricksError as e: print(f"Error getting file info: {e}") ``` -------------------------------- ### Initialize WorkspaceClient Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/docs/databricks_apis/databricks_sdk.md Create a client instance to interact with the Databricks workspace. ```python from databricks.sdk import WorkspaceClient from databricks.sdk.errors import DatabricksError # Initialize client (automatically uses your configured authentication) client = WorkspaceClient() ``` -------------------------------- ### Integration Test Execution Report Source: https://github.com/databrickslabs/sandbox/blob/main/acceptance/README.md Example of a normalized test report in test-report.json. ```json {"ts":"2024-02-03T15:28:16.940198637Z","project":"go-libs","package":"sqlexec","name":"TestAccErrorMapping","flaky":false,"pass":false,"skip":true,"output":"=== RUN TestAccErrorMapping\n init.go:96: Environment variable CLOUD_ENV is missing\n--- SKIP: TestAccErrorMapping (0.00s)\n","elapsed":0} {"ts":"2024-02-03T15:28:16.940348556Z","project":"go-libs","package":"sqlexec","name":"TestAccMultiChunk","flaky":true,"pass":true,"skip":false,"output":"...","elapsed":0} ``` -------------------------------- ### Model Serving Management Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/docs/databricks_apis/databricks_sdk.md Shows how to list existing serving endpoints and create a new serving endpoint with a specific model configuration. ```python # List serving endpoints endpoints = client.serving_endpoints.list() # Create serving endpoint endpoint_spec = { 'name': 'my-endpoint', 'config': { 'served_models': [{ 'model_name': 'my_model', 'model_version': '1', 'workload_size': 'Small' }] } } client.serving_endpoints.create(**endpoint_spec) ``` -------------------------------- ### Initialize Databricks API Connection Source: https://github.com/databrickslabs/sandbox/blob/main/uc-quickstart/utils/abac/CreateTags.ipynb Sets up the workspace URL and authentication headers using the current notebook's context. ```python import requests import json # Workspace settings workspace_url = "https://adb-3133034344545702.2.azuredatabricks.net" token = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiToken().get() headers = { "Authorization": f"Bearer {token}", "Content-Type": "application/json" } # Tag policy endpoint base_url = f"{workspace_url}/api/2.0/tag-policies" ``` -------------------------------- ### GitHub Actions Event Artifact Source: https://github.com/databrickslabs/sandbox/blob/main/acceptance/README.md Example of the raw event information captured in event.json. ```json { "action": "synchronize", "after": "faa8a5ba40d987cb981362581e556eb676761161", "before": "3d0b4fe5bdc45292358a29dde28718e1605efcad", "enterprise": { ... }, "number": 69, "organization": { ... }, "pull_request": { "_links": { ``` -------------------------------- ### Configure Widgets Source: https://github.com/databrickslabs/sandbox/blob/main/dbsql/dbr_eos/dbr-eos-monitor-dabs/resources/jobs/DBR_Lookup_Table.ipynb Sets up and retrieves input widgets for catalog, schema, and table names. ```python dbutils.widgets.text("wn_catalog", "dbdemos") dbutils.widgets.text("wn_schema", "lookup") dbutils.widgets.text("wn_table", "dbr_version") ``` ```python # Get widget values wn_catalog = dbutils.widgets.get("wn_catalog") wn_schema = dbutils.widgets.get("wn_schema") wn_table = dbutils.widgets.get("wn_table") ``` -------------------------------- ### Navigate to Cloud Provider Directory Source: https://github.com/databrickslabs/sandbox/blob/main/uc-quickstart/README.md Commands to switch into the specific directory for AWS or Azure deployment configurations. ```bash cd aws/ ``` ```bash cd azure/ ``` -------------------------------- ### Upgrade Databricks Labs Sandbox Source: https://github.com/databrickslabs/sandbox/blob/main/README.md Updates the existing sandbox installation to the latest available version. ```bash databricks labs upgrade sandbox ``` -------------------------------- ### Deploy Application Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/README-DEV.md Builds the frontend, syncs code to the Databricks workspace, and deploys the Databricks App. ```bash ./deploy.sh ``` -------------------------------- ### POST /serving-endpoints/{endpoint_name}/predict Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/docs/databricks_apis/model_serving.md Queries a serving endpoint with input data to get predictions. ```APIDOC ## POST /serving-endpoints/{endpoint_name}/predict ### Description Query a serving endpoint with input data. ### Method POST ### Endpoint /serving-endpoints/{endpoint_name}/predict ### Parameters #### Path Parameters - **endpoint_name** (string) - Required - The name of the serving endpoint #### Request Body - **inputs** (List[List[float]]) - Required - The input data for the model ### Request Example { "inputs": [[0.1, 0.2, 0.3]] } ### Response #### Success Response (200) - **predictions** (List[float]) - The model predictions - **endpoint_name** (string) - The name of the endpoint used ``` -------------------------------- ### Go Test Error Output Source: https://github.com/databrickslabs/sandbox/blob/main/acceptance/README.md Example of standard error output from the go test command. ```text go: downloading github.com/spf13/cobra v1.8.0 go: downloading github.com/fatih/color v1.16.0 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/spf13/viper v1.18.2 go: downloading github.com/stretchr/testify v1.8.4 ``` -------------------------------- ### Configure VSCode debugging for Go Source: https://github.com/databrickslabs/sandbox/blob/main/CONTRIBUTING.md Create a .vscode/launch.json file to enable debugging for Go-based tools within the repository. ```json { "version": "0.2.0", "configurations": [ { "name": "metascan clone-all", "type": "go", "request": "launch", "mode": "debug", "program": "${workspaceFolder}/metascan/main.go", "args": [ "clone-all", "--debug" ] } ] } ``` -------------------------------- ### Initialize MLflow tracking Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/docs/databricks_apis/mlflow_genai.md Configure the tracking URI to point to Databricks. ```python import mlflow import mlflow.genai # Set tracking URI (automatic in Databricks) mlflow.set_tracking_uri("databricks") ``` -------------------------------- ### Install MLflow GenAI dependencies Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/docs/databricks_apis/mlflow_genai.md Include the required MLflow package with Databricks support in your project configuration. ```toml # pyproject.toml mlflow[databricks]>=3.1.1 ``` -------------------------------- ### Import necessary libraries Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/scan_dbfs_nb.ipynb Initializes the required modules for the Databricks SDK, file system analysis, and data handling. ```python from databricks.sdk import WorkspaceClient from helpers.dbfs_analyzer import scan_dbfs import sys import os import pprint import json ``` -------------------------------- ### Run IP Access List Analyzer Source: https://github.com/databrickslabs/sandbox/blob/main/ip_access_list_analyzer/README.md Execute the tool using the command-line interface. Use the --help flag for more options. ```sh databricks labs sandbox ip-access-list-analyzer [options] ``` -------------------------------- ### Initialize Databricks WorkspaceClient Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/docs/databricks_apis/README.md Instantiate the client to automatically utilize configured authentication settings. ```python from databricks.sdk import WorkspaceClient # Automatically uses your configured authentication client = WorkspaceClient() ``` -------------------------------- ### Execute Terraform deployment Source: https://github.com/databrickslabs/sandbox/blob/main/uc-quickstart/aws/README.md Run these commands in sequence to initialize, validate, plan, and apply the infrastructure changes. ```bash terraform init terraform validate terraform plan terraform apply ``` -------------------------------- ### Initialize workspace test in Go Source: https://github.com/databrickslabs/sandbox/blob/main/downstreams/README.md Use the fixtures package to initialize a workspace test, which enables logging to the go-slog.json artifact. ```go import ( "testing" "github.com/databrickslabs/sandbox/go-libs/fixtures" ) func TestShowDatabases(t *testing.T) { ctx, w := fixtures.WorkspaceTest(t) // ... } ``` -------------------------------- ### Create a Databricks User Source: https://github.com/databrickslabs/sandbox/blob/main/uc-quickstart/utils/abac/Create Users and Roles.ipynb Initializes the WorkspaceClient and sends a POST request to the SCIM Users endpoint to provision a new user. ```python #Create User from databricks.sdk import WorkspaceClient import json w = WorkspaceClient() # Step 1: Create a user user_email = "L19693912@sandpit0x8a.onmicrosoft.com" create_user_resp = w.api_client.do( "POST", "/api/2.0/preview/scim/v2/Users", data=json.dumps({ "userName": user_email, "emails": [{"value": user_email, "primary": True}], "displayName": "User 101" }) ) user_id = create_user_resp["id"] print(f"✅ Created user: {user_email} (id={user_id})") ``` -------------------------------- ### Example dbfs_scan_results.json structure Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/README.md Maps specific DBFS paths to their total size and identifies file types or specific file lists. ```json { "/FileStore/init-scripts": { "size": 2406 }, "/cluster-logs/1130-094127-3ij40m7s": { "type": "cluster_logs", "size": 684830 }, "/pipelines/28ae6515-b6f4-4e2e-8b65-879009332b7d": { "type": "dlt_storage", "size": 589529922 }, "/tmp": { "whl_files": [ "2.whl", "4.whl" ], "jar_files": [ "dbutils-in-jar-0.0.1-jar-with-dependencies.jar", "kafka-eventhubs-aad-auth-0.0.1-dbr_10.4.jar" ], "size": 3560732064 } } ``` -------------------------------- ### Configure deployment variables Source: https://github.com/databrickslabs/sandbox/blob/main/feature-registry-app/README.md Define the destination workspace path and application name in the deployment configuration file. ```sh # Path to a destination folder in default Databricks workspace where source code will be sync'ed export DEST=/Workspace/Users/Path/To/App/Code # Name of the App to deploy export APP_NAME=your-app-name ``` -------------------------------- ### Retrieve Policy Definition via API Source: https://github.com/databrickslabs/sandbox/blob/main/uc-quickstart/utils/abac/GetPolicyDefinition.ipynb Uses the WorkspaceClient to execute a GET request against the Unity Catalog policies endpoint. Ensure the securable type, full name, and policy name are correctly specified in the path. ```python from databricks.sdk import WorkspaceClient import json w = WorkspaceClient() # securable_type = "CATALOG" # securable_full_name = "t3_a00cd1_snd_arg02_dbws01" # policy_name = "3INTERNALUSE" policies = w.api_client.do("GET", "/api/2.1/unity-catalog/policies/CATALOG/t3_a00cd1_snd_arg02_dbws01/13PII") policies ``` -------------------------------- ### Import CloneCatalog Class Source: https://github.com/databrickslabs/sandbox/blob/main/uc-catalog-cloning/README.md Import the main class from the clonecatalog module to begin the cloning process. ```python from clonecatalog import CloneCatalog ``` -------------------------------- ### Execute DBFS scan Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/scan_dbfs_nb.ipynb Initializes the WorkspaceClient and performs the scan on the specified path. ```python wc = WorkspaceClient() results = {} print("Starting scanning DBFS at path:", path) scan_dbfs(wc, results, path) ``` -------------------------------- ### Configure Databricks via Environment Variables Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/docs/databricks_apis/databricks_sdk.md Set workspace credentials and profile names using environment variables or a .env.local file. ```bash # Set in your .env.local file DATABRICKS_HOST=https://your-workspace.cloud.databricks.com DATABRICKS_TOKEN=your-personal-access-token # Or use profile DATABRICKS_CONFIG_PROFILE=your-profile-name ``` -------------------------------- ### Create FastAPI Endpoints with Databricks SDK Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/docs/databricks_apis/databricks_sdk.md Demonstrates how to wrap Databricks SDK calls within FastAPI routes to manage workspace files, clusters, and SQL statement execution. ```python from fastapi import APIRouter, HTTPException from databricks.sdk import WorkspaceClient from databricks.sdk.errors import DatabricksError router = APIRouter() client = WorkspaceClient() @router.get("/workspace/files") async def list_workspace_files(path: str = "/"): try: files = client.workspace.list(path) return [{"name": f.path, "type": f.object_type.value} for f in files] except DatabricksError as e: raise HTTPException(status_code=400, detail=str(e)) @router.post("/clusters/{cluster_id}/start") async def start_cluster(cluster_id: str): try: client.clusters.start(cluster_id) return {"message": "Cluster start initiated"} except DatabricksError as e: raise HTTPException(status_code=400, detail=str(e)) @router.post("/sql/execute") async def execute_sql(warehouse_id: str, query: str): try: response = client.statement_execution.execute_statement( warehouse_id=warehouse_id, statement=query ) return {"statement_id": response.statement_id, "status": response.status.state} except DatabricksError as e: raise HTTPException(status_code=400, detail=str(e)) ``` -------------------------------- ### Register and Load Prompts with MLflow Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/docs/databricks_apis/mlflow_genai.md Use the prompt registry to store, version, and retrieve prompt templates for centralized management. ```python # Register prompts for centralized management mlflow.genai.register_prompt( name="customer_support_prompt", template="You are a helpful customer support agent. Question: {question}", commit_message="Initial customer support prompt", tags={"version": "1.0", "team": "support"} ) # Load prompts prompt = mlflow.genai.load_prompt( name="customer_support_prompt", version=1 ) # Use prompt with variables formatted_prompt = prompt.format(question="How do I create a cluster?") ``` -------------------------------- ### Manage Human Labeling Sessions Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/docs/databricks_apis/mlflow_genai.md Create and retrieve labeling sessions for expert review, and access the review application URL. ```python # Create labeling session for expert review labeling_session = mlflow.genai.create_labeling_session( name="customer_support_evaluation", assigned_users=["expert1@company.com", "expert2@company.com"], agent="customer_support_agent", description="Evaluate customer support responses for quality" ) # Get labeling sessions sessions = mlflow.genai.get_labeling_sessions() for session in sessions: print(f"Session: {session.name}, Status: {session.status}") # Get review app for UI-based labeling review_app = mlflow.genai.get_review_app(experiment_id="your-experiment-id") print(f"Review app URL: {review_app.url}") ``` -------------------------------- ### Import dependencies Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/scan_compute_nb.ipynb Imports necessary SDK clients, analysis helpers, and standard libraries. ```python from databricks.sdk import WorkspaceClient from helpers.compute_analyzer import analyze_clusters, analyze_cluster_policies, analyze_jobs, analyze_dlt_pipelines import sys import os import pprint import json ``` -------------------------------- ### Initialize WorkspaceClient Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/scan_compute_nb.ipynb Initializes the Databricks WorkspaceClient and a dictionary to store scan results. ```python wc = WorkspaceClient() full_results: dict = {} ``` -------------------------------- ### Handle Non-Deterministic Masking Functions Source: https://github.com/databrickslabs/sandbox/blob/main/uc-quickstart/utils/abac/healthcare/ABAC_Performance_Anti_Patterns.md Demonstrates the use of non-deterministic functions and a recommended deterministic alternative using time-based seeds. ```sql -- BE VERY CAREFUL - Non-deterministic mask function CREATE OR REPLACE FUNCTION mask_with_random() RETURNS STRING NOT DETERMINISTIC -- Customer may explicitly want this! RETURN CASE WHEN is_member('full_access_group') THEN input_value ELSE CONCAT('MASKED_', CAST(RAND() * 1000000 AS INT)) END; ``` ```sql -- BETTER APPROACH - Deterministic with time-based seed CREATE OR REPLACE FUNCTION mask_with_time_seed(input_value STRING) RETURNS STRING DETERMINISTIC RETURN CASE WHEN is_member('full_access_group') THEN input_value ELSE CONCAT('MASKED_', SHA2(CONCAT(input_value, DATE_FORMAT(current_date(), 'yyyy-MM-dd')), 256)) END; ``` -------------------------------- ### Define Toolchain Configuration in .codegen.json Source: https://github.com/databrickslabs/sandbox/blob/main/acceptance/README.md Specifies versioning, required tools, and test paths for the acceptance action. ```json { "version": { "src/databricks/labs/project_name/__about__.py": "__version__ = \"$VERSION\"" }, "toolchain": { "required": ["python3", "hatch"], "pre_setup": ["hatch env create"], "prepend_path": ".venv/bin", "acceptance_path": "tests/integration" } } ``` -------------------------------- ### client.workspace.download(path, format) Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/docs/databricks_apis/workspace_apis.md Downloads the content of a file from the Databricks workspace. ```APIDOC ## client.workspace.download(path, format) ### Description Downloads the content of a workspace file or notebook as bytes. ### Parameters - **path** (string) - Required - The workspace path to download. - **format** (string) - Optional - The format to download the file in (e.g., SOURCE). ``` -------------------------------- ### Instantiate CloneCatalog Class Source: https://github.com/databrickslabs/sandbox/blob/main/uc-catalog-cloning/README.md Create an instance of the CloneCatalog class by providing the source and target catalog details and schema location mappings. ```python cloning = CloneCatalog( source_catalog_external_location_name, source_catalog_name, target_catalog_external_location_pre_req, target_catalog_name, schemas_locations_dict ) ``` -------------------------------- ### Execute Supervisor Agent Replication Workflow Source: https://github.com/databrickslabs/sandbox/blob/main/supervisor-agent-export-import/README.md Use these commands to export an agent configuration from a source workspace and import it into a target workspace with specific catalog mappings. ```bash # 1. Export from source workspace DATABRICKS_CONFIG_PROFILE=source-workspace \ python export_supervisor_agent.py --name "threat-analyst" --output-dir ./exports # 2. Import into target workspace DATABRICKS_CONFIG_PROFILE=target-workspace \ python import_supervisor_agent.py \ --input-dir ./exports/threat-analyst \ --warehouse-id abc123def456 \ --volume-path /Volumes/cybersecurity/default/documents \ --catalog-map "dsl_dlt=new_catalog,sentinel_poc.logs=cyber.sentinel_logs" ``` -------------------------------- ### List Serving Endpoints Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/docs/databricks_apis/model_serving.md Retrieves and prints a list of all available serving endpoints in the workspace. ```python # List all serving endpoints try: endpoints = client.serving_endpoints.list() print("Available serving endpoints:") for endpoint in endpoints: print(f"Name: {endpoint.name}") print(f"State: {endpoint.state}") print(f"Creation Time: {endpoint.creation_timestamp}") print(f"Creator: {endpoint.creator}") print(f"URL: {endpoint.serving_endpoint_url}") print("---") except DatabricksError as e: print(f"Error listing endpoints: {e}") ``` -------------------------------- ### Manage Jobs and Workflows Source: https://github.com/databrickslabs/sandbox/blob/main/judge-builder/docs/databricks_apis/databricks_sdk.md List jobs, trigger immediate runs, and check the status of job executions. ```python # List jobs jobs = client.jobs.list() # Run a job run = client.jobs.run_now(job_id=123) # Get job run status run_status = client.jobs.get_run(run.run_id) ```