### Example README.md Frontmatter for New Projects

Source: https://github.com/databrickslabs/sandbox/blob/main/CONTRIBUTING.md

This snippet provides an example of the required frontmatter metadata for a `README.md` file in a new project folder. It specifies fields like title, language, author, date, and tags, which are used for project categorization and display. The frontmatter must be at the top of the README file.

```markdown
---
title: "Your awesome project title"
language: python
author: "Your Name"
date: 2022-10-15

tags: 
- list
- of
- awesome
- tags
---

# Your awesome project title

... markdown documentation about your project. People will be reading it.
```

--------------------------------

### Programmatic Test Environment Setup in Go

Source: https://github.com/databrickslabs/sandbox/blob/main/acceptance/README.md

Demonstrates how to initialize, load, and start a test environment using the `testenv` library in Go. It includes error handling and deferred cleanup, showing how to run all detected tests.

```Go
testEnv := testenv.NewWithAzureCLI(vaultURI)
loaded, err := testEnv.Load(ctx)
if err != nil {
  return fmt.Errorf("load: %w", err)
}
ctx, stop, err := loaded.Start(ctx)
if err != nil {
  return fmt.Errorf("start: %w", err)
}
defer stop()
cwd, err := os.Getwd()
if err != nil {
  return fmt.Errorf("cwd: %w", err)
}
// detect and run all tests
report, err := ecosystem.RunAll(ctx, loaded.Redaction(), cwd)
if err != nil {
  return fmt.Errorf("test: %w", err)
}
return fmt.Errorf("failed: %s", report.String())
```

--------------------------------

### VSCode Debug Configuration for Go Projects

Source: https://github.com/databrickslabs/sandbox/blob/main/CONTRIBUTING.md

This JSON snippet illustrates a typical `launch.json` configuration for debugging Go applications within VSCode. It sets up a debug profile for a specific Go program, defining its entry point and command-line arguments. This configuration is essential for developers to step through Go code and inspect variables during execution.

```json
{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "metascan clone-all",
            "type": "go",
            "request": "launch",
            "mode": "debug",
            "program": "${workspaceFolder}/metascan/main.go",
            "args": [
                "clone-all",
                "--debug"
            ]
        }
    ]
}
```

--------------------------------

### Install Databricks Labs Sandbox and SQL Migration Assistant

Source: https://github.com/databrickslabs/sandbox/blob/main/sql_migration_assistant/README.md

This command installs the Databricks Labs Sandbox and the SQL Migration Assistant. Ensure the Databricks CLI is installed and configured with the correct workspace before running.

```bash
databricks labs install sandbox && databricks labs sandbox sql-migration-assistant
```

--------------------------------

### Install Frontend Dependencies

Source: https://github.com/databrickslabs/sandbox/blob/main/ka-chat-bot/README.md

Installs all required Node.js packages and dependencies for the React frontend application, as specified in its 'package.json' file.

```bash
npm install
```

--------------------------------

### Install Databricks Connect for Local Development

Source: https://github.com/databrickslabs/sandbox/blob/main/uc-catalog-cloning/README.md

Install the `databricks-connect` library, specifically for running the `uc-catalog-cloning` script from a local development environment that connects to Databricks.

```Bash
databricks-connect==13.3.3
```

--------------------------------

### Install Databricks Labs Sandbox using CLI

Source: https://github.com/databrickslabs/sandbox/blob/main/README.md

This command installs the Databricks Labs Sandbox project. Ensure Databricks CLI v0.210.1+ is installed and configured before running. It sets up the necessary components for the experimental project.

```bash
databricks labs install sandbox
```

--------------------------------

### Go Test Log Artifact Example

Source: https://github.com/databrickslabs/sandbox/blob/main/downstreams/README.md

This Go code snippet demonstrates how to use the `github.com/databrickslabs/sandbox/go-libs/fixtures` package in a Go test. It shows the import of the `fixtures` package and the `WorkspaceTest` function, which is relevant for generating logs available in the `go-slog.json` artifact.

```go
import (
	"testing"
	"github.com/databrickslabs/sandbox/go-libs/fixtures"
)

func TestShowDatabases(t *testing.T) {
	ctx, w := fixtures.WorkspaceTest(t)
  // ...
}
```

--------------------------------

### Example `go test -json` Output

Source: https://github.com/databrickslabs/sandbox/blob/main/acceptance/README.md

Illustrates the structured JSON output generated by the `go test -json` command, useful for programmatic parsing of test events and results.

```JSON
{"Time":"2024-02-03T15:28:06.204157427Z","Action":"start","Package":"github.com/databrickslabs/sandbox/go-libs"}
{"Time":"2024-02-03T15:28:06.204217579Z","Action":"output","Package":"github.com/databrickslabs/sandbox/go-libs","Output":"?   \tgithub.com/databrickslabs/sandbox/go-libs\t[no test files]\n"}
```

--------------------------------

### Install Python Dependencies

Source: https://github.com/databrickslabs/sandbox/blob/main/ka-chat-bot/README.md

Installs the necessary Python packages listed in the 'requirements.txt' file for the backend application, ensuring all required libraries are available.

```bash
pip install -r requirements.txt
```

--------------------------------

### Run FastAPI Server

Source: https://github.com/databrickslabs/sandbox/blob/main/ka-chat-bot/README.md

Starts the FastAPI backend application, making the chatbot's API endpoints accessible. This command initiates the main server process.

```bash
python main.py
```

--------------------------------

### Clone Databricks Unity Catalog with Managed External Location

Source: https://github.com/databrickslabs/sandbox/blob/main/uc-catalog-cloning/example.ipynb

This example demonstrates cloning a Databricks Unity Catalog where the source catalog is associated with a managed external location. It configures the `inputs` dictionary with source and target catalog names, the source's external location, and target external location pre-requisites. Additionally, it defines a `schemas_locations_dict` to map specific schemas to their respective external storage paths. The `CloneCatalog` instance is then initialized and executed to perform the cloning operation.

```python
#If you are cloning a Catalog with a managed external location
inputs = dict(
    source_catalog_external_location_name = 'eo000_ext_loc_ctg2',
    source_catalog_name = 'eo000_ctg_ext_loc2',
    target_catalog_external_location_pre_req = ['eo000_ext_loc_ctg6', 'storage_credential', 'abfss://container0@accountname0.dfs.core.windows.net/'],
    target_catalog_name = 'eo000_ctg_ext_loc6')

schemas_locations_dict = {
    'db1': ['eo000_ext_db_loc01', 'field_demos_credential', 'abfss://container1@accountname1.dfs.core.windows.net/'],
    'db2': ['eo000_ext_db_loc01', 'field_demos_credential', 'abfss://container2@accountname2.dfs.core.windows.net/'],
    'db_empty': ['eo000_ext_db_loc02', 'field_demos_credential', 'abfss://container3@accountname3.dfs.core.windows.net/']
}


clone = CloneCatalog(**inputs, schemas_locations_dict=schemas_locations_dict)
clone()
```

--------------------------------

### Install Databricks Labs Sandbox Tool

Source: https://github.com/databrickslabs/sandbox/blob/main/ip_access_list_analyzer/README.md

Command to install the Databricks Labs sandbox tool, which includes the IP Access List Analyzer and its dependencies.

```sh
databricks labs install sandbox
```

--------------------------------

### GitHub Action Workflow Configuration for Acceptance Testing

Source: https://github.com/databrickslabs/sandbox/blob/main/acceptance/README.md

Example YAML configuration for integrating the acceptance testing GitHub Action into a repository's `.github/workflows` folder. It defines triggers for pull requests, necessary permissions (`id-token: write`, `contents: read`, `pull-requests: write`), and steps for checking out code, setting up Go, and running the `databrickslabs/sandbox/acceptance` action with specified directory and GITHUB_TOKEN.

```yaml
name: acceptance

on:
  pull_request:
    types: [opened, synchronize]
    paths: ['go-libs/**']

permissions:
  id-token: write
  contents: read
  pull-requests: write

jobs:
  acceptance:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Setup Go
        uses: actions/setup-go@v5
        with:
          go-version: 1.21

      - name: Acceptance
        uses: databrickslabs/sandbox/acceptance@actions/artifact
        with:
          directory: go-libs
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
```

--------------------------------

### Example `go test` Standard Error Output

Source: https://github.com/databrickslabs/sandbox/blob/main/acceptance/README.md

Shows typical standard error messages from the `go test` command, specifically when Go modules are downloading dependencies.

```Go
go: downloading github.com/spf13/cobra v1.8.0
go: downloading github.com/fatih/color v1.16.0
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/spf13/viper v1.18.2
go: downloading github.com/stretchr/testify v1.8.4
```

--------------------------------

### Go Code Coverage Profile (`go-coverprofile`)

Source: https://github.com/databrickslabs/sandbox/blob/main/acceptance/README.md

Example content of the `go-coverprofile` artifact, which is a standard Go code coverage profile. It specifies the coverage mode (`set`) and lists covered code blocks by file path, line and column ranges, and coverage counts, indicating which parts of the code were executed during tests.

```text
mode: set
github.com/databrickslabs/sandbox/go-libs/env/context.go:13.53,15.22 2 0
github.com/databrickslabs/sandbox/go-libs/env/context.go:15.22,17.3 1 0
github.com/databrickslabs/sandbox/go-libs/env/context.go:18.2,18.12 1 0
github.com/databrickslabs/sandbox/go-libs/env/context.go:21.52,22.16 1 0
github.com/databrickslabs/sandbox/go-libs/env/context.go:22.16,24.3 1 0
github.com/databrickslabs/sandbox/go-libs/env/context.go:25.2,26.9 2 0
github.com/databrickslabs/sandbox/go-libs/env/context.go:26.9,28.3 1 0
```

--------------------------------

### Navigate to Frontend Directory

Source: https://github.com/databrickslabs/sandbox/blob/main/ka-chat-bot/README.md

Changes the current working directory to the 'frontend' folder, which is necessary before performing any frontend-specific build or installation commands.

```bash
cd frontend
```

--------------------------------

### Import CloneCatalog Class for Databricks Catalog Operations

Source: https://github.com/databrickslabs/sandbox/blob/main/uc-catalog-cloning/example.ipynb

This snippet imports the `CloneCatalog` class from the `clonecatalog` module, which is the primary utility for programmatically cloning Databricks Unity Catalogs. This class encapsulates the logic required to replicate catalog structures and data.

```python
from clonecatalog import CloneCatalog
```

--------------------------------

### Install Core Python Libraries on Databricks

Source: https://github.com/databrickslabs/sandbox/blob/main/uc-catalog-cloning/README.md

Install essential Python libraries, `databricks-sdk` and `termcolor`, required for running the `uc-catalog-cloning` script directly on Databricks clusters.

```Bash
databricks-sdk==0.14.0
termcolor
```

--------------------------------

### Install Tableau Taco Toolkit

Source: https://github.com/databrickslabs/sandbox/blob/main/tableau-delta-sharing-connector-oauth/README.md

Command to globally install the Tableau Web Data Connector 3.0 SDK toolkit using npm, which is a prerequisite for developing and testing Tableau connectors.

```Shell
npm install -g @tableau/taco-toolkit@2.0.0
```

--------------------------------

### CLI Output: metascan summary table

Source: https://github.com/databrickslabs/sandbox/blob/main/metascan/README.md

An example of the tabular output generated by a metascan command, listing repository metadata.

```text
Name                      Title                                              Author         Updated
metascan                  Databricks Sandboxes metadata magagement           Serge Smertin  2023-11-29
runtime-packages          Databricks Runtime package discovery               Serge Smertin  2023-11-29
go-libs                   Utility libraries for Go                           Serge Smertin  2023-11-29
ip_access_list_analyzer   Analyzer/fix tool for Databricks IP Access Lists   Alex Ott       2023-11-29
database-diagram-builder  UML diagram builder for specified Spark Databases  Alex Ott       2023-11-29
```

--------------------------------

### Normalized Integration Test Report (`test-report.json`)

Source: https://github.com/databrickslabs/sandbox/blob/main/acceptance/README.md

Example JSON lines from the `test-report.json` artifact, providing a normalized integration test execution report. Each line represents a test run, detailing timestamp, project, package, test name, flaky status, pass/fail/skip status, output logs, and elapsed time, useful for cross-ecosystem analysis.

```json
{"ts":"2024-02-03T15:28:16.940198637Z","project":"go-libs","package":"sqlexec","name":"TestAccErrorMapping","flaky":false,"pass":false,"skip":true,"output":"=== RUN   TestAccErrorMapping\n    init.go:96: Environment variable CLOUD_ENV is missing\n--- SKIP: TestAccErrorMapping (0.00s)\n","elapsed":0}
{"ts":"2024-02-03T15:28:16.940348556Z","project":"go-libs","package":"sqlexec","name":"TestAccMultiChunk","flaky":true,"pass":true,"skip":false,"output":"...","elapsed":0}
```

--------------------------------

### Example dbfs_scan_results.json Structure

Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/README.md

Illustrates the structure of the `dbfs_scan_results.json` file, showing how DBFS paths map to their size, optional type classification, and lists of JAR or WHL files.

```json
{
  "/FileStore/init-scripts": {
      "size": 2406
  },
  "/cluster-logs/1130-094127-3ij40m7s": {
      "type": "cluster_logs",
      "size": 684830
  },
  "/pipelines/28ae6515-b6f4-4e2e-8b65-879009332b7d": {
      "type": "dlt_storage",
      "size": 589529922
  },
  "/tmp": {
    "whl_files": [
        "2.whl",
        "4.whl"
    ],
    "jar_files": [
        "dbutils-in-jar-0.0.1-jar-with-dependencies.jar",
        "kafka-eventhubs-aad-auth-0.0.1-dbr_10.4.jar"
    ],
    "size": 3560732064
  }
}
```

--------------------------------

### Initialize Workspace Client and Start DBFS Scan

Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/scan_dbfs_nb.ipynb

This code initializes a `WorkspaceClient` instance to interact with the Databricks workspace APIs. It then prepares an empty dictionary named `results` to store the output of the DBFS scan. Finally, it calls the `scan_dbfs` helper function, passing the `WorkspaceClient`, the `results` dictionary, and the `path` obtained from the widget to begin the recursive DBFS scan.

```python
wc = WorkspaceClient()
results = {}
print("Starting scanning DBFS at path:", path)
scan_dbfs(wc, results, path)
```

--------------------------------

### GitHub Actions Raw Event Data (`event.json`)

Source: https://github.com/databrickslabs/sandbox/blob/main/acceptance/README.md

Example JSON structure of the `event.json` artifact, which contains the raw event information from GitHub Actions. This includes details such as the action type (`synchronize`), `after` and `before` commit SHAs, enterprise and organization data, and pull request metadata.

```json
{
  "action": "synchronize",
  "after": "faa8a5ba40d987cb981362581e556eb676761161",
  "before": "3d0b4fe5bdc45292358a29dde28718e1605efcad",
  "enterprise": {
    ...
  },
  "number": 69,
  "organization": {
    ...
  },
  "pull_request": {
    "_links": {

```

--------------------------------

### Upgrade Databricks Labs Sandbox via CLI

Source: https://github.com/databrickslabs/sandbox/blob/main/README.md

This command upgrades an existing Databricks Labs Sandbox installation to its latest version. It ensures you have the most recent features and bug fixes for the experimental project.

```bash
databricks labs upgrade sandbox
```

--------------------------------

### Analyze IP Access Lists from JSON File

Source: https://github.com/databrickslabs/sandbox/blob/main/ip_access_list_analyzer/README.md

Example command to run the IP Access List Analyzer in file-based analysis mode, using a specified JSON file as input. This mode performs analysis without applying any fixes to the workspace.

```sh
databricks labs sandbox ip-access-list-analyzer --json_file=test.json
```

--------------------------------

### Clone Databricks Unity Catalog with Default Managed Location

Source: https://github.com/databrickslabs/sandbox/blob/main/uc-catalog-cloning/example.ipynb

This snippet illustrates how to clone a Databricks Unity Catalog when the source catalog utilizes a default managed location. The `inputs` dictionary is set up with source and target catalog names, and target external location pre-requisites, with `source_catalog_external_location_name` set to `None`. Similar to the external location scenario, `schemas_locations_dict` specifies storage paths for individual schemas. The `CloneCatalog` class is instantiated with these parameters and invoked to initiate the cloning process.

```python
#If you are cloning a Catalog with a default managed location
inputs = dict(
    source_catalog_external_location_name = None,
    source_catalog_name = 'eo000_ctg1',
    target_catalog_external_location_pre_req = ['eo000_ext_loc_ctg8', 'storage_credential', 'abfss://container0@accountname0.dfs.core.windows.net/root'],
    target_catalog_name = 'eo000_ctg_ext_loc8')

schemas_locations_dict = {
    'db1': ['eo000_ext_db_loc01', 'storage_credential', 'abfss://container1@accountname1.dfs.core.windows.net/rootdb'],
    'db2': ['eo000_ext_db_loc01', 'storage_credential', 'abfss://container2@accountname2.dfs.core.windows.net/'],
    'db_empty': ['eo000_ext_db_loc02', 'storage_credential', 'abfss://container3@accountname3.dfs.core.windows.net/']
}


clone = CloneCatalog(**inputs, schemas_locations_dict=schemas_locations_dict)
clone()
```

--------------------------------

### GitHub Actions Workflow for Downstream Dependency Testing

Source: https://github.com/databrickslabs/sandbox/blob/main/downstreams/README.md

This YAML configuration defines a GitHub Actions workflow named 'downstreams'. It triggers on pull requests, merge groups, and pushes to the 'main' branch. The workflow checks out the repository, sets up Python, installs a toolchain (hatch), and then runs an 'Acceptance' step using a custom GitHub Action to test against specified downstream projects (ucx, lsql, remorph).

```yaml
name: downstreams

on:
  pull_request:
    types: [opened, synchronize]
  merge_group:
    types: [checks_requested]
  push:
    # Always run on push to main. The build cache can only be reused
    # if it was saved by a run from the repository's default branch.
    # The run result will be identical to that from the merge queue
    # because the commit is identical, yet we need to perform it to
    # seed the build cache.
    branches:
      - main

permissions:
  id-token: write
  contents: read
  pull-requests: write

jobs:
  ci:
    strategy:
      fail-fast: false
      matrix:
        downstream:
          - name: ucx
          - name: lsql
          - name: remorph
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: Install Python
        uses: actions/setup-python@v4
        with:
          cache: 'pip'
          cache-dependency-path: '**/pyproject.toml'
          python-version: 3.10

      - name: Install toolchain
        run: |
          pip install hatch==1.9.4

      - name: Acceptance
        uses: databrickslabs/sandbox/downstreams@downstreams/v0.0.1
        with:
          repo: ${{ matrix.downstream.name }}
          org: databrickslabs
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
```

--------------------------------

### Go Test Logging with `fixtures` Package

Source: https://github.com/databrickslabs/sandbox/blob/main/acceptance/README.md

Demonstrates how to integrate `github.com/databrickslabs/sandbox/go-libs/fixtures` into a Go test function. Using `fixtures.WorkspaceTest(t)` initializes a test context and workspace, enabling logs to be captured and made available in the `go-slog.json` artifact for debugging and analysis.

```go
import (
	"testing"
	"github.com/databrickslabs/sandbox/go-libs/fixtures"
)

func TestShowDatabases(t *testing.T) {
	ctx, w := fixtures.WorkspaceTest(t)
  // ...
}
```

--------------------------------

### Build and Run Databricks Runtime Package Discovery Tool from Source

Source: https://github.com/databrickslabs/sandbox/blob/main/runtime-packages/README.md

These commands provide instructions for compiling the `runtime-packages` tool from its source code and then executing the compiled binary. The `make dist` command handles the build process, and the subsequent command runs the generated executable.

```bash
make dist
dist/runtime-packages
```

--------------------------------

### Run Tableau Connector Locally in Desktop

Source: https://github.com/databrickslabs/sandbox/blob/main/tableau-delta-sharing-connector-oauth/README.md

Launches Tableau Desktop and runs the `.taco` file, initiating both the interactive phase (for user authentication/configuration) and the data gathering phase of the connector for testing purposes.

```Shell
taco run Desktop
```

--------------------------------

### Package Tableau Connector for Desktop Testing

Source: https://github.com/databrickslabs/sandbox/blob/main/tableau-delta-sharing-connector-oauth/README.md

Creates a `.taco` file from the compiled project, which is the deployable package format for Tableau Desktop, enabling local testing of the connector.

```Shell
taco pack
```

--------------------------------

### CLI Usage: metascan help command

Source: https://github.com/databrickslabs/sandbox/blob/main/metascan/README.md

Demonstrates how to access help information for 'metascan' commands.

```cli
metascan [command] --help
```

--------------------------------

### Instantiate CloneCatalog Class with Configuration

Source: https://github.com/databrickslabs/sandbox/blob/main/uc-catalog-cloning/README.md

Create an instance of the `CloneCatalog` class, providing the necessary configuration parameters such as source and target catalog names, external locations, and a dictionary of schema locations. Replace placeholders with your specific details.

```Python
cloning = CloneCatalog(
         source_catalog_external_location_name,
         source_catalog_name,
         target_catalog_external_location_pre_req,
         target_catalog_name,
         schemas_locations_dict
)
```

--------------------------------

### Acceptance Testing Workflow Sequence Diagram

Source: https://github.com/databrickslabs/sandbox/blob/main/acceptance/README.md

Illustrates the detailed sequence of operations for the acceptance testing workflow, including interactions with GitHub Actions, Microsoft Entra, Azure Key Vault, Metadata Server, and Databricks API, showing how tokens and data flow.

```mermaid
sequenceDiagram
    acceptance->>+ACTIONS_ID_TOKEN_REQUEST_URL: (1) ACTIONS_ID_TOKEN_REQUEST_TOKEN
    ACTIONS_ID_TOKEN_REQUEST_URL->>-acceptance: (2) JWT assertion
    acceptance->>+Microsoft Entra: (3) JWT assertion + client ID + resource ID
    Microsoft Entra->>-acceptance: (4) Access Token for Azure Key Vault
    acceptance->>+Azure Key Vault: (5) request environment variables
    Azure Key Vault->>-acceptance: (6) test environment
    acceptance->>+Metadata Server: (7) start auth token proxy in a thread
    Metadata Server->>-acceptance: (8) http://localhost:<random-port>/<random-prefix>
    acceptance->>+Test Runner: (9) start test runner subprocess with relevant environment
    Test Runner->>+test: (10) start test execution
    test->>+SDK: (11) call API
    SDK->>+Metadata Server: (12) call localhost:<random-port>/<random-prefix>
    Metadata Server->>+ACTIONS_ID_TOKEN_REQUEST_URL: (13) ACTIONS_ID_TOKEN_REQUEST_TOKEN
    ACTIONS_ID_TOKEN_REQUEST_URL->>-Metadata Server: (14) JWT assertion to request token for Databricks
    Metadata Server->>+Microsoft Entra: (15) JWT assertion + client ID + resource ID
    Microsoft Entra->>-Metadata Server: (16) Access token for Databricks
    Metadata Server->>-SDK: (17) Access token for Databricks
    SDK->>+Databricks API: (18) call API
    Databricks API->>-SDK: (19) deserialized result
    SDK->>-test: (20) deserialized result
    test->>-Test Runner: (21) success or failure
    Test Runner->>-acceptance: (22) success or failure + redacted logs
```

--------------------------------

### Define Databricks Widgets for User Input

Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/scan_dbfs_nb.ipynb

This code defines two Databricks `dbutils` text widgets. The `start_dir` widget allows users to specify the initial directory for the DBFS scan, defaulting to the root ('/'). The `output_dir` widget allows specifying an optional output location for scan results, such as a Unity Catalog Volume.

```python
dbutils.widgets.text("start_dir", "/", "Directory to scan")
dbutils.widgets.text("output_dir", "", "Output directory (UC Volume, etc.)")
```

--------------------------------

### Build Production Frontend

Source: https://github.com/databrickslabs/sandbox/blob/main/ka-chat-bot/README.md

Generates an optimized, production-ready build of the React frontend application. This build is minified and optimized for deployment to a production environment.

```bash
npm run build:prod
```

--------------------------------

### CLI Command: scan

Source: https://github.com/databrickslabs/sandbox/blob/main/metascan/README.md

API documentation for the 'scan' command, which processes README.md frontmatter.

```APIDOC
scan:
  Description: Scans frontmatter of README.md files in current directory and shows summary
```

--------------------------------

### CLI Usage for Databricks Runtime Package Discovery Tool

Source: https://github.com/databrickslabs/sandbox/blob/main/runtime-packages/README.md

This snippet illustrates the basic command-line interface usage for the `runtime-packages` tool. It indicates that the tool accepts various flags to control its behavior, output, and connectivity to Databricks workspaces.

```bash
runtime-packages [flags]
```

--------------------------------

### Build Tableau Connector Project with Taco Toolkit

Source: https://github.com/databrickslabs/sandbox/blob/main/tableau-delta-sharing-connector-oauth/README.md

Compiles and builds the Tableau Web Data Connector project using the `taco build` command, preparing it for packaging or local execution.

```Shell
taco build
```

--------------------------------

### Run Databricks IP Access List Analyzer

Source: https://github.com/databrickslabs/sandbox/blob/main/ip_access_list_analyzer/README.md

General command to execute the Databricks IP Access List Analyzer tool with optional parameters. Use `--help` for built-in assistance and `--debug` for detailed log output.

```sh
databricks labs sandbox ip-access-list-analyzer [options]
```

--------------------------------

### Build Local Frontend

Source: https://github.com/databrickslabs/sandbox/blob/main/ka-chat-bot/README.md

Generates a development-ready build of the React frontend application. This build is typically optimized for local testing and debugging.

```bash
npm run build
```

--------------------------------

### CLI Flags for Databricks Runtime Package Discovery Tool

Source: https://github.com/databrickslabs/sandbox/blob/main/runtime-packages/README.md

This section details the various command-line flags available for the `runtime-packages` tool, categorized by their function. It includes flags for configuring connectivity to Databricks and controlling the output behavior, such as specifying cluster IDs, output format, and package types to include.

```APIDOC
Command: runtime-packages

Flags:
  Connectivity:
    --profile:
      Type: string
      Description: Connection profile specified within ~/.databrickscfg.
    --host:
      Type: string
      Description: Databricks workspace host.
  Output behavior:
    --cluster-id:
      Type: string
      Description: Databricks interactive cluster to run package discovery job. Shows packages only for one runtime.
    --instance-pool-id:
      Type: string
      Description: Instance pool to run discovery. Scans all listed Databricks Runtime Versions and stores them in the cache.
    --json:
      Type: boolean
      Description: Output in JSON instead of a table.
    --include-java:
      Type: boolean
      Description: Include JVM packages in the output table.
    --include-ml:
      Type: boolean
      Description: Include Databricks Runtime for Machine Learning packages in the output table (default true).
    --include-python:
      Type: boolean
      Description: Include Python packages in the output table (default true).
    --last-runtimes:
      Type: int
      Description: Maximum number of Databricks Runtime versions to display (default 10).
    --lts:
      Type: boolean
      Description: Only include Databricks Runtimes with the long-term support.
```

--------------------------------

### CLI Usage: metascan basic command

Source: https://github.com/databrickslabs/sandbox/blob/main/metascan/README.md

Shows the basic command structure for the 'metascan' CLI tool.

```cli
metascan [command]
```

--------------------------------

### CLI Command: metascan scan-thos

Source: https://github.com/databrickslabs/sandbox/blob/main/metascan/README.md

API documentation for the 'scan-thos' command, including its specific flags.

```APIDOC
scan-thos:
  Description: Scans frontmatter of README.md files in current directory and shows summary
  Flags:
    --fail: fail on validation errors
```

--------------------------------

### Execute Catalog Cloning Process

Source: https://github.com/databrickslabs/sandbox/blob/main/uc-catalog-cloning/README.md

Initiate the catalog cloning operation by calling the instantiated `CloneCatalog` object. This command triggers the full cloning workflow as configured by the instance parameters.

```Python
cloning()
```

--------------------------------

### Import Python Modules and Libraries

Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/scan_dbfs_nb.ipynb

This snippet imports necessary modules and libraries for the script. It includes `WorkspaceClient` from the Databricks SDK for API interactions, `scan_dbfs` from a local helper module, and standard Python modules like `sys`, `os`, `pprint`, and `json` for system operations, pretty printing, and JSON handling.

```python
from databricks.sdk import WorkspaceClient
from helpers.dbfs_analyzer import scan_dbfs

import sys
import os
import pprint
import json
```

--------------------------------

### Retrieve Databricks Custom App Integration Details via CLI

Source: https://github.com/databrickslabs/sandbox/blob/main/conversational-agent-app/README.md

Demonstrates how to use the Databricks CLI to fetch the configuration details of a custom OAuth2 application integration, including its client ID, scopes, and token access policy. This is useful for verifying current settings or preparing for updates.

```CLI
databricks account custom-app-integration get '12345667-1234-5678-a85d-eac774235aea'

{
  "client_id":"12345667-1234-5678-a85d-eac774235aea",
  "confidential":true,
  "create_time":"2025-02-18T09:59:07.876Z",
  "created_by":4377920341700116,
  "integration_id":"abcdefgg-1234-5678-a85d-eac774235aea",
  "name":"app-abcdeft 1234xp",
  "redirect_urls": [
    "http://localhost:7070",
    "https://mzeni-obo-app-url.azure.databricksapps.com/.auth/callback"
  ],
  "scopes": [
    "openid",
    "profile",
    "email",
    "all-apis",
    "offline_access"
  ],
  "token_access_policy": {
    "access_token_ttl_in_minutes":60,
    "refresh_token_ttl_in_minutes":10080
  }
}
```

--------------------------------

### Initialize Databricks Workspace Client and Results Dictionary

Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/scan_compute_nb.ipynb

This code initializes an instance of the WorkspaceClient from the Databricks SDK, which is used to interact with the Databricks workspace. It also initializes an empty dictionary full_results to store the outcomes of various compute resource scans.

```python
wc = WorkspaceClient()
full_results: dict = {}
```

--------------------------------

### CLI Flag: --debug

Source: https://github.com/databrickslabs/sandbox/blob/main/metascan/README.md

API documentation for the global '--debug' flag to enable verbose logging.

```APIDOC
--debug:
  Description: Enable debug log output
```

--------------------------------

### Import Python Libraries for Databricks Compute Analysis

Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/scan_compute_nb.ipynb

This snippet imports necessary Python libraries for interacting with the Databricks Workspace and custom helper functions for analyzing compute resources. It includes the Databricks SDK, utility modules like sys, os, pprint, and json.

```python
from databricks.sdk import WorkspaceClient
from helpers.compute_analyzer import analyze_clusters, analyze_cluster_policies, analyze_jobs, analyze_dlt_pipelines

import sys
import os
import pprint
import json
```

--------------------------------

### Import CloneCatalog Class

Source: https://github.com/databrickslabs/sandbox/blob/main/uc-catalog-cloning/README.md

Import the `CloneCatalog` class from the `clonecatalog` module into your Python script or Jupyter notebook to begin using the cloning functionality.

```Python
from clonecatalog import CloneCatalog
```

--------------------------------

### Define Databricks Widget for Output Directory

Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/scan_compute_nb.ipynb

This code defines a Databricks dbutils text widget named 'output_dir'. This widget allows users to specify an output directory, such as a Unity Catalog Volume, for storing scan results.

```python
dbutils.widgets.text("output_dir", "", "Output directory (UC Volume, etc.)")
```

--------------------------------

### Retrieve Widget Values in Databricks

Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/scan_dbfs_nb.ipynb

This snippet retrieves the values entered by the user into the previously defined Databricks widgets. The value from the `start_dir` widget is assigned to the `path` variable, and the value from the `output_dir` widget is assigned to the `output_dir` variable, preparing them for use in the script.

```python
path = dbutils.widgets.get("start_dir")
output_dir = dbutils.widgets.get("output_dir")
```

--------------------------------

### Display and Persist DBFS Scan Results

Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/scan_dbfs_nb.ipynb

This snippet first prints the collected DBFS scan results to the console using `pprint.pprint` for enhanced readability. If an `output_dir` was provided via the widget, it then writes the `results` dictionary as a pretty-printed JSON file named `dbfs_scan_results.json` to the specified output directory, allowing for persistence and further analysis.

```python
print("Scan results: ")
pprint.pprint(results)
if output_dir:
    with open(os.path.join(output_dir, "dbfs_scan_results.json"), "w") as f:
        f.write(json.dumps(results, indent=4))
```

--------------------------------

### Display and Save Databricks Compute Scan Results

Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/scan_compute_nb.ipynb

This final snippet displays the aggregated scan results using pprint. If an output_dir was provided via the widget, it serializes the full_results dictionary to a JSON file named 'compute_scan_results.json' within that directory, ensuring persistent storage of the analysis.

```python
print("Scan results: ")
pprint.pprint(full_results)
if output_dir:
    with open(os.path.join(output_dir, "compute_scan_results.json"), "w") as f:
        f.write(json.dumps(full_results, indent=4))
```

--------------------------------

### Databricks Account Custom App Integration API Reference

Source: https://github.com/databrickslabs/sandbox/blob/main/conversational-agent-app/README.md

API reference for managing custom OAuth2 application integrations within Databricks accounts. This includes operations for retrieving and updating application configurations, specifically focusing on scopes and client details.

```APIDOC
APIDOC: Databricks Account Custom App Integration

Resource: CustomAppIntegration
  Description: Represents an OAuth2 custom application integration within a Databricks account.

  Properties:
    client_id: string (Read-only) - The unique client ID of the OAuth2 application.
    confidential: boolean (Read-only) - Indicates if the application is confidential.
    create_time: string (ISO 8601 datetime, Read-only) - Timestamp when the integration was created.
    created_by: number (Read-only) - User ID of the creator.
    integration_id: string (Read-only) - The unique ID of the integration.
    name: string (Read-only) - The name of the custom application.
    redirect_urls: array<string> (Read-only) - List of allowed redirect URIs for OAuth2.
    scopes: array<string> (Updatable) - List of OAuth2 scopes granted to the application.
    token_access_policy: object (Read-only) - Defines token lifetime policies.
      access_token_ttl_in_minutes: number - Lifetime of access tokens in minutes.
      refresh_token_ttl_in_minutes: number - Lifetime of refresh tokens in minutes.

Endpoints:
  GET /api/2.0/accounts/{account_id}/custom-app-integrations/{integration_id}
    Description: Retrieves the details of a specific custom application integration.
    Parameters:
      account_id: string (Path) - The ID of the Databricks account.
      integration_id: string (Path) - The ID of the custom app integration.
    Returns: CustomAppIntegration object

  PUT /api/2.0/accounts/{account_id}/custom-app-integrations/{integration_id}
    Description: Updates the configuration of a specific custom application integration.
    Parameters:
      account_id: string (Path) - The ID of the Databricks account.
      integration_id: string (Path) - The ID of the custom app integration.
      body: object - Request body containing fields to update.
        scopes: array<string> (Required) - The complete list of scopes to be granted to the application. Must include all desired scopes, not just additions.
    Returns: CustomAppIntegration object (updated state)
```

--------------------------------

### Retrieve Output Directory from Databricks Widget

Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/scan_compute_nb.ipynb

This snippet retrieves the value of the 'output_dir' widget, previously defined using dbutils.widgets.text. The retrieved path will be used to save the scan results.

```python
output_dir = dbutils.widgets.get("output_dir")
```

--------------------------------

### Schema for compute_scan_results.json Output File

Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/README.md

Defines the structure of the JSON file generated by `scan_compute.py`, detailing findings related to DBFS usage across Databricks compute resources. This includes configurations for interactive clusters, cluster policies, jobs, and Delta Live Tables (DLT) pipelines, specifically highlighting paths and references to DBFS.

```APIDOC
compute_scan_results.json (object):
  Purpose: Stores findings from scanning Databricks compute resources for DBFS dependencies.

  Properties:
    clusters (object): Information about interactive clusters.
      Keys: Cluster IDs (string)
      Value (object):
        cluster_log_conf (string, optional): Path to cluster logs, e.g., "dbfs:/mnt/cluster-logs".
        init_scripts (array of string, optional): List of init script paths, e.g., "dbfs:/path/to/script.sh".
        cluster_name (string, optional): Name of the cluster.
        libraries (array of string, optional): List of library paths (JAR/wheel files), e.g., "dbfs:/FileStore/whl/test.whl".

    cluster_policies (object): Information about cluster policies.
      Keys: Policy IDs (string)
      Value (object):
        cluster_log_conf (string, optional): Path to cluster logs.
        libraries (array of string, optional): List of library paths (JAR/wheel files).
        policy_name (string, optional): Name of the policy.

    jobs (object): Information about jobs.
      Keys: Job IDs (string)
      Value (object):
        tasks (object): Individual task configurations.
          Keys: Task names (string)
          Value (object):
            python_file_on_dbfs (string, optional): Path to Python file on DBFS.
            libraries (array of string, optional): List of library paths (JAR/wheel files).
            init_scripts (array of string, optional): List of init script paths.
        job_clusters (object, optional): Job-specific cluster configurations.
          Keys: Job Cluster IDs (string)
          Value (object):
            cluster_log_conf (string, optional): Path to cluster logs.
            init_scripts (array of string, optional): List of init script paths.
        job_name (string, optional): Name of the job.

    dlt (object): Information about Delta Live Tables pipelines.
      Keys: Pipeline IDs (string)
      Value (object):
        libraries (object, optional): Library references within the DLT pipeline.
          Keys: Library paths (string)
          Value (object):
            dbfs_file_refs (array of string, optional): List of DBFS file references.
        storage (string, optional): Storage location for the pipeline, e.g., "dbfs:/pipelines/...".
        name (string, optional): Name of the DLT pipeline.
```

```json
{
  "clusters": {
    "0313-121649-cxqqcvz0": {
      "cluster_log_conf": "dbfs:/mnt/cluster-logs",
      "cluster_name": "EvHub",
      "libraries": [
        "dbfs:/FileStore/whl/test.whl",
        "dbfs:/FileStore/jars/test.jar"
      ]
    }
  },
  "cluster_policies": {
    "30640179C40010EF": {
      "cluster_log_conf": "dbfs:/mnt/cluster-logs",
      "policy_name": "minimum_global"
    },
    "0005268696213EA3": {
      "libraries": [
        "dbfs:/FileStore/jars/jar-tests-0.0.1.jar"
      ],
      "policy_name": "Test Libraries"
    }
  },
  "jobs": {
    "75390238197996": {
      "tasks": {
        "python_script_on_dbfs": {
          "python_file_on_dbfs": "dbfs:/1.py"
        },
        "abc": {
          "libraries": [
            "dbfs:/1.jar",
            "dbfs:/1.whl"
          ]
        }
      }
    }
  },
  "dlt": {
    "713294fd-9e53-4f2b-b8fe-1d924fb4905c": {
      "libraries": {
        "/Users/user@domain.com/Test/test": {
          "dbfs_file_refs": [
            "dbfs:/FileStore/temp/test_data/input"
          ]
        }
      },
      "storage": "dbfs:/pipelines/713294fd-9e53-4f2b-b8fe-1d924fb4905c",
      "name": "dlt_dabs_pipeline"
    }
  }
}
```

--------------------------------

### Update Databricks Custom App Integration Scopes via CLI

Source: https://github.com/databrickslabs/sandbox/blob/main/conversational-agent-app/README.md

Illustrates how to modify the scopes of an existing Databricks custom OAuth2 application integration using the CLI. This command allows adding new permissions, such as 'serving.serving-endpoints', while preserving existing scopes.

```CLI
databricks account custom-app-integration update '65d90ec2-54ba-4fcb-a85d-eac774235aea' --json '{"scopes": ["openid", "profile", "email", "all-apis", "offline_access", "serving.serving-endpoints"]}'
```

--------------------------------

### Scan Databricks DLT Pipelines for DBFS Usage

Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/scan_compute_nb.ipynb

This code scans Databricks Delta Live Tables (DLT) pipelines using analyze_dlt_pipelines to identify those that utilize DBFS. The discovered pipelines are recorded in the full_results dictionary under the 'dlt' key, and a status message is printed.

```python
print("Starting scanning DLT pipelines")
dlt_results = analyze_dlt_pipelines(wc)
if dlt_results:
    print(f"Found {len(dlt_results)} DLT pipelines using DBFS")
    full_results["dlt"] = dlt_results
else:
    print("No DLT pipelines using DBFS found")
```

--------------------------------

### Scan Databricks Clusters for DBFS Usage

Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/scan_compute_nb.ipynb

This snippet initiates a scan of Databricks clusters using the analyze_clusters helper function. It checks for clusters utilizing DBFS and stores the findings in the full_results dictionary under the 'clusters' key, providing a count of discovered clusters.

```python
print("Starting scanning clusters")
cluster_results = analyze_clusters(wc)
if cluster_results:
    print(f"Found {len(cluster_results)} clusters using DBFS")
    full_results["clusters"] = cluster_results
else:
    print("No clusters using DBFS found")
```

--------------------------------

### Scan Databricks Cluster Policies for DBFS Usage

Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/scan_compute_nb.ipynb

This code scans Databricks cluster policies using analyze_cluster_policies to identify any policies configured to use DBFS. The results are then added to the full_results dictionary under the 'cluster_policies' key, along with a confirmation message.

```python
print("Starting scanning cluster policies")
cluster_policy_results = analyze_cluster_policies(wc)
if cluster_policy_results:
    print(f"Found {len(cluster_policy_results)} cluster policies using DBFS")
    full_results["cluster_policies"] = cluster_policy_results
else:
    print("No cluster policies using DBFS found")
```

--------------------------------

### Scan Databricks Jobs for DBFS Usage

Source: https://github.com/databrickslabs/sandbox/blob/main/dbfs-scanner/scan_compute_nb.ipynb

This snippet performs a scan of Databricks jobs using the analyze_jobs function to detect jobs that interact with DBFS. The findings are stored in the full_results dictionary under the 'jobs' key, and a message indicates the number of jobs found.

```python
print("Starting scanning jobs")
jobs_results = analyze_jobs(wc)
if jobs_results:
    print(f"Found {len(jobs_results)} jobs using DBFS")
    full_results["jobs"] = jobs_results
else:
    print("No jobs using DBFS found")
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.