Azure Databricks

https://learn.microsoft.com/en-us/azure/databricks/
Admin
Azure Databricks is a unified analytics platform that enables data analysts, engineers, scientists,...
Tokens:9,083,213
Snippets:74,068
Trust Score:10
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### List installations example

Source: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/consumer-installations-commands

Example of listing all installations across all listings.

```bash
databricks consumer-installations list

```

--------------------------------

### Start a sandbox example

Source: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/sandbox-commands

Example of starting a sandbox with ID my-sandbox-1234.

```bash
databricks sandbox start my-sandbox-1234

```

--------------------------------

### Create installation example

Source: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/consumer-installations-commands

Example of creating an installation for a listing. Replace <listing-id> with the actual listing ID.

```bash
databricks consumer-installations create <listing-id>

```

--------------------------------

### List listing installations example

Source: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/consumer-installations-commands

Example of listing installations for a specific listing with ID 12345.

```bash
databricks consumer-installations list-listing-installations 12345

```

--------------------------------

### Update installation example

Source: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/consumer-installations-commands

Example of updating an installation with listing ID 12345 and installation ID 67890.

```bash
databricks consumer-installations update 12345 67890

```

--------------------------------

### Get activation URL information example

Source: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/recipient-activation-commands

Example that gets activation URL information for the specified activation URL.

```bash
databricks recipient-activation get-activation-url-info "https://example.com/activation-url"
```

--------------------------------

### Get input file block start offset with example output

Source: https://learn.microsoft.com/en-us/azure/databricks/pyspark/reference/functions/input_file_block_start

Reads a CSV file with a comma line separator and selects the input file block start offset for each row. The output shows eight rows, each with a start offset of 0.

```python
from pyspark.sql import functions as sf
df = spark.read.text("python/test_support/sql/ages_newlines.csv", lineSep=",")
df.select(sf.input_file_block_start()).show()

```

```text
+------------------------+
|input_file_block_start()|
+------------------------+
|                       0|
|                       0|
|                       0|
|                       0|
|                       0|
|                       0|
|                       0|
|                       0|
+------------------------+

```

--------------------------------

### Delete installation example

Source: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/consumer-installations-commands

Example of deleting an installation with listing ID 12345 and installation ID 67890.

```bash
databricks consumer-installations delete 12345 67890

```

--------------------------------

### LIST FILE examples

Source: https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/sql-ref-syntax-aux-resource-mgmt-list-file

Examples showing adding files and listing them. The output shows the file paths.

```sql
> ADD FILE /tmp/test /tmp/test_2;

> LIST FILE;
  file:/private/tmp/test
  file:/private/tmp/test_2

> LIST FILE /tmp/test /some/random/file /another/random/file
  file:/private/tmp/test
```

--------------------------------

### Run databricks quickstart

Source: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/quickstart-command

Runs the quickstart command to print the Databricks CLI quickstart guide. By default, it prints a human-friendly guide; when stdout is not an interactive terminal, it prints a denser, agent-oriented version instead.

```bash
databricks quickstart
```

--------------------------------

### Example: Create share, add tables, and list content

Source: https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/sql-ref-syntax-aux-show-all-in-share

This example creates a share named customer_share, adds two tables to it, and then lists the content of the share. The output shows the shared objects, their types, and partition information.

```sql
-- Create share `customer_share` only if share with same name doesn't exist, with a comment.
> CREATE SHARE IF NOT EXISTS customer_share COMMENT 'This is customer share';

-- Add 2 tables to the share.
-- Expose my_schema.tab1 a different name.
-- Expose only two partitions of other_schema.tab2
> ALTER SHARE customer_share ADD TABLE my_schema.tab1 AS their_schema.tab1;
> ALTER SHARE customer_share ADD TABLE other_schema.tab2 PARTITION (c1 = 5), (c1 = 7);

-- List the content of the share
> SHOW ALL IN SHARE customer_share;
  name              type  shared_object          added_at                     added_by                   comment partitions
  ----------------- ----- ---------------------- ---------------------------- -------------------------- ------- ------------------
  other_schema.tab2 TABLE main.other_schema.tab2 2022-01-01T00:00:01.000+0000 alwaysworks@databricks.com NULL
  their_schema.tab1 TABLE main.myschema.tab2     2022-01-01T00:00:00.000+0000 alwaysworks@databricks.com NULL    (c1 = 5), (c1 = 7)

```

--------------------------------

### Get password permissions example

Source: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/users-v2-commands

Example command that gets password permissions.

```bash
databricks users-v2 get-permissions
```

--------------------------------

### databricks labs installed example

Source: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/labs-commands

Example of listing installed Databricks Labs applications.

```bash
databricks labs installed

```

--------------------------------

### SHOW PARTITIONS examples

Source: https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/sql-ref-syntax-aux-show-partitions

Creates a partitioned table, inserts rows, and demonstrates SHOW PARTITIONS with no partition spec, a qualified table name, a full partition spec, and partial partition specs. The output shows the matching partition paths.

```sql
-- create a partitioned table and insert a few rows.
> USE salesdb;
> CREATE TABLE customer(id INT, name STRING) PARTITIONED BY (state STRING, city STRING);
> INSERT INTO customer PARTITION (state = 'CA', city = 'Fremont') VALUES (100, 'John');
> INSERT INTO customer PARTITION (state = 'CA', city = 'San Jose') VALUES (200, 'Marry');
> INSERT INTO customer PARTITION (state = 'AZ', city = 'Peoria') VALUES (300, 'Daniel');

-- Lists all partitions for table `customer`
> SHOW PARTITIONS customer;
   state=AZ/city=Peoria
  state=CA/city=Fremont
 state=CA/city=San Jose

-- Lists all partitions for the qualified table `customer`
> SHOW PARTITIONS salesdb.customer;
   state=AZ/city=Peoria
  state=CA/city=Fremont
 state=CA/city=San Jose

-- Specify a full partition spec to list specific partition
> SHOW PARTITIONS customer PARTITION (state = 'CA', city = 'Fremont');
 |state=CA/city=Fremont|

-- Specify a partial partition spec to list the specific partitions
> SHOW PARTITIONS customer PARTITION (state = 'CA');
  state=CA/city=Fremont
 state=CA/city=San Jose

-- Specify a partial spec to list specific partition
> SHOW PARTITIONS customer PARTITION (city =  'San Jose');
 state=CA/city=San Jose
```

--------------------------------

### Get workspace setting example

Source: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/workspace-settings-v2-commands

Example that gets a workspace setting named my-setting-name.

```bash
databricks workspace-settings-v2 get-public-workspace-setting my-setting-name

```

--------------------------------

### Run quickstart and start-app scripts

Source: https://learn.microsoft.com/en-us/azure/databricks/agents/custom-agents/multi-agent-apps

Run these commands to set up Azure Databricks authentication, create an MLflow experiment for tracing, and launch the agent server with a chat UI at http://localhost:8000.

```bash
uv run quickstart
uv run start-app

```

--------------------------------

### Run quickstart script

Source: https://learn.microsoft.com/en-us/azure/databricks/agents/custom-agents/migrate-agent-to-apps

Run `uv run quickstart` to configure authentication, create the MLflow experiment, and generate the `.env` file.

```bash
uv run quickstart

```

--------------------------------

### Install and start HAProxy on RHEL 9.4

Source: https://learn.microsoft.com/en-us/azure/databricks/repos/connect-on-prem-git-server

Commands to install and start HAProxy on Red Hat Enterprise Linux 9.4. The sequence updates the system, installs HAProxy, opens the configuration file for editing, starts the service, checks its status, and enables it to start on boot.

```bash
sudo yum update -y
sudo yum install haproxy -y
sudo vim /etc/haproxy/haproxy.cfg
sudo systemctl start haproxy
sudo systemctl status haproxy
sudo systemctl enable haproxy

```

--------------------------------

### databricks quickstart

Source: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/quickstart-command

Documentation for the Databricks CLI `quickstart` command, including its syntax, global flags, and an example invocation.

```APIDOC
## `databricks quickstart`

### Description
Prints a short introduction to the Databricks CLI, covering authentication, profiles, building with Declarative Automation Bundles, and where to go next. By default, prints a human-friendly guide; when stdout is not an interactive terminal, prints a denser, agent-oriented version.

### Command
`databricks quickstart [flags]`

### Options
This command has no options other than the global flags.

### Global Flags
- `--debug` (boolean) - Enable debug logging.
- `-h`, `--help` (boolean) - Display help for the Databricks CLI or the related command group or the related command.
- `--log-file` (string) - File to write output logs to. Default: stderr.
- `--log-format` (format) - Log format type: `text` or `json`. Default: `text`.
- `--log-level` (string) - Log format level. If not specified, disabled.
- `-o`, `--output` (type) - Command output type: `text` or `json`. Default: `text`.
- `-p`, `--profile` (string) - Name of the profile in `~/.databrickscfg` to use. Default: `DEFAULT` if it exists.
- `--progress-format` (format) - Format to display progress logs: `default`, `append`, `inplace`, or `json`.
- `-t`, `--target` (string) - If applicable, the bundle target to use.

### Examples
```bash
databricks quickstart
```
```

--------------------------------

### Get current user example

Source: https://learn.microsoft.com/en-us/azure/databricks/pyspark/reference/functions/current_user

Example of getting the current user with current_user() and the expected output.

```python
from pyspark.sql import functions as sf
spark.range(1).select(sf.current_user()).show()

```

```text
+--------------+
|current_user()|
+--------------+
| ruifeng.zheng|
+--------------+

```

--------------------------------

### Get workspace warehouse configuration example

Source: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/warehouses-commands

Example command that gets the workspace warehouse configuration.

```bash
databricks warehouses get-workspace-warehouse-config
```

--------------------------------

### List all system schemas for a metastore example

Source: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/system-schemas-commands

Example command that lists all system schemas for a metastore.

```bash
databricks system-schemas list my-metastore-id

```

--------------------------------

### SHOW COLUMNS examples

Source: https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/sql-ref-syntax-aux-show-columns

Example showing how to create a customer table and list its columns using SHOW COLUMNS with different schema qualifications. The output displays the column names.

```sql
-- Create `customer` table in the `salessc` schema;
> USE SCHEMA salessc;
> CREATE TABLE customer(
    cust_cd INT,
    name VARCHAR(100),
    cust_addr STRING);

-- List the columns of `customer` table in current schema.
> SHOW COLUMNS IN customer;
  col_name
 ---------
   cust_cd
      name
 cust_addr

-- List the columns of `customer` table in `salessc` schema.
> SHOW COLUMNS IN salessc.customer;
  col_name
 ---------
   cust_cd
      name
 cust_addr

-- List the columns of `customer` table in `salesdb` schema
> SHOW COLUMNS IN customer IN salessc;
  col_name
 ---------
   cust_cd
      name
 cust_addr
```

--------------------------------

### Get warehouse information example

Source: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/warehouses-commands

Example command that gets information about a warehouse with the specified ID.

```bash
databricks warehouses get 1234567890abcdef
```

--------------------------------

### Install dependencies and create local directory

Source: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/get-started

Installs the Python libraries required by the example app and creates a local directory for the app's source and configuration files. Run these commands in a terminal before creating the app.

```bash
pip install gradio
pip install pandas
mkdir gradio-hello-world
cd gradio-hello-world

```

--------------------------------

### Start a conversation example

Source: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/genie-commands

Example command to start a new conversation with space abc123 and a message.

```bash
databricks genie start-conversation abc123 "What are our top performing products this month?"
```

--------------------------------

### Example: Using window_time with window function

Source: https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/functions/window_time

Example query that uses window_time to get the inclusive end time of time windows. The query groups data by 5-minute windows and orders by window start. The output shows the inclusive end time as 0.000001 seconds before the window end.

```sql
> SELECT a, window.start as start, window.end as end, window_time(window), cnt
    FROM (SELECT a, window, count(*) as cnt
           FROM VALUES ('A1', '2021-01-01 00:00:00'),
                       ('A1', '2021-01-01 00:04:30'),
                       ('A1', '2021-01-01 00:06:00'),
                       ('A2', '2021-01-01 00:01:00') AS tab(a, b)
           GROUP by a, window(b, '5 MINUTES'))
    ORDER BY a, window.start;
  A1  2021-01-01 00:00:00  2021-01-01 00:05:00  2021-01-01 00:04:59.999999 2
  A1  2021-01-01 00:05:00  2021-01-01 00:10:00  2021-01-01 00:09:59.999999 1
  A2  2021-01-01 00:00:00  2021-01-01 00:05:00  2021-01-01 00:04:59.999999 1
```

--------------------------------

### SHOW CREDENTIALS examples

Source: https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/sql-ref-syntax-aux-show-credentials

Example output for SHOW STORAGE CREDENTIALS, SHOW SERVICE CREDENTIALS, and SHOW CREDENTIALS. The first lists storage credentials, the second lists service credentials, and the third lists all accessible credentials.

```sql
> SHOW STORAGE CREDENTIALS
  name         comment
  ------------ -----------------
  some_creds   Used to access s3

> SHOW SERVICE CREDENTIALS
  name          comment
  ------------- ----------------------
  service_creds Used to access secrets

> SHOW CREDENTIALS
  name          comment
  ------------- ----------------------
  service_creds Used to access secrets
  some_creds    Used to access s3

```

--------------------------------

### ADD JAR Examples

Source: https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/sql-ref-syntax-aux-resource-mgmt-add-jar

Examples of adding JAR files using ADD JAR and ADD JARS commands, including quoted paths and paths with spaces.

```sql
> ADD JAR /tmp/test.jar;

> ADD JAR "/path/to/some.jar";

> ADD JAR '/some/other.jar';

> ADD JARS "/path with space/abc.jar" "/path with space/def.jar";
```

--------------------------------

### Get a federation policy example

Source: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/recipient-federation-policies-commands

Example that gets the federation policy named my_policy for the recipient my_recipient.

```bash
databricks recipient-federation-policies get-federation-policy my_recipient my_policy
```

--------------------------------

### databricks quickstart command syntax

Source: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/quickstart-command

Shows the syntax for the quickstart command, including the optional [flags] placeholder. The command prints a short introduction to the Databricks CLI covering authentication, profiles, building with Declarative Automation Bundles, and next steps.

```bash
databricks quickstart [flags]
```

--------------------------------

### Get a published dashboard example

Source: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/lakeview-commands

Example that gets a published dashboard using the dashboard ID abc123.

```bash
databricks lakeview get-published abc123
```

--------------------------------

### Sync knowledge sources for an assistant example

Source: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/knowledge-assistants-commands

Example that syncs all knowledge sources for an assistant.

```bash
databricks knowledge-assistants sync-knowledge-sources knowledge-assistants/my-assistant-id

```

--------------------------------

### Example: create a Knowledge Assistant

Source: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/knowledge-assistants-commands

Example command that creates a Knowledge Assistant with display name 'Sales Assistant' and description 'Answers questions about sales data'.

```bash
databricks knowledge-assistants create-knowledge-assistant "Sales Assistant" "Answers questions about sales data"

```

--------------------------------

### Get storage credential example

Source: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/account-storage-credentials-commands

Example of getting a storage credential named my-storage-credential from metastore metastore-abc123.

```bash
databricks account storage-credentials get metastore-abc123 my-storage-credential
```
Tokens:
Raw