dstack

https://dstack.ai/docs
Admin
dstack is a unified control plane for GPU provisioning and orchestration that works with any GPU...
Tokens:285,322
Snippets:2,307
Trust Score:9.1
Update:3 months ago
Show doc for...
Tokens:
Raw
### Initialize Dev Environment with Pip Install

Source: https://dstack.ai/docs/concepts/dev-environments

Use the 'init' property to run commands when the dev environment starts. This example installs the 'wandb' package.

```yaml
type: dev-environment
name: vscode

python: "3.11"
ide: vscode

init:
  - pip install wandb
```

--------------------------------

### Dynamo Frontend Setup

Source: https://dstack.ai/docs/concepts/services

Sets up the Dynamo frontend, including installing dependencies and starting the HTTP frontend with etcd discovery. Requires Docker.

```yaml
type: service
name: dynamo-pd

env:
  - HF_TOKEN
  - MODEL_ID=zai-org/GLM-4.5-Air-FP8

replicas:
  - count: 1
    docker: true
    commands:
      - apt-get update
      - apt-get install -y python3-dev python3-venv
      - python3 -m venv ~/dyn-venv
      - source ~/dyn-venv/bin/activate
      - pip install -U pip
      - pip install "ai-dynamo[sglang]==1.1.1"
      - git clone https://github.com/ai-dynamo/dynamo.git
      # Brings up the NATS / etcd compose stack and runs the Dynamo HTTP frontend.
      - docker compose -f dynamo/dev/docker-compose.yml up -d
      - |
        python3 -m dynamo.frontend \
          --http-host 0.0.0.0 --http-port 8000 \
          --discovery-backend etcd --router-mode kv \
          --kv-cache-block-size 64
    resources:
      cpu: 4
    router:
      type: dynamo

port: 8000
model: zai-org/GLM-4.5-Air-FP8
```

--------------------------------

### Install and Run dstack Server with uv

Source: https://dstack.ai/docs/guides/server-deployment

Install the dstack server using uv and start it. This method is suitable for Linux, macOS, and Windows (via WSL 2), requiring Git and OpenSSH.

```shell
$ uv tool install 'dstack[all]' -U
$ dstack server

Applying ~/.dstack/server/config.yml...

The admin token is "bbae0f28-d3dd-4820-bf61-8f4bb40815da"
The server is running at http://127.0.0.1:3000/
```

--------------------------------

### Install and Run dstack Server with pip

Source: https://dstack.ai/docs/guides/server-deployment

Install the dstack server using pip and start it. This method is suitable for Linux, macOS, and Windows (via WSL 2), requiring Git and OpenSSH.

```shell
$ pip install "dstack[all]" -U
$ dstack server

Applying ~/.dstack/server/config.yml...

The admin token is "bbae0f28-d3dd-4820-bf61-8f4bb40815da"
The server is running at http://127.0.0.1:3000/
```

--------------------------------

### Clone dstack Examples Repository

Source: https://dstack.ai/docs/examples/inference/nim

Clone the dstack repository to access example configurations and scripts. This is a prerequisite for running the deployment examples.

```shell
$ git clone https://github.com/dstackai/dstack
$ cd dstack
```

--------------------------------

### On Gateway Apply cURL Example

Source: https://dstack.ai/docs/reference/plugins/rest

Example of how to apply policies on the gateway using a cURL command. This demonstrates the structure of the POST request and its JSON payload.

```shell
curl -X POST 'http://localhost:8000/apply_policies/on_gateway_apply' \
 -H 'Accept: application/json' \
 -H 'Content-Type: application/json' \
 -d '{ "user": "string", "project": "string", "spec": { "run_name": "string", "repo_id": "string", "repo_data": { "repo_type": "remote", "repo_name": "string", "repo_branch": "string", "repo_hash": "string", "repo_diff": "string", "repo_config_name": "string", "repo_config_email": "string" }, "repo_code_hash": "string", "repo_dir": "string", "file_archives": [], "working_dir": "string", "configuration_path": "string", "configuration": { "ide": "vscode", "version": "string", "init": [], "inactivity_duration": "off", "ports": [], "type": "dev-environment", "name": "string", "image": "string", "user": "string", "privileged": false, "entrypoint": "string", "working_dir": "string", "home_dir": "/root", "registry_auth": { "username": "string", "password": "string" }, "python": "3.9", "nvcc": true, "single_branch": true, "env": { "__root__": {} }, "shell": "string", "resources": { "cpu": { "min": 2 }, "memory": { "min": 8 }, "gpu": { "count": { "min": 0 } }, "disk": { "size": { "min": 100 } } }, "priority": 100, "volumes": [], "docker": true, "repos": [], "files": [], "setup": [], "backends": [ "amddevcloud" ], "regions": [ "string" ], "availability_zones": [ "string" ], "instance_types": [ "string" ], "reservation": "string", "spot_policy": "spot", "retry": { "on_events": [ "no-capacity" ], "duration": 0 }, "max_duration": "off", "stop_duration": "off", "max_price": 1, "creation_policy": "reuse", "idle_duration": 0, "utilization_policy": { "min_gpu_utilization": 100, "time_window": 0 }, "startup_order": "any", "stop_criteria": "all-done", "schedule": { "cron": [ "string" ] }, "fleets": [ { "project": "string", "name": "string" }, "string" ], "instances": [ { "name": "string" }, { "hostname": "string" }, { "fleet": { "project": "string", "name": "string" }, "instance": 0 }, "string" ], "tags": { "additionalProp1": "string", "additionalProp2": "string", "additionalProp3": "string" }, "backend_options": [ { "type": "vastai", "offer_order": "score", "min_reliability": 1, "min_score": 0 } ] }, "profile": { "backends": [ "amddevcloud" ], "regions": [ "string" ], "availability_zones": [ "string" ], "instance_types": [ "string" ], "reservation": "string", "spot_policy": "spot", "retry": { "on_events": [ "no-capacity" ], "duration": 0 }, "max_duration": "off", "stop_duration": "off", "max_price": 1, "creation_policy": "reuse", "idle_duration": 0, "utilization_policy": { "min_gpu_utilization": 100, "time_window": 0 }, "startup_order": "any", "stop_criteria": "all-done", "schedule": { "cron": [ "string" ] }, "fleets": [ { "project": "string", "name": "string" }, "string" ], "instances": [ { "name": "string" }, { "hostname": "string" }, { "fleet": { "project": "string", "name": "string" }, "instance": 0 }, "string" ], "tags": { "additionalProp1": "string", "additionalProp2": "string", "additionalProp3": "string" }, "backend_options": [ { "type": "vastai", "offer_order": "score", "min_reliability": 1, "min_score": 0 } ], "name": "", "default": false }, "ssh_key_pub": "string" } }' 
restart ↻
```

--------------------------------

### Schedule Dev Environment Start Time

Source: https://dstack.ai/docs/concepts/dev-environments

Specify a schedule using cron syntax to start a dev environment periodically at specific UTC times. This example starts the environment at 8:00 UTC from Monday through Friday.

```yaml
type: dev-environment
ide: vscode
schedule:
  cron: "0 8 * * mon-fri" # at 8:00 UTC from Monday through Friday
```

--------------------------------

### Install dstack CLI with uv

Source: https://dstack.ai/docs/installation

Use this command to install the dstack CLI using the 'uv' package manager. Ensure 'uv' is installed and configured.

```shell
$ uv tool install dstack -U
```

--------------------------------

### Install dstack with Fluent-bit Extras

Source: https://dstack.ai/docs/guides/server-deployment

Install the dstack Python package with the necessary extras for Fluent-bit integration. Use 'all' for all extras or 'fluentbit' specifically.

```shell
$ pip install "dstack[all]" -U
# or
$ pip install "dstack[fluentbit]" -U
```

--------------------------------

### Install dstack CLI with pip

Source: https://dstack.ai/docs/installation

Use this command to install the dstack CLI using 'pip'. Ensure 'pip' is installed and updated.

```shell
$ pip install dstack -U
```

--------------------------------

### On Volume Apply cURL Example

Source: https://dstack.ai/docs/reference/plugins/rest

Example of how to call the 'on_volume_apply' endpoint using cURL. This demonstrates the required headers and the structure of the JSON request body for applying policies to a volume.

```shell
curl -X POST 'http://localhost:8000/apply_policies/on_volume_apply' \
 -H 'Accept: application/json' \
 -H 'Content-Type: application/json' \
 -d '{ "user": "string", "project": "string", "spec": { "run_name": "string", "repo_id": "string", "repo_data": { "repo_type": "remote", "repo_name": "string", "repo_branch": "string", "repo_hash": "string", "repo_diff": "string", "repo_config_name": "string", "repo_config_email": "string" }, "repo_code_hash": "string", "repo_dir": "string", "file_archives": [], "working_dir": "string", "configuration_path": "string", "configuration": { "ide": "vscode", "version": "string", "init": [], "inactivity_duration": "off", "ports": [], "type": "dev-environment", "name": "string", "image": "string", "user": "string", "privileged": false, "entrypoint": "string", "working_dir": "string", "home_dir": "/root", "registry_auth": { "username": "string", "password": "string" }, "python": "3.9", "nvcc": true, "single_branch": true, "env": { "__root__": {} }, "shell": "string", "resources": { "cpu": { "min": 2 }, "memory": { "min": 8 }, "gpu": { "count": { "min": 0 } }, "disk": { "size": { "min": 100 } } }, "priority": 100, "volumes": [], "docker": true, "repos": [], "files": [], "setup": [], "backends": [ "amddevcloud" ], "regions": [ "string" ], "availability_zones": [ "string" ], "instance_types": [ "string" ], "reservation": "string", "spot_policy": "spot", "retry": { "on_events": [ "no-capacity" ], "duration": 0 }, "max_duration": "off", "stop_duration": "off", "max_price": 1, "creation_policy": "reuse", "idle_duration": 0, "utilization_policy": { "min_gpu_utilization": 100, "time_window": 0 }, "startup_order": "any", "stop_criteria": "all-done", "schedule": { "cron": [ "string" ] }, "fleets": [ { "project": "string", "name": "string" }, "string" ], "instances": [ { "name": "string" }, { "hostname": "string" }, { "fleet": { "project": "string", "name": "string" }, "instance": 0 }, "string" ], "tags": { "additionalProp1": "string", "additionalProp2": "string", "additionalProp3": "string" }, "backend_options": [ { "type": "vastai", "offer_order": "score", "min_reliability": 1, "min_score": 0 } ] }, "profile": { "backends": [ "amddevcloud" ], "regions": [ "string" ], "availability_zones": [ "string" ], "instance_types": [ "string" ], "reservation": "string", "spot_policy": "spot", "retry": { "on_events": [ "no-capacity" ], "duration": 0 }, "max_duration": "off", "stop_duration": "off", "max_price": 1, "creation_policy": "reuse", "idle_duration": 0, "utilization_policy": { "min_gpu_utilization": 100, "time_window": 0 }, "startup_order": "any", "stop_criteria": "all-done", "schedule": { "cron": [ "string" ] }, "fleets": [ { "project": "string", "name": "string" }, "string" ], "instances": [ { "name": "string" }, { "hostname": "string" }, { "fleet": { "project": "string", "name": "string" }, "instance": 0 }, "string" ], "tags": { "additionalProp1": "string", "additionalProp2": "string", "additionalProp3": "string" }, "backend_options": [ { "type": "vastai", "offer_order": "score", "min_reliability": 1, "min_score": 0 } ], "name": "", "default": false }, "ssh_key_pub": "string" } }' 
restart ↻
```

--------------------------------

### Kubernetes Resource Configuration Example

Source: https://dstack.ai/docs/concepts/backends

Example of a dstack .dstack.yml configuration specifying resource requests and limits for CPU, memory, disk, and GPUs. Note the specific handling of GPU ranges where only the lower limit is provisioned.

```yaml
type: dev-environment
ide: vscode

resources:
  cpu: 32..64
  memory: 1024GB
  disk: 100GB..
  gpu: nvidia:4..8

```

--------------------------------

### Example Request Body for Gateway Creation

Source: https://dstack.ai/docs/reference/plugins/rest

An example JSON payload for creating a development environment gateway, specifying user, project, and detailed configuration parameters.

```json
{
  "user": "string",
  "project": "string",
  "spec": {
    "run_name": "string",
    "repo_id": "string",
    "repo_data": {
      "repo_type": "remote",
      "repo_name": "string",
      "repo_branch": "string",
      "repo_hash": "string",
      "repo_diff": "string",
      "repo_config_name": "string",
      "repo_config_email": "string"
    },
    "repo_code_hash": "string",
    "repo_dir": "string",
    "file_archives": [],
    "working_dir": "string",
    "configuration_path": "string",
    "configuration": {
      "ide": "vscode",
      "version": "string",
      "init": [],
      "inactivity_duration": "off",
      "ports": [],
      "type": "dev-environment",
      "name": "string",
      "image": "string",
      "user": "string",
      "privileged": false,
      "entrypoint": "string",
      "working_dir": "string",
      "home_dir": "/root",
      "registry_auth": {
        "username": "string",
        "password": "string"
      },
      "python": "3.9",
      "nvcc": true,
      "single_branch": true,
      "env": {
        "__root__": {}
      },
      "shell": "string",
      "resources": {
        "cpu": {
          "min": 2
        },
        "memory": {
          "min": 8
        },
        "gpu": {
          "count": {
            "min": 0
          }
        },
        "disk": {
          "size": {
            "min": 100
          }
        }
      },
      "priority": 100,
      "volumes": [],
      "docker": true,
      "repos": [],
      "files": [],
      "setup": [],
      "backends": [
        "amddevcloud"
      ],
      "regions": [
        "string"
      ],
      "availability_zones": [
        "string"
      ],
      "instance_types": [
        "string"
      ],
      "reservation": "string",
      "spot_policy": "spot",
      "retry": {
        "on_events": [
          "no-capacity"
        ],
        "duration": 0
      },
      "max_duration": "off",
      "stop_duration": "off",
      "max_price": 1,
      "creation_policy": "reuse",
      "idle_duration": 0,
      "utilization_policy": {
        "min_gpu_utilization": 100,
        "time_window": 0
      },
      "startup_order": "any",
      "stop_criteria": "all-done",
      "schedule": {
        "cron": [
          "string"
        ]
      },
      "fleets": [
        {
          "project": "string",
          "name": "string"
        },
        "string"
      ],
      "instances": [
        {
          "name": "string"
        },
        {
          "hostname": "string"
        },
        {
          "fleet": {
            "project": "string",
            "name": "string"
          },
          "instance": 0
        },
        "string"
      ],
      "tags": {
        "additionalProp1": "string",
        "additionalProp2": "string",
        "additionalProp3": "string"
      },
      "backend_options": [
        {
          "type": "vastai",
          "offer_order": "score",
          "min_reliability": 1,
          "min_score": 0
        }
      ]
    },
    "profile": {
      "backends": [
        "amddevcloud"
      ],
      "regions": [
        "string"
      ],
      "availability_zones": [
        "string"
      ],
      "instance_types": [
        "string"
      ],
      "reservation": "string",
      "spot_policy": "spot",
      "retry": {
        "on_events": [
          "no-capacity"
        ],
        "duration": 0
      },
      "max_duration": "off",
      "stop_duration": "off",
      "max_price": 1,
      "creation_policy": "reuse",
      "idle_duration": 0,
      "utilization_policy": {
        "min_gpu_utilization": 100,
        "time_window": 0
      },
      "startup_order": "any",
      "stop_criteria": "all-done",
      "schedule": {
        "cron": [
          "string"
        ]
      },
      "fleets": [
        {

```

--------------------------------

### Specify Any GPU

Source: https://dstack.ai/docs/guides/protips

Example of specifying any available GPU without specific constraints.

```yaml
gpu: 1
```

--------------------------------

### Install dstack with uv

Source: https://dstack.ai/docs/installation

Use uv to install the dstack package with all extras. This method is suitable for Linux, macOS, and Windows (via WSL 2) and requires Git and OpenSSH.

```shell
$ uv tool install "dstack[all]" -U
$ dstack server

Applying ~/.dstack/server/config.yml...

The admin token is "bbae0f28-d3dd-4820-bf61-8f4bb40815da"
The server is running at http://127.0.0.1:3000/

```

--------------------------------

### Tenstorrent GPU Specification Examples

Source: https://dstack.ai/docs/examples/accelerators/tenstorrent

Examples of specifying GPU resources for different Tenstorrent hardware configurations. Includes Galaxy systems and PCIe cards with various chip counts and memory.

```yaml
resources:
  gpu: tt-galaxy-wh:32  # Galaxy Wormhole, 32 chips
  # gpu: tt-galaxy-bh:32  # Galaxy Blackhole, 32 chips
  # gpu: n300:4           # TT-LoudBox or TT-QuietBox Wormhole, 4 n300 cards
  # gpu: p150:4           # TT-QuietBox Blackhole, 4 p150 cards
  # gpu: p300:64GB:2      # TT-QuietBox 2 Blackhole, 2 p300 cards
  # gpu: p300:32GB:4      # TT-QuietBox 2 Blackhole, if exposed per chip
```

--------------------------------

### Python API Usage Example

Source: https://dstack.ai/docs/reference/api/python

This example demonstrates how to use the Python API to submit a task for running, attach to its logs, and manage its lifecycle. It covers task configuration, applying the configuration, and handling logs.

```APIDOC
## Python API

The Python API enables running tasks, services, and managing runs programmatically.

### Usage Example

Below is a quick example of submitting a task for running and displaying its logs.

```python
import sys

from dstack.api import Task, GPU, Client, Resources

client = Client.from_config()

task = Task(
    name="my-awesome-run",  # If not specified, a random name is assigned
    image="ghcr.io/huggingface/text-generation-inference:latest",
    env={"MODEL_ID": "TheBloke/Llama-2-13B-chat-GPTQ"},
    commands=[
        "text-generation-launcher --trust-remote-code --quantize gptq",
    ],
    ports=["80"],
    resources=Resources(gpu=GPU(memory="24GB")),
)

run = client.runs.apply_configuration(
    configuration=task, # Can also be dstack.api.Service or dstack.api.DevEnvironment
    repo=None, # Specify to mount additional files
)

run.attach() # Waits for the run to start, sets up SSH tunnel, and forwards ports

try:
    for log in run.logs():
        sys.stdout.buffer.write(log)
        sys.stdout.buffer.flush()
except KeyboardInterrupt:
    run.stop(abort=True)
finally:
    run.detach()

```

**Notes:**

1.  The `configuration` argument in the `apply_configuration` method can be either `dstack.api.Task`, `dstack.api.Service`, or `dstack.api.DevEnvironment`.
2.  When creating `dstack.api.Task`, `dstack.api.Service`, or `dstack.api.DevEnvironment`, you can specify the `image` argument. If `image` isn't specified, the default image will be used. For a private Docker registry, ensure you also pass the `registry_auth` argument.
3.  The `repo` argument in the `apply_configuration` method allows the mounting of a remote repo or a programmatically created repo. In this case, the `commands` argument can refer to the files within this repo.
4.  The `attach` method waits for the run to start and, for `dstack.api.Task` sets up an SSH tunnel and forwards configured `ports` to `localhost`.
```

--------------------------------

### Display dstack init help

Source: https://dstack.ai/docs/reference/cli/dstack/init

View available options for initializing dstack, including how to set custom Git credentials.

```shell
$ dstack init --help
```

--------------------------------

### Run a Ray cluster with RAGEN

Source: https://dstack.ai/docs/examples/training/ray-ragen

This YAML task defines a Ray cluster setup using RAGEN's Docker image. It installs RAGEN, verl, and Ray, then starts a Ray head or worker node based on the DSTACK_NODE_RANK environment variable. Ensure your fleet has 'cluster' placement and the WANDB_API_KEY environment variable is set.

```yaml
type: task
name: ray-cluster

nodes: 2

env:
- WANDB_API_KEY
image: whatcanyousee/verl:ngc-cu124-vllm0.8.5-sglang0.4.6-mcore0.12.0-te2.2
commands:
  - wget -O miniconda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
  - bash miniconda.sh -b -p /workflow/miniconda
  - eval "$(/workflow/miniconda/bin/conda shell.bash hook)"
  - git clone https://github.com/RAGEN-AI/RAGEN.git
  - cd RAGEN
  - bash scripts/setup_ragen.sh
  - conda activate ragen
  - cd verl
  - pip install --no-deps -e .
  - pip install hf_transfer hf_xet
  - pip uninstall -y ray
  - pip install -U "ray[default]"
  - |
    if [ $DSTACK_NODE_RANK = 0 ]; then 
        ray start --head --port=6379;
    else
        ray start --address=$DSTACK_MASTER_NODE_IP:6379
    fi

# Expose Ray dashboard port
ports:
  - 8265

resources:
  gpu: 80GB:8
  shm_size: 128GB

# Save checkpoints on the instance
volumes:
  - /checkpoints:/checkpoints

```

--------------------------------

### Running Axolotl Training with Dstack

Source: https://dstack.ai/docs/examples/training/axolotl

Demonstrates the command-line interface for setting environment variables and applying a dstack configuration to initiate Axolotl training. It shows the interactive prompt for confirming resource provisioning.

```shell
$ HF_TOKEN=...
$ WANDB_API_KEY=...
$ WANDB_PROJECT=...
$ HUB_MODEL_ID=...
$ dstack apply -f train.dstack.yml

 #  BACKEND              RESOURCES                     INSTANCE TYPE  PRICE
 1  vastai (cz-czechia)  cpu=64 mem=128GB H100:80GB:2  18794506       $3.8907
 2  vastai (us-texas)    cpu=52 mem=64GB  H100:80GB:2  20442365       $3.6926
 3  vastai (fr-france)   cpu=64 mem=96GB  H100:80GB:2  20379984       $3.7389

Submit the run axolotl-nvidia-llama-scout-train? [y/n]:

Provisioning...
---> 100%

```

--------------------------------

### Distributed PyTorch Training Task

Source: https://dstack.ai/docs/examples/clusters/aws

Configure a dstack task for multi-node distributed training using PyTorch and torchrun. This example clones a PyTorch examples repository and installs dependencies.

```yaml
type: task
name: train-distrib

nodes: 2

python: 3.12
env:
  - NCCL_DEBUG=INFO
commands:
  - git clone https://github.com/pytorch/examples.git pytorch-examples
  - cd pytorch-examples/distributed/ddp-tutorial-series
  - uv pip install -r requirements.txt
  - |
    torchrun \
      --nproc-per-node=$DSTACK_GPUS_PER_NODE \
      --node-rank=$DSTACK_NODE_RANK \
      --nnodes=$DSTACK_NODES_NUM \
      --master-addr=$DSTACK_MASTER_NODE_IP \
      --master-port=12345 \
      multinode.py 50 10

resources:
  gpu: 1..8
  shm_size: 16GB
```

--------------------------------

### Runpod Backend Instance Selection

Source: https://dstack.ai/docs/concepts/backends

Example output showing instance availability and pricing for Runpod backends, differentiating between Secure Cloud and Community Cloud regions.

```shell
$ dstack apply -f .dstack.yml -b runpod

 #  BACKEND  REGION    INSTANCE               SPOT  PRICE
 1  runpod   CA        NVIDIA A100 80GB PCIe  yes   $0.6
 2  runpod   CA-MTL-3  NVIDIA A100 80GB PCIe  yes   $0.82
```

--------------------------------

### Get Prometheus Metrics Request

Source: https://dstack.ai/docs/reference/http/metrics

Example cURL command to fetch Prometheus metrics from the /metrics endpoint. Requires an authorization token.

```bash
curl -X GET 'https://sky.dstack.ai/metrics' \
 -H 'Authorization: Bearer {user token}' \
 -H 'Accept: application/json'
```

--------------------------------

### Create a Backend Fleet Configuration

Source: https://dstack.ai/docs/concepts/fleets

Define a backend fleet using a YAML file. This example shows how to set the fleet name, the number of nodes, idle duration, and resource limits for GPUs.

```yaml
type: fleet
name: my-fleet

# Allow to provision of up to 2 instances
nodes: 0..2

# Uncomment to ensure instances are inter-connected
#placement: cluster

# Deprovision instances above the minimum if they remain idle
idle_duration: 1h

resources:
  # Allow to provision up to 8 GPUs
  gpu: 0..8

```

--------------------------------

### Run a Streamlit Application Task

Source: https://dstack.ai/docs/guides/protips

This YAML defines a dstack task to run a Streamlit application. It specifies the Python version, installs Streamlit, and starts the application.

```yaml
type: task
name: streamlit-task

python: 3.12

commands:
  - uv pip install streamlit
  - streamlit hello
ports: 
  - 8501


```

--------------------------------

### Inspect Available Instance Offers

Source: https://dstack.ai/docs/guides/troubleshooting

Use 'dstack offer' to view available instances. You can filter by specific GPU types or by a particular fleet configuration.

```shell
# All matching offers, ignoring fleet configurations
$ dstack offer --gpu H100

# Offers available through a specific fleet
$ dstack offer --gpu H100 --fleet my-fleet
```

--------------------------------

### Enable NVCC for CUDA Kernel Compilation

Source: https://dstack.ai/docs/concepts/dev-environments

Set the 'nvcc' property to true if you need to build custom CUDA kernels. This example also includes installing 'flash_attn'.

```yaml
type: dev-environment
name: vscode

python: 3.12
nvcc: true

ide: vscode
init:
  - uv pip install flash_attn --no-build-isolation
```

--------------------------------

### Get Nebius Kubernetes Credentials

Source: https://dstack.ai/docs/examples/clusters/nebius

Use this command to retrieve the kubeconfig file for your Nebius Kubernetes cluster. Ensure you have the Nebius CLI installed and replace `<cluster id>` with your actual cluster ID.

```shell
$ nebius mk8s cluster get-credentials --id <cluster id> --external

```

--------------------------------

### Private Docker Registry Authentication

Source: https://dstack.ai/docs/concepts/tasks

Configures a task to pull an image from a private Docker registry using provided username and API key. Installs PyTorch examples and runs a distributed training script.

```yaml
type: task
name: train

env:
  - NGC_API_KEY

image: nvcr.io/nvidia/pytorch:25.05-py3
registry_auth:
  username: $oauthtoken
  password: ${{ env.NGC_API_KEY }}

commands:
  - git clone https://github.com/pytorch/examples.git pytorch-examples
  - cd pytorch-examples/distributed/ddp-tutorial-series
  - pip install -r requirements.txt
  - |
    torchrun \
      --nproc-per-node=$DSTACK_GPUS_PER_NODE \
      --nnodes=$DSTACK_NODES_NUM \
      multinode.py 50 10

resources:
  gpu: H100:1..2
  shm_size: 24GB
```

--------------------------------

### dstack CLI Configuration Example

Source: https://dstack.ai/docs/guides/cli-api

Configure dstack projects by specifying project name, server URL, and user token in ~/.dstack/config.yml. This file manages multiple project configurations.

```yaml
projects:
  - name: main
    url: http://127.0.0.1:3000
    token: <user token>
    default: true
  - name: octocat
    url: https://sky.dstack.ai
    token: <user token>
```

--------------------------------

### Configure Distributed Task with Torchrun

Source: https://dstack.ai/docs/concepts/tasks

Set up a task to run distributed training across multiple nodes using torchrun. This example demonstrates installing dependencies and launching a PyTorch distributed training script.

```yaml
type: task
name: train-distrib

nodes: 2

python: 3.12
env:
  - NCCL_DEBUG=INFO
commands:
  - git clone https://github.com/pytorch/examples.git pytorch-examples
  - cd pytorch-examples/distributed/ddp-tutorial-series
  - uv pip install -r requirements.txt
  - |
    torchrun \
      --nproc-per-node=$DSTACK_GPUS_PER_NODE \
      --node-rank=$DSTACK_NODE_RANK \
      --nnodes=$DSTACK_NODES_NUM \
      --master-addr=$DSTACK_MASTER_NODE_IP \
      --master-port=12345 \
      multinode.py 50 10

resources:
  gpu: 24GB:1..2
  shm_size: 24GB
```

--------------------------------

### Initialize Plugin Package

Source: https://dstack.ai/docs/reference/plugins/python

Use the 'uv init --library' command to initialize a new Python package for your dstack plugin.

```bash
uv init --library
```

--------------------------------

### Display help for dstack offer

Source: https://dstack.ai/docs/reference/cli/dstack/offer

Use the --help flag to view all available options and arguments for the `dstack offer` command.

```shell
dstack offer --help
```

--------------------------------

### TRL Single-Node Training Configuration

Source: https://dstack.ai/docs/examples/training/trl

Defines a dstack task for fine-tuning Llama-3.1-8B using TRL, SFT, and QLoRA. Ensure necessary environment variables and dependencies are installed. This configuration is suitable for single-GPU or multi-GPU setups on one node.

```yaml
type: task
name: trl-train

python: 3.12
# Ensure nvcc is installed (req. for Flash Attention) 
nvcc: true

env:
  - HF_TOKEN
  - WANDB_API_KEY
  - HUB_MODEL_ID
commands:
  # Pin torch==2.6.0 to avoid building Flash Attention from source.
  # Prebuilt Flash Attention wheels are not available for the latest torch==2.7.0.
  - uv pip install torch==2.6.0
  - uv pip install transformers bitsandbytes peft wandb
  - uv pip install flash_attn --no-build-isolation
  - git clone https://github.com/huggingface/trl
  - cd trl
  - uv pip install .
  - |
    accelerate launch \
      --config_file=examples/accelerate_configs/multi_gpu.yaml \
      --num_processes $DSTACK_GPUS_PER_NODE \
      trl/scripts/sft.py \
      --model_name meta-llama/Meta-Llama-3.1-8B \
      --dataset_name OpenAssistant/oasst_top1_2023-08-25 \
      --dataset_text_field="text" \
      --per_device_train_batch_size 1 \
      --per_device_eval_batch_size 1 \
      --gradient_accumulation_steps 4 \
      --learning_rate 2e-4 \
      --report_to wandb \
      --bf16 \
      --max_seq_length 1024 \
      --lora_r 16 \
      --lora_alpha 32 \
      --lora_target_modules q_proj k_proj v_proj o_proj \
      --load_in_4bit \
      --use_peft \
      --attn_implementation "flash_attention_2" \
      --logging_steps=10 \
      --output_dir models/llama31 \
      --hub_model_id peterschmidt85/FineLlama-3.1-8B

resources:
  gpu:
    # 24GB or more VRAM
    memory: 24GB..
    # One or more GPU
    count: 1..
  # Shared memory (for multi-gpu)
  shm_size: 24GB

```

--------------------------------

### List Available Offers using dstack CLI

Source: https://dstack.ai/docs/guides/cli-api

Use 'dstack offer' to view available instance configurations (offers) that match resource requirements. Specify GPU and max offers for filtering.

```shell
$ dstack offer --gpu H100 --max-offers 10
```

--------------------------------

### Interact with Deployed Service via cURL

Source: https://dstack.ai/docs/examples/inference/dynamo

This cURL command demonstrates how to send a POST request to the deployed service endpoint to get chat completions. Replace placeholders like `<dstack server URL>` and `<user token>` with your actual values. This example assumes no gateway is configured.

```shell
curl http://127.0.0.1:3000/proxy/services/main/dynamo-pd/v1/chat/completions \
    -X POST \
    -H 'Authorization: Bearer <user token>'
    -H 'Content-Type: application/json' \
    -d '{
      "model": "zai-org/GLM-4.5-Air-FP8",
      "messages": [
        {
          "role": "user",
          "content": "What is prefill-decode disaggregation?"
        }
      ],
      "max_tokens": 1024
    }'
```

--------------------------------

### Task Configuration for Optimum TPU Llama Fine-tuning

Source: https://dstack.ai/docs/examples/accelerators/tpu

This dstack configuration defines a task for fine-tuning Llama 3.1 8B using Optimum TPU. It includes environment setup, file inclusions, and commands for cloning the Optimum TPU repository, installing dependencies, and running the training script.

```yaml
type: task
name: optimum-tpu-llama-train

python: "3.11"
env:
  - HF_TOKEN
files:
  - train.py
  - config.yaml
commands:
  - git clone -b add_llama_31_support https://github.com/dstackai/optimum-tpu.git
  - mkdir -p optimum-tpu/examples/custom/
  - cp train.py optimum-tpu/examples/custom/train.py
  - cp config.yaml optimum-tpu/examples/custom/config.yaml
  - cd optimum-tpu
  - pip install -e . -f https://storage.googleapis.com/libtpu-releases/index.html
  - pip install datasets evaluate
  - pip install accelerate -U
  - pip install peft
  - python examples/custom/train.py examples/custom/config.yaml

resources:
  gpu: v5litepod-8

```

--------------------------------

### Deploy Qwen2.5-72B-Instruct on AMD MI300X with PD Disaggregation

Source: https://dstack.ai/docs/concepts/services

This configuration deploys a large language model on a multi-node cluster with AMD MI300X GPUs, utilizing PD disaggregation for efficient model serving. It specifies the container image, environment variables for model configuration and disaggregation, and resource requirements for router, prefill, and decode replicas.

```yaml
type: service
name: amd-sglang-pd-service

image: rocm/sgl-dev:v0.5.10.post1-rocm720-mi30x-20260427
privileged: true

env:
  - MODEL_ID=Qwen/Qwen2.5-72B-Instruct
  - HF_TOKEN
  - SGLANG_USE_AITER=0
  - SGLANG_ROCM_FUSED_DECODE_MLA=0
  - SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
  - SGLANG_DISAGGREGATION_WAITING_TIMEOUT=600
  - RDMA_DEVICES=bnxt_re0,bnxt_re1,bnxt_re2,bnxt_re3,bnxt_re4,bnxt_re5,bnxt_re6,bnxt_re7
  - NCCL_IB_DISABLE=1

replicas:
  - count: 1
    commands:
      - pip install smg
      - |
        smg launch \
          --pd-disaggregation \
          --host 0.0.0.0 \
          --port 30000
    resources:
      cpu: 4..
    router:
      type: sglang

  - count: 1..
    scaling:
      metric: rps
      target: 300
    commands:
      - |
        python3 -m sglang.launch_server \
          --model $MODEL_ID \
          --disaggregation-mode prefill \
          --disaggregation-transfer-backend mooncake \
          --host 0.0.0.0 \
          --port 30000 \
          --tp $DSTACK_GPUS_NUM \
          --trust-remote-code \
          --disaggregation-ib-device $RDMA_DEVICES \
          --disaggregation-bootstrap-port 8998 \
          --disable-radix-cache \
          --disable-cuda-graph \
          --disable-overlap-schedule \
          --mem-fraction-static 0.8 \
          --max-running-requests 1024
    resources:
      gpu: MI300X:8
      cpu: 96..
      memory: 512GB..

  - count: 1..4
    scaling:
      metric: rps
      target: 300
    commands:
      - |
        python3 -m sglang.launch_server \
          --model $MODEL_ID \
          --disaggregation-mode decode \
          --disaggregation-transfer-backend mooncake \
          --host 0.0.0.0 \
          --port 30000 \
          --tp $DSTACK_GPUS_NUM \
          --trust-remote-code \
          --disaggregation-ib-device $RDMA_DEVICES \
          --disable-radix-cache \
          --disable-cuda-graph \
          --disable-overlap-schedule \
          --decode-attention-backend triton \
          --mem-fraction-static 0.8 \
          --max-running-requests 1024
    resources:
      gpu: MI300X:8
      cpu: 96..
      memory: 512GB..

port: 30000
model: Qwen/Qwen2.5-72B-Instruct

# Custom probe is required for PD disaggregation.
probes:
  - type: http
    url: /health
    interval: 15s

volumes:
  - /usr/lib64/libibverbs/libbnxt_re-rdmav34.so:/usr/lib/x86_64-linux-gnu/libibverbs/libbnxt_re-rdmav34.so

```

--------------------------------

### Apply Configuration with Repo

Source: https://dstack.ai/docs/reference/api/python

Applies a configuration to a run, specifying the initialized repository. This is done after initializing the repo.

```python
run = client.runs.apply_configuration(
    configuration=...,
    repo=repo,
)

```

--------------------------------

### Initialize Repository

Source: https://dstack.ai/docs/reference/api/python

Initializes a repository and configures its credentials. It can use default Git credentials or allow overriding via SSH key or OAuth token.

```python
repo = RemoteRepo.from_url(
    repo_url="https://github.com/dstackai/dstack-examples",
    repo_branch="main",
)
client.repos.init(repo)

```

--------------------------------

### Create and Configure a Virtual Repository

Source: https://dstack.ai/docs/reference/api/python

Use `VirtualRepo` to programmatically create a repository and add files from a package. This is useful for dynamically building repository content before applying it to a run.

```python
virtual_repo = VirtualRepo(repo_id="some-unique-repo-id")
virtual_repo.add_file_from_package(package=some_package, path="requirements.txt")
virtual_repo.add_file_from_package(package=some_package, path="train.py")

run = client.runs.apply_configuration(
    configuration=...,
    repo=virtual_repo,
)

```

--------------------------------

### Setup Commands

Source: https://dstack.ai/docs/reference/plugins/rest

An array of strings representing setup commands to be executed within the container.

```json
        "setup": {
          "type": "array",
          "items": {
            "type": "string"
          },
          "default": []
        }
```

--------------------------------

### Bad Request Response Example

Source: https://dstack.ai/docs/reference/http/metrics

Example of a bad request response, indicating an error with a specific code and message.

```json
{
  "detail": {
    "code": "error",
    "msg": "string"
  }
}
```

--------------------------------

### Get Upstream API Sshproxy Get Upstream Post

Source: https://dstack.ai/docs/reference/http/openapi.json

Retrieves upstream information. Requires a GetUpstreamRequestRequest in the body.

```APIDOC
## POST /api/sshproxy/get_upstream

### Description
Retrieves upstream information.

### Method
POST

### Endpoint
/api/sshproxy/get_upstream

#### Request Body
- **GetUpstreamRequestRequest** (object) - Required - The request body for getting upstream information.

### Response
#### Success Response (200)
- **GetUpstreamResponseRequest** (object) - The upstream response information.
```

--------------------------------

### Get dstack resource information in JSON format

Source: https://dstack.ai/docs/guides/cli-api

Use the `--json` flag with commands like `dstack ps`, `dstack run get`, or `dstack fleet get` to retrieve detailed information in JSON format, suitable for scripting and automation.

```shell
dstack ps --json
```

```shell
dstack run get <run name> --json
```

```shell
dstack fleet get <fleet name> --json
```

--------------------------------

### Install Ray locally

Source: https://dstack.ai/docs/examples/training/ray-ragen

Install the Ray client library locally to interact with a remote Ray cluster. This command is necessary before submitting jobs.

```shell
pip install ray

```

--------------------------------

### Configure Dev Environment Resources

Source: https://dstack.ai/docs/concepts/dev-environments

Specify resource requirements like CPU, memory, GPU, and disk size. Ranges can be used for CPU, memory, and GPU. This example requests at least 16 CPU cores, 200GB RAM, and 2-4 GPUs with 40-80GB memory each.

```yaml
type: dev-environment
# The name is optional, if not specified, generated randomly
name: vscode    

ide: vscode

resources:
  # 16 or more x86_64 cores
  cpu: 16..
  # 200GB or more RAM
  memory: 200GB..
  # 4 GPUs from 40GB to 80GB
  gpu: 40GB..80GB:4
  # Shared memory (required by multi-gpu)
  shm_size: 16GB
  # Disk size
  disk: 500GB
```

--------------------------------

### REST Plugin OpenAPI Spec Example

Source: https://dstack.ai/docs/reference/plugins/rest

This is an example of the OpenAPI specification for the REST plugin, illustrating the structure of various configuration options and parameters.

```json
{
  "project": "string",
  "name": "string"
}

```

```json
{
  "name": "string"
}

```

```json
{
  "hostname": "string"
}

```

```json
{
  "fleet": {
    "project": "string",
    "name": "string"
  },
  "instance": 0
}

```

```json
"string"

```

```json
{
  "additionalProp1": "string",
  "additionalProp2": "string",
  "additionalProp3": "string"
}

```

```json
{
  "type": "vastai",
  "offer_order": "score",
  "min_reliability": 1,
  "min_score": 0
}

```

```json
{
  "ssh_key_pub": "string"
}

```

--------------------------------

### Display dstack server help

Source: https://dstack.ai/docs/reference/cli/dstack/server

Use the --help flag to view all available options for the dstack server command.

```shell
$ dstack server --help
```
Tokens:
Raw