### Install plugin server with uv pip

Source: https://github.com/dstackai/dstack/blob/master/examples/plugins/example_plugin_server/README.md

Installs the example plugin server package from the local examples directory.

```bash
uv pip install examples/plugins/example_plugin_server
```

--------------------------------

### Install dstack with Nebius support and start server

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/blog/posts/nebius.md

Install the dstack CLI with Nebius support and start the dstack server. Run these commands after configuring the backend.

```shell
$ pip install "dstack[nebius]"
$ dstack server
```

--------------------------------

### Install and launch dstack server

Source: https://github.com/dstackai/dstack/blob/master/README.md

Installs the dstack server and starts it. The output shows the server applying its configuration, printing the admin token, and reporting the server URL. Requires Git and OpenSSH.

```shell
$ uv tool install "dstack[all]" -U
$ dstack server

Applying ~/.dstack/server/config.yml...

The admin token is "bbae0f28-d3dd-4820-bf61-8f4bb40815da"
The server is running at http://127.0.0.1:3000/

```

--------------------------------

### Install and run dstack server via uv

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/docs/guides/server-deployment.md

Installs the dstack tool using uv and starts the server. Requires Git and OpenSSH.

```shell
$ uv tool install 'dstack[all]' -U
$ dstack server

Applying ~/.dstack/server/config.yml...

The admin token is "bbae0f28-d3dd-4820-bf61-8f4bb40815da"
The server is running at http://127.0.0.1:3000/
```

--------------------------------

### Install dstack CLI with uv

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/blog/posts/nebius-in-dstack-sky.md

Install the dstack CLI using uv. This is one of two installation options presented in the getting started section.

```shell
$ uv tool install dstack -U
```

--------------------------------

### Configure AMD Developer Cloud backend in ~/.dstack/server/config.yml and start dstack server

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/blog/posts/digitalocean-and-amd-dev-cloud.md

Create the main project backend with type: amddevcloud and an API key in ~/.dstack/server/config.yml. For DigitalOcean, change type to digitalocean. Then install dstack[server] and run dstack server.

```yaml
projects:
- name: main
  backends:
    - type: amddevcloud
      project_name: my-amd-project
      creds:
        type: api_key
        api_key: ...

```

```shell
$ pip install "dstack[server]"
$ dstack server

```

--------------------------------

### Configure init commands in dev-environment

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/docs/concepts/dev-environments.md

Use the init property to run commands at startup, such as installing packages. The example installs wandb with pip.

```yaml
type: dev-environment
name: vscode

python: "3.11"
ide: vscode

init:
  - pip install wandb

```

--------------------------------

### Install dstack Fluent-bit extras

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/docs/guides/server-deployment.md

Install the `fluentbit` extras to use Fluent-bit log storage. The `dstack[all]` install includes all extras; the `dstack[fluentbit]` install includes only the Fluent-bit extras.

```shell
$ pip install "dstack[all]" -U
# or
$ pip install "dstack[fluentbit]" -U
```

--------------------------------

### Install dstack with uv sync --all-extras

Source: https://github.com/dstackai/dstack/blob/master/contributing/DEVELOPMENT.md

Installs dstack with all extras and dev dependencies into the project's .venv in editable mode. Run with `uv run dstack` to use it. Alternatively, use `uv sync --all-extras --active` to install into the currently activated virtual environment.

```shell
uv sync --all-extras
```

--------------------------------

### Set up accounts and usage quotas with sacctmgr

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/docs/guides/migration/slurm.md

Creates an account named research, assigns user1 to it, and sets usage quotas: user1 gets 100 hours and 4 GPUs, the account gets 1000 hours and 16 GPUs. Run as administrator.

```shell
$ sacctmgr add account research
$ sacctmgr modify user user1 set account=research
$ sacctmgr modify user user1 set MaxWall=100:00:00 MaxTRES=gres/gpu=4
$ sacctmgr modify account research set MaxWall=1000:00:00 MaxTRES=gres/gpu=16

```

--------------------------------

### Access logs on compute nodes with srun

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/docs/guides/migration/slurm.md

If logs are on compute nodes, find the node from `scontrol show job`, then access via `srun --jobid` (running jobs) or SSH (completed jobs). This example shows using `srun` to get a shell on the node and then viewing the log file.

```shell
$ srun --jobid=12345 --nodelist=gpu-node01 --pty bash
$ cat slurm-12345.out
```

--------------------------------

### Start dstack dev-environment with dstack apply

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/docs/guides/migration/slurm.md

Starts the dev environment defined in dev.dstack.yml. After confirmation, dstack provisions the instance and prints a VS Code Desktop link for remote access.

```shell
$ dstack apply -f dev.dstack.yml
  BACKEND  REGION    RESOURCES                SPOT  PRICE
  1  runpod   CA-MTL-1  9xCPU, 48GB, A5000:24GB  yes   $0.11
  Submit the run ml-dev? [y/n]: y
  Launching `ml-dev`...
  ---> 100%
  To open in VS Code Desktop, use this link:
    vscode://vscode-remote/ssh-remote+ml-dev/workflow
```

--------------------------------

### Slurm MPI distributed training script

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/docs/guides/migration/slurm.md

Slurm batch script for MPI-based distributed training, using mpirun with a generated hostfile. It requests 2 nodes with 16 tasks and 8 GPUs per node, sets up the master address and port, and runs an NCCL test binary (all_reduce_perf) as an example MPI application. The temporary hostfile is removed after execution.

```bash
#!/bin/bash
#SBATCH --nodes=2
#SBATCH --ntasks=16
#SBATCH --gres=gpu:8
#SBATCH --mem=200G
#SBATCH --time=24:00:00

export MASTER_ADDR=$(scontrol show hostnames $SLURM_NODELIST | head -n1)
export MASTER_PORT=12345

# Convert SLURM_JOB_NODELIST to hostfile format
HOSTFILE=$(mktemp)
scontrol show hostnames $SLURM_JOB_NODELIST | awk -v slots=$SLURM_NTASKS_PER_NODE '{print $0" slots="slots}' > $HOSTFILE

# MPI with NCCL tests or custom MPI application
mpirun \
--allow-run-as-root \
--hostfile $HOSTFILE \
-n $SLURM_NTASKS \
--bind-to none \
/opt/nccl-tests/build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

rm -f $HOSTFILE
```

--------------------------------

### Check queue status with squeue

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/docs/guides/migration/slurm.md

Shows the queue status for the current user. Jobs are listed in scheduling order by priority; the example output displays a pending job with reason 'Priority'.

```shell
$ squeue -u $USER
  JOBID PARTITION     NAME     USER ST  TIME  NODES REASON
  12345     gpu    training   user1 PD  0:00      2 Priority

```

--------------------------------

### Check quota usage with sacct

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/docs/guides/migration/slurm.md

Reports quota consumption per user, account, or QOS for a date range. The example shows total CPU time and GPU usage for user1 in the research account.

```shell
$ sacct -S 2024-01-01 -E 2024-01-31 --format=User,Account,TotalCPU,TotalTRES
  User     Account   TotalCPU  TotalTRES
  user1    research  100:00:00 gres/gpu=50

```

--------------------------------

### Build and preview combined site: just site-build and just site-serve

Source: https://github.com/dstackai/dstack/blob/master/contributing/DOCS.md

Builds the combined site by building the landing page and the MkDocs docs/blog, then overlays them into ./site. The second command previews the combined site on http://127.0.0.1:8001. In the combined build, the React index.html owns /, while MkDocs serves /docs, /blog, and /assets.

```shell
just site-build             # website/dist + `mkdocs build` -> ./site (scripts/docs/build_site.sh)
```

```shell
just site-serve             # preview the combined site on http://127.0.0.1:8001
```

--------------------------------

### Generate a new backend with scripts/add_backend.py

Source: https://github.com/dstackai/dstack/blob/master/contributing/BACKENDS.md

Run this command from the repository root to generate the backend files and classes for a new backend named ExampleXYZ. It creates an examplexyz backend directory under src/dstack/_internal/core/backends with backend.py, compute.py, configurator.py, and models.py.

```shell
python scripts/add_backend.py -n ExampleXYZ
```

--------------------------------

### Start plugin server with python -m

Source: https://github.com/dstackai/dstack/blob/master/examples/plugins/example_plugin_server/README.md

Runs the plugin server main module.

```bash
python -m example_plugin_server.main
```

--------------------------------

### Run JupyterLab as a dstack task

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/docs/guides/protips.md

Run JupyterLab as a dstack task. Install JupyterLab and start it with root access, exposing port 8888.

```yaml
type: task

commands:
    - pip install jupyterlab
    - jupyter lab --allow-root

ports:
    - 8888

```

--------------------------------

### dstack init --help

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/docs/reference/cli/dstack/init.md

Displays help for the dstack init command, which is used to provide custom Git credentials for private repositories. Run it in the repo's directory or pass the repo path or URL with --repo (or -P).

```shell
$ dstack init --help
#GENERATE#
```

--------------------------------

### dstack service configuration for Docker-in-Docker

Source: https://github.com/dstackai/dstack/blob/master/docker/dind/README.md

Use this configuration to run a Docker-in-Docker service with dstack. It starts the Docker daemon, lists stored images, runs an NVIDIA GPU example with nvidia-smi, and launches a Docker Compose example (Gitea and PostgreSQL). The volume preserves the Docker data root between runs.

```yaml
type: service

name: dind

image: dstackai/dind
privileged: true

port: 3000
auth: false

commands:
  # start docker daemon
  - start-dockerd
  # list stored images
  - docker image ls
  # run docker with nvidia gpu example (nvidia-smi)
  - docker run --rm --gpus all debian nvidia-smi
  # run docker compose example (gitea+postgres)
  - git clone --depth 1 https://github.com/docker/awesome-compose.git
  - cd awesome-compose/gitea-postgres
  - docker compose up

# preserve docker data root between runs (including volumes and image store)
volumes:
  - name: dind-volume
    path: /var/lib/docker
```

--------------------------------

### Apply fleet configuration with dstack apply

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/docs/examples/accelerators/tenstorrent.md

Run dstack apply -f tt-fleet.dstack.yml to create the fleet. The output shows the fleet status, resources, and price.

```bash
$ dstack apply -f tt-fleet.dstack.yml

 FLEET     RESOURCES                                                          PRICE  STATUS  CREATED
 tt-fleet  cpu=64 mem=566.1GB disk=749.6GB gpu=tt-galaxy-wh:12GB:32           $0     idle    18 sec ago
```

--------------------------------

### Run Streamlit app as a dstack task

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/docs/guides/protips.md

Define a dstack task that runs a Streamlit app. Install Streamlit with uv and start the hello app, exposing port 8501.

```yaml
type: task
name: streamlit-task

python: 3.12

commands:
  - uv pip install streamlit
  - streamlit hello
ports: 
  - 8501

```

--------------------------------

### Enable nvcc for custom CUDA kernels

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/docs/concepts/dev-environments.md

Enable nvcc (CUDA compiler) by setting nvcc to true. The example also installs flash_attn without build isolation via uv pip.

```yaml
type: dev-environment
name: vscode

python: 3.12
nvcc: true

ide: vscode
init:
  - uv pip install flash_attn --no-build-isolation

```

--------------------------------

### Apply backend fleet (dstack apply -f fleet.dstack.yml)

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/docs/quickstart.md

Runs 'dstack apply -f fleet.dstack.yml' to create the backend fleet, showing backend, region, resources, spot, and price options. The command prompts 'Create the fleet?' and then displays the fleet table. Note the nodes range starts at 0, so only a template is created until runs are submitted.

```shell
$ dstack apply -f fleet.dstack.yml
    
  #  BACKEND  REGION           RESOURCES                 SPOT  PRICE
  1  gcp      us-west4         2xCPU, 8GB, 100GB (disk)  yes   $0.010052
  2  azure    westeurope       2xCPU, 8GB, 100GB (disk)  yes   $0.0132
  3  gcp      europe-central2  2xCPU, 8GB, 100GB (disk)  yes   $0.013248

Create the fleet? [y/n]: y

  FLEET    INSTANCE  BACKEND  RESOURCES  PRICE  STATUS  CREATED 
  default  -         -        -          -      -       10:36

```

--------------------------------

### Run dstack server with Docker

Source: https://github.com/dstackai/dstack/blob/master/docker/server/README.md

Starts the dstack server in a Docker container, mapping port 3000 and mounting the server config directory. The output shows the server URL and an example admin token.

```shell
docker run -p 3000:3000 -v $HOME/.dstack/server/:/root/.dstack/server dstackai/dstack

The dstack server is running at http://0.0.0.0:3000
The admin user token is 'bbae0f28-d3dd-4820-bf61-8f4bb40815da'
```

--------------------------------

### Apply gpt-oss-120b service with dstack apply -f examples/models/gpt-oss/120b.dstack.yml

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/blog/posts/digitalocean-and-amd-dev-cloud.md

Service configuration for gpt-oss-120b served with vLLM on 8x AMD MI300X GPUs, with AITER enabled and a Hugging Face cache volume. The block is labeled examples/modes/gpt-oss/120b.dstack.yml; apply it with `dstack apply -f examples/models/gpt-oss/120b.dstack.yml`. The output lists the amddevcloud backend, resources, price, and prompts to submit the run.

```yaml
type: service
name: gpt-oss-120b

model: openai/gpt-oss-120b

env:
  - HF_TOKEN
  - MODEL=openai/gpt-oss-120b
  # To enable AITER, set below to 1. Otherwise, set it to 0.
  - VLLM_ROCM_USE_AITER=1
  # To enable AITER Triton unified attention
  - VLLM_USE_AITER_UNIFIED_ATTENTION=1
  # below is required in order to enable AITER unified attention by disabling AITER MHA
  - VLLM_ROCM_USE_AITER_MHA=0
image: rocm/vllm-dev:open-mi300-08052025
commands:
  - |
    vllm serve $MODEL \
      --tensor-parallel $DSTACK_GPUS_NUM \
      --no-enable-prefix-caching \
      --disable-log-requests \
      --compilation-config '{"full_cuda_graph": true}'
port: 8000

volumes:
  # Cache downloaded models
  - /root/.cache/huggingface:/root/.cache/huggingface

resources:
  gpu: MI300X:8
  shm_size: 32GB

```

```shell
$ dstack apply -f examples/models/gpt-oss/120b.dstack.yml

 #  BACKEND             RESOURCES                                   PRICE   
 1  amddevcloud (alt1)  cpu=20 mem=240GB disk=720GB MI300X:192GB:8  $15.92
 
 Submit the run? [y/n]:

```

--------------------------------

### Install Ray and submit a Ray job to a cluster

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/docs/examples/training/ray-ragen.md

Install Ray locally before submitting jobs. Then submit a training job to a Ray cluster at localhost:8265, setting RAY_ADDRESS and using ray job submit with a bash command that runs the training script with the specified parameters.

```shell
$ pip install ray
```

```shell
$ RAY_ADDRESS=http://localhost:8265
$ ray job submit \
  -- bash -c "\
    export PYTHONPATH=/workflow/RAGEN; \
    cd /workflow/RAGEN; \
    /workflow/miniconda/envs/ragen/bin/python train.py \
      --config-name base \
      system.CUDA_VISIBLE_DEVICES=[0,1,2,3,4,5,6,7] \
      model_path=Qwen/Qwen2.5-7B-Instruct \
      trainer.experiment_name=agent-fine-tuning-Qwen2.5-7B \
      trainer.n_gpus_per_node=8 \
      trainer.nnodes=2 \
      micro_batch_size_per_gpu=2 \
      trainer.default_local_dir=/checkpoints \
      trainer.save_freq=50 \
      actor_rollout_ref.rollout.tp_size_check=False \
      actor_rollout_ref.rollout.tensor_model_parallel_size=4"
```

--------------------------------

### SGLang PD service on AMD (amd-pd.dstack.yml)

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/docs/examples/inference/sglang.md

Starts an AMD SGLang PD service configuration for multi-node clusters with AMD MI300X GPUs. The example is incomplete in the source; only the service type, name, image, and privileged flag are shown.

```yaml
type: service
name: amd-sglang-pd-service

image: rocm/sgl-dev:v0.5.10.post1-rocm720-mi30x-20260427
privileged: true
```

--------------------------------

### Apply dev environment with dstack apply and view Cursor link

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/blog/posts/cursor.md

Run dstack apply -f examples/.dstack.yml to provision the dev environment. The output shows available instances and prompts for confirmation. After submission, a cursor:// link is provided to open the environment in Cursor.

```shell
$ dstack apply -f examples/.dstack.yml

 #  BACKEND  REGION    RESOURCES                SPOT  PRICE
 1  runpod   CA-MTL-1  9xCPU, 48GB, A5000:24GB  yes   $0.11
 2  runpod   EU-SE-1   9xCPU, 43GB, A5000:24GB  yes   $0.11
 3  gcp      us-west4  4xCPU, 16GB, L4:24GB     yes   $0.21

Submit the run vscode? [y/n]: y

Launching `vscode`...
---> 100%

To open in Cursor, use this link:
  cursor://vscode-remote/ssh-remote+vscode/workflow

```

--------------------------------

### Distributed PyTorch Training with torchrun in train-distrib.dstack.yml

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/docs/examples/clusters/aws.md

Runs multi-node PyTorch training using torchrun with NCCL debug settings. Clones the examples repo, installs requirements, and launches multinode.py with 50 epochs and 10 workers. Ensure the image is EFA-compatible.

```yaml
type: task
name: train-distrib

nodes: 2

python: 3.12
env:
  - NCCL_DEBUG=INFO
commands:
  - git clone https://github.com/pytorch/examples.git pytorch-examples
  - cd pytorch-examples/distributed/ddp-tutorial-series
  - uv pip install -r requirements.txt
  - |
    torchrun \
      --nproc-per-node=$DSTACK_GPUS_PER_NODE \
      --node-rank=$DSTACK_NODE_RANK \
      --nnodes=$DSTACK_NODES_NUM \
      --master-addr=$DSTACK_MASTER_NODE_IP \
      --master-port=12345 \
      multinode.py 50 10

resources:
  gpu: 1..8
  shm_size: 16GB
```

--------------------------------

### Apply service configuration with dstack apply

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/docs/concepts/services.md

Runs the service by passing the configuration file to dstack apply. The command prompts for confirmation, provisions instances, and prints the service and model endpoints. The service is published at http://localhost:3000/proxy/services/main/qwen36/ and the model at http://localhost:3000/proxy/models/main/.

```shell
$ dstack apply -f .dstack.yml

Submit the run qwen36? [y/n]: y

Provisioning...
---> 100%

Service is published at:
  http://localhost:3000/proxy/services/main/qwen36/
Model Qwen/Qwen3.6-27B is published at:
  http://localhost:3000/proxy/models/main/
```

--------------------------------

### Launch Ray cluster with dstack task (cluster.dstack.yaml)

Source: https://github.com/dstackai/dstack/blob/master/examples/misc/ray/README.md

Defines a task named 'ray-cluster' with 4 nodes. It installs Ray and starts the head node on rank 0 and worker nodes on others. Exposes the Ray dashboard on port 8265. Apply with 'dstack apply -f cluster.dstack.yaml'.

```yaml
type: task
name: ray-cluster
nodes: 4
commands:
  - pip install -U "ray[default]"
  - |
    if [ $DSTACK_NODE_RANK = 0 ]; then 
      ray start --head --port=6379;
    else
      ray start --address=$DSTACK_MASTER_NODE_IP:6379
    fi
ports:
  - 8265 # ray dashboard port
resources:
  shm_size: 8GB

```

```shell
dstack apply -f cluster.dstack.yaml

```

--------------------------------

### Install dstack CLI and add project

Source: https://github.com/dstackai/dstack/blob/master/docker/server/README.md

Installs the dstack CLI and adds a project pointing to the server with the admin token. The output confirms the configuration is updated at ~/.dstack/config.yml.

```shell
$ pip install dstack
$ dstack project add --name main \
    --url http://127.0.0.1:3000 \
    --token bbae0f28-d3dd-4820-bf61-8f4bb40815da
    
Configuration is updated at ~/.dstack/config.yml
```

--------------------------------

### Run Docker Compose in a task with dstackai/dind

Source: https://github.com/dstackai/dstack/blob/master/mkdocs/blog/posts/docker-inside-containers.md

Use this task configuration to run Docker Compose inside a GPU-enabled container. Set image to dstackai/dind, privileged to true, and include start-dockerd as the first command. The example starts a web service on port 9000 and requests a GPU between 16GB and 24GB.

```yaml
type: task
name: compose-task

image: dstackai/dind
privileged: true

commands:
  - start-dockerd
  - |
    cat > compose.yaml <<'EOF'
    services:
      web:
        image: python:3.11-slim
        command: python -m http.server 9000
        ports:
          - "9000:9000"
    EOF
  - docker compose up
ports: [9000]

resources:
  gpu: 16GB..24GB

```

--------------------------------

### Initialize plugin package with uv init --library

Source: https://github.com/dstackai/dstack/blob/master/examples/plugins/example_plugin/README.md

Initializes a new Python library package for the plugin. Run this command in the plugin's root directory.

```shell
uv init --library
```