### Running Examples with Pixi

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/AGENTS.md

Command to start an example, which requires the Slurm Docker environment to be set up.

```bash
pixi run -e build --frozen start
```

--------------------------------

### Install and Run Slidev Project

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/slides/README.md

Install project dependencies, start the development server, and visit the local URL to view the presentation. Ensure you have pnpm installed.

```bash
pnpm install
pnpm dev
visit <http://localhost:3030>
```

--------------------------------

### Start Staging Environment

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/contributing/develop_slurm_locally.md

Run this command after setting up your .env file to start the staging environment. This will submit assets through Slurm and enable log collection.

```bash
pixi run start-staging
```

--------------------------------

### Clone and Run Dagster Slurm Example

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/slides/slides_scipy2026.md

Clone the repository, navigate to the examples directory, and run the development server. Use 'start-staging' to submit jobs to a local Dockerized Slurm environment.

```bash
# clone
git clone https://github.com/ascii-supply-networks/dagster-slurm
cd dagster-slurm/examples

# dev on the laptop — Dagster UI on :3000
pixi run start

# flip config → submits to Slurm-in-Docker
pixi run start-staging
```

--------------------------------

### Clone Repository and Install Dependencies

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/README.md

Clone the dagster-slurm repository and run the initial setup script using pixi.

```bash
git clone https://github.com/ascii-supply-networks/dagster-slurm.git
cd dagster-slurm
pixi run prek-install
pixi run prek-run
```

--------------------------------

### Install pixi

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/contributing/contributing.md

Install the pixi package manager using a curl command. This is a prerequisite for development setup.

```bash
curl -fsSL https://pixi.sh/install.sh | sh
```

--------------------------------

### Configure Docling Production (No Warmup)

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/applications/document-preprocessing-docling.md

Example configuration for production environments where automatic model warmup is disabled. This setup assumes models are pre-deployed and specifies paths for pre-downloaded models.

```python
# Production: Disable warmup and use pre-deployed models
completed_run = compute_ray.run(
    context=context,
    payload_path=script_path,
    launcher=RayLauncher(num_gpus_per_node=0),
    extra_env={
        "INPUT_GLOB": "data/**/*.pdf",
        "NUM_WORKERS": "2",
        "DOCLING_WARMUP_MODELS": "false",  # Skip warmup in prod
        "HF_HOME": "/shared/models/huggingface",  # Use pre-deployed models
    },
)
```

--------------------------------

### Configure Ray Launcher Overrides

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/projects/dagster-slurm/README.md

Configure pre_start_commands for OS-level tweaks before Ray starts, such as increasing file-descriptor limits. Use ray_start_args to pass extra arguments to ray start, like disabling the dashboard.

```python
"launchers": {
    "ray": {
        "pre_start_commands": [
            "ulimit -n 65536",
        ],
    },
}
```

--------------------------------

### Markdown Image Example (Absolute Path)

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/tutorial-basics/markdown-features.mdx

Reference images located in the static directory using absolute paths starting with '/'.

```md
![Docusaurus logo](/img/docusaurus.png)
```

--------------------------------

### Environment Setup with Pixi

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/AGENTS.md

Commands to set up and synchronize the development environment using Pixi. Use `--frozen sync` for consistent environments.

```bash
pixi run -e build --frozen sync
```

```bash
pixi run -e build --frozen sync-lib-with-upgrade
```

--------------------------------

### Clone Dagster Slurm Example

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/examples/README.md

Clone the Dagster Slurm repository to access the example files. Navigate into the examples directory.

```bash
git clone https://github.com/ascii-supply-networks/dagster-slurm.git
cd dagster-slurm/examples
```

--------------------------------

### Install Global Tools with pixi

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/contributing/contributing.md

Install essential global development tools like git, make, uv, and prek using pixi. This command ensures necessary utilities are available.

```bash
pixi global install git make uv prek
```

--------------------------------

### ComputeResource Usage Example

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/api/api_core.md

Demonstrates how to use the ComputeResource to run a script with a specific launcher. The ComputeResource abstracts the underlying execution mode (local, Slurm, etc.).

```python
@asset
def my_asset(context: AssetExecutionContext, compute: ComputeResource):
    return compute.run(
        context=context,
        payload_path="script.py",
        launcher=RayLauncher(num_gpus_per_node=2)
    )
```

--------------------------------

### Check Release Version

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/contributing/release.md

Run this command to preview the version that would be released without actually publishing. Requires semantic-release to be installed.

```bash
uv run semantic-release -v --noop version
```

--------------------------------

### Clone Repository and Start Docker Compose

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/intro.md

Clone the dagster-slurm repository and start the necessary Docker services to set up a local HPC environment.

```bash
git clone https://github.com/ascii-supply-networks/dagster-slurm.git
cd dagster-slurm
docker compose up
cd examples
```

--------------------------------

### Setup Model Download Directory

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/applications/document-preprocessing-docling.md

Set environment variables to specify the model cache directory and Hugging Face home directory on a shared filesystem. This is crucial for air-gapped or restricted HPC clusters.

```bash
# Create download directory on shared filesystem
export MODEL_CACHE_DIR=/shared/models
export HF_HOME=/shared/models/huggingface
```

--------------------------------

### Set Deployment Path and Run Production

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/projects/dagster-slurm/README.md

Set the environment variable for the deployment path and then run the production start command. Ensure CI_DEPLOYED_ENVIRONMENT_PATH is set to prevent accidental live builds.

```bash
export CI_DEPLOYED_ENVIRONMENT_PATH=/home/submitter/pipelines/deployments/prod-env-20251018
export DAGSTER_DEPLOYMENT=production_supercomputer
pixi run start-production-supercomputer
```

--------------------------------

### Start Dagster UI Locally

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/slides/slides.md

Starts the Dagster UI on http://localhost:3000 using Pixi. This command is used for local development and testing of your Dagster assets and jobs.

```bash
cd examples
pixi run start
```

--------------------------------

### Control-plane Asset Example with RayLauncher Override

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/integration-ray/ray.md

Define a Dagster asset that uses the ComputeResource and overrides the RayLauncher for specific resource requirements. This example shows how to run a Ray job using a specified working directory and environment variables.

```python
import dagster as dg
from dagster_slurm import ComputeResource, RayLauncher


@dg.asset(required_resource_keys={"compute"})
def train_model(context: dg.AssetExecutionContext) -> None:
    ray_launcher = RayLauncher(
        num_cpus_per_node=16,
        num_gpus_per_node=2,
        working_dir="ray_jobs/train_model",
    )

    completed = context.resources.compute.run(
        context=context,
        payload_path="python -m train.entrypoint",
        launcher=ray_launcher,
        extra_env={"WANDB_MODE": "offline"},
        extras={"experiment": context.run.run_id},
    )
    yield from completed.get_results()
```

--------------------------------

### Asset with SlurmRunConfig

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/api/api_core.md

Example of an asset using ComputeResource.run with SlurmRunConfig for per-run configuration of environment caching and payload upload behavior.

```python
@dg.asset
def my_asset(
    context: dg.AssetExecutionContext,
    compute: ComputeResource,
    config: SlurmRunConfig,
):
    return compute.run(
        context=context,
        payload_path="script.py",
        config=config,
    ).get_results()
```

--------------------------------

### Execute Asset with Custom Launcher

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/api/api_core.md

Override the default compute launcher for a specific asset execution. This example uses a RayLauncher with custom GPU settings.

```python
# Override launcher for this asset
ray_launcher = RayLauncher(num_gpus_per_node=4)
yield from compute.run(context, "script.py", launcher=ray_launcher)
```

--------------------------------

### Configure Docling Automatic Warmup (Development)

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/applications/document-preprocessing-docling.md

Example configuration for development or staging environments using Dagster. It demonstrates how to enable automatic model warmup by default, allowing Ray workers to use cached models.

```python
# Development/Staging: Use automatic warmup (default)
completed_run = compute_ray.run(
    context=context,
    payload_path=script_path,
    launcher=RayLauncher(num_gpus_per_node=0),
    extra_env={
        "INPUT_GLOB": "data/**/*.pdf",
        "NUM_WORKERS": "2",
        # DOCLING_WARMUP_MODELS defaults to "true"
    },
)
```

--------------------------------

### Markdown Link Example

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/tutorial-basics/markdown-features.mdx

Standard Markdown syntax for creating links to other pages using relative file paths.

```md
Let's see how to [Create a page](./create-a-page.md).
```

--------------------------------

### Ray Launcher Configuration for MUSICA

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/how-to/hpc-musica.md

Configure Ray to increase file descriptor limits for MUSICA. This override automatically applies `ulimit -n 65536` before starting Ray. If the Ray dashboard fails, consider adding `ray_start_args: ["--include-dashboard=false"]`.

```python
SUPERCOMPUTER_SITE_OVERRIDES = {
    "musica": {
        "launchers": {
            "ray": {
                "pre_start_commands": ["ulimit -n 65536"],
            }
        },
    }
}
```

--------------------------------

### Markdown Image Example (Relative Path)

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/tutorial-basics/markdown-features.mdx

Reference images located relative to the current Markdown file, useful for co-locating assets.

```md
![Docusaurus logo](./img/docusaurus.png)
```

--------------------------------

### Start Site in French Locale

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/tutorial-extras/translate-your-site.md

Run your Docusaurus development server with the '--locale fr' flag to view and test your site in French.

```bash
npm run start -- --locale fr
```

--------------------------------

### Recommended Setup for Laptop Deployments

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/how-to/troubleshooting.md

This Python code configures an in-process executor for Dagster jobs, which is suitable for laptop deployments as it keeps runs 'green' through Dagster restarts and avoids unnecessary multiprocessing when submitting to Slurm.

```python
from dagster import Definitions, define_asset_job, in_process_executor

# In-process executor keeps runs green through Dagster restarts.
# For laptop use this is almost always the right choice — there is
# no benefit to multiprocess when you are submitting to Slurm anyway.
slurm_job = define_asset_job(
    "slurm_job",
    executor_def=in_process_executor,
)

```

--------------------------------

### SparkLauncher for Apache Spark Workloads

Source: https://context7.com/ascii-supply-networks/dagster-slurm/llms.txt

Generates a bash script to start a Spark standalone cluster and submit jobs with spark-submit. Supports multi-node clusters and connecting to existing Spark masters. Configure executor and driver memory.

```python
from dagster_slurm import SparkLauncher, ComputeResource


spark_launcher = SparkLauncher(
    executor_cores=4,
    executor_memory="16g",
    driver_memory="8g",
    num_executors=None,        # None = auto (one executor per node)
    master_url=None,           # None = start a new standalone cluster
    spark_home="/opt/spark",
)

@dg.asset
def spark_etl(
    context: dg.AssetExecutionContext,
    compute_spark: ComputeResource,
    config: SlurmRunConfig,
):
    return compute_spark.run(
        context=context,
        payload_path="workloads/etl.py",
        launcher=spark_launcher,
        extra_slurm_opts={"nodes": 2, "cpus_per_task": 8, "mem": "64G"},
        config=config,
    ).get_results()
```

--------------------------------

### Automate Model Deployment to HPC

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/applications/document-preprocessing-docling.md

Automate the pre-downloading of Docling models as part of a cluster setup or deployment pipeline. This script unpacks the environment and downloads models to a specified shared directory.

```python
# deploy_models.py - Run once during cluster setup
import subprocess
from pathlib import Path

def deploy_models_to_hpc(env_cache_path: str, shared_models_dir: str):
    """Deploy models to shared HPC filesystem."""
    # 1. Unpack environment
    env_dir = Path(env_cache_path)
    pack_file = next(env_dir.glob("environment-*.sh"))
    subprocess.run([str(pack_file)], cwd=env_dir, check=True)

    # 2. Download models
    activate_script = env_dir / "activate.sh"
    download_script = "/path/to/examples/scripts/download_docling_models.py"

    subprocess.run(
        f"source {activate_script} && "
        f"export HF_HOME={shared_models_dir} && "
        f"python {download_script}",
        shell=True,
        check=True,
    )

    print(f"✓ Models deployed to {shared_models_dir}")

# Usage
deploy_models_to_hpc(
    env_cache_path="/home/user/dagster_runs/env-cache/c661e6dbdf4f9b47",
    shared_models_dir="/shared/models/huggingface"
)
```

--------------------------------

### Typical Asset Pattern with ComputeResource

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/AGENTS.md

Example of how to define a Dagster asset that utilizes the ComputeResource facade to run a script, potentially on a Slurm cluster. Ensure the ComputeResource is injected into the asset.

```python
@dg.asset
def my_asset(context: dg.AssetExecutionContext, compute: ComputeResource):
    return compute.run(
        context=context,
        payload_path="script.py",
        extra_env={"KEY": "value"},
    ).get_results()
```

--------------------------------

### Serve and Build Documentation

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/README.md

Commands to serve the documentation locally for preview or to build static documentation files. These are useful for development and deployment.

```bash
pixi run -e docs --frozen docs-serve
```

```bash
pixi run -e docs --frozen docs-build
```

--------------------------------

### Serve Documentation Locally

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/contributing/contributing.md

Build and serve the project documentation locally using pixi. This allows for previewing documentation changes.

```bash
pixi run -e docs --frozen docs-serve
```

--------------------------------

### Build and Publish Package

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/contributing/new_package.md

Build all packages and publish them to test PyPI and then PyPI. Ensure the UV_PUBLISH_TOKEN environment variable is set.

```bash
uv build --all-packages
UV_PUBLISH_TOKEN=pypi-<<token>> uv publish --index testpypi

UV_PUBLISH_TOKEN=pypi-<<token>> uv publish
```

--------------------------------

### Install dagster-slurm

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/slides/slides_scipy2026.md

Install the dagster-slurm package using pip or pixi.

```bash
pip install dagster-slurm
```

```bash
pixi add --pypi dagster-slurm
```

--------------------------------

### Build Documentation

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/contributing/contributing.md

Build the project documentation using pixi. This command generates static documentation files.

```bash
pixi run -e docs --frozen docs-build
```

--------------------------------

### Configure ComputeResource for Different Modes

Source: https://context7.com/ascii-supply-networks/dagster-slurm/llms.txt

Demonstrates setting up ComputeResource for local development, per-asset Slurm jobs, and production-ready Slurm sessions. Switching between modes requires only a configuration change.

```python
import dagster as dg
from dagster_slurm import (
    BashLauncher, ComputeResource, RayLauncher,
    SlurmResource, SlurmQueueConfig, SSHConnectionResource,
    SlurmSessionResource, SlurmRunConfig,
)

# ---- Local / dev mode (no SSH or Slurm needed) ----
compute_local = ComputeResource(
    mode="local",
    default_launcher=BashLauncher(),
)

# ---- Per-asset Slurm mode (staging) ----
ssh = SSHConnectionResource(
    host="login.cluster.example.com",
    user="alice",
    key_path="~/.ssh/id_ed25519",
)
queue = SlurmQueueConfig(
    partition="compute",
    qos="normal",
    num_nodes=1,
    cpus=8,
    mem="32G",
    time_limit="01:00:00",
)
slurm = SlurmResource(ssh=ssh, queue=queue, remote_base="$HOME/dagster_runs")

compute_slurm = ComputeResource(
    mode="slurm",
    slurm=slurm,
    default_launcher=BashLauncher(),
    auto_detect_platform=True,
    cache_inject_globs=["../dist/mylib-*.whl"],
)

# ---- Session mode (production: shared allocation) ----
session = SlurmSessionResource(
    slurm=slurm,
    num_nodes=4,
    time_limit="04:00:00",
    partition="compute",
    max_concurrent_jobs=10,
)
compute_session = ComputeResource(
    mode="slurm-session",
    slurm=slurm,
    session=session,
    default_launcher=BashLauncher(),
)

# ---- Heterogeneous job mode ----
compute_hetjob = ComputeResource(
    mode="slurm-hetjob",
    slurm=slurm,
    default_launcher=BashLauncher(),
)
```

--------------------------------

### Build the Library

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/contributing/release.md

Use this command to build the library. Ensure you are in the correct environment.

```bash
pixi run -e build --frozen build-lib
```

--------------------------------

### Start Local Slurm Cluster

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/contributing/contributing.md

Start a local Slurm cluster using Docker Compose. This command is used for testing and development environments that require Slurm.

```bash
docker compose up
ssh submitter@localhost -p 2223
# password: submitter
sinfo
```

--------------------------------

### Build and Publish Library

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/README.md

Commands to build the library for distribution and publish it to a registry. Ensure you are in the project root directory.

```bash
pixi run -e build --frozen build-lib
```

```bash
pixi run -e build --frozen publish-lib
```

--------------------------------

### Build and Upload Pixi Environment

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/how-to/prepare_production_style_runs.md

Use this command to build and upload the Pixi environment for production use. This step is crucial for reusing pre-built runtimes.

```bash
pixi run deploy-prod-docker  # builds and uploads the pixi environment
```

--------------------------------

### ComputeLauncher Execution Preparation

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/api/api_core.md

Prepares an execution plan for a compute launcher. This method is abstract and must be implemented by subclasses. It takes details about the script, environment, and execution context.

```python
compute_launcher.prepare_execution(
    payload_path="script.py",
    python_executable="/path/to/python",
    working_dir="/path/to/working_dir",
    pipes_context={"DAGSTER_PIPES_STEP_KEY": "my_step"},
    extra_env={"MY_VAR": "my_value"},
    allocation_context={"partition": "gpu"},
    activation_script="/path/to/activate.sh"
)
```

--------------------------------

### Build Docker Image

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/services/slurm/README.md

Builds the Docker image for the Slurm cluster. Ensure Docker and Docker Compose are installed.

```bash
docker compose build
```

--------------------------------

### Create First Blog Post

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/tutorial-basics/create-a-blog-post.md

Use Markdown files with front matter to create blog posts. The front matter includes metadata like slug, title, authors, and tags.

```markdown
---
slug: greetings
title: Greetings!
authors:
  - name: Joel Marcey
    title: Co-creator of Docusaurus 1
    url: https://github.com/JoelMarcey
    image_url: https://github.com/JoelMarcey.png
  - name: Sébastien Lorber
    title: Docusaurus maintainer
    url: https://sebastienlorber.com
    image_url: https://github.com/slorber.png
tags: [greetings]
---

Congratulations, you have made your first post!

Feel free to play around and edit this post as much as you like.
```

--------------------------------

### Admonition Examples

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/tutorial-basics/markdown-features.mdx

Docusaurus provides custom syntax for creating admonitions (callouts) like 'tip' and 'danger' to highlight important information.

```md
:::tip My tip

Use this awesome feature option

:::

:::danger Take care

This action is dangerous

:::
```

--------------------------------

### Upgrade Dependencies

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/contributing/contributing.md

Update project dependencies using pixi. This includes updating the main library dependencies and then synchronizing them with the examples directory.

```bash
pixi update
pixi run -e build --frozen sync-lib-with-upgrade
cd examples pixi update
```

--------------------------------

### Configure ComputeResource with RayLauncher

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/integration-ray/ray.md

Set up a ComputeResource for Slurm execution with RayLauncher as the default. This defines the execution mode, Slurm configuration, and Ray-specific parameters like CPU/GPU per node and Ray version.

```python
import dagster as dg
from dagster_slurm import (
    ComputeResource,
    ExecutionMode,
    RayLauncher,
    SlurmQueueConfig,
    SlurmResource,
    SSHConnectionResource,
)

ssh = SSHConnectionResource.from_env(prefix="SLURM_EDGE_NODE_")
slurm = SlurmResource(
    ssh=ssh,
    queue=SlurmQueueConfig(partition="compute", qos="normal", time_limit="01:00:00"),
)

compute = ComputeResource(
    mode=ExecutionMode.SLURM,
    slurm=slurm,
    default_launcher=RayLauncher(
        num_cpus_per_node=32,
        num_gpus_per_node=4,
        ray_version="2.9.3",
    ),
)
```

--------------------------------

### Start Slurm Docker Cluster

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/services/slurm/README.md

Deploys the Slurm cluster using Docker Compose with the default Slurm version. SSH access is provided.

```bash
docker compose up
```

```bash
ssh submitter@localhost -p 2223
```

--------------------------------

### Deploy Execution Environment

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/projects/dagster-slurm/README.md

For production, pre-build and upload the execution environment via CI/CD. Capture the output path and expose it to Dagster as CI_DEPLOYED_ENVIRONMENT_PATH.

```python
python scripts/deploy_environment.py --platform linux-64  # run from CI
```

--------------------------------

### MDX with React Components

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/tutorial-basics/markdown-features.mdx

MDX allows embedding React components directly within Markdown. This example defines and uses a custom 'Highlight' component.

```jsx
export const Highlight = ({children, color}) => (
  <span
    style={{
      backgroundColor: color,
      borderRadius: '20px',
      color: '#fff',
      padding: '10px',
      cursor: 'pointer',
    }}
    onClick={() => {
      alert(`You clicked the color ${color} with label ${children}`)
    }}>
    {children}
  </span>
);

This is <Highlight color="#25c2a0">Docusaurus green</Highlight> !

This is <Highlight color="#1877F2">Facebook blue</Highlight> !
```

--------------------------------

### Production Deployment (Bash)

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/how-to/hpc-leonardo.md

Command to start the production Dagster deployment on the Leonardo cluster after publishing the environment bundle. This ensures that environments are not built during critical runs.

```bash
pixi run start-production-supercomputer
```

--------------------------------

### BashLauncher Execution Preparation

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/api/api_core.md

Generates a bash execution plan for running Python scripts. Requires specifying the payload path, Python executable, working directory, and Pipes context. Optional parameters include extra environment variables, Slurm allocation context, and an activation script.

```python
bash_launcher.prepare_execution(
    payload_path="script.py",
    python_executable="/path/to/python",
    working_dir="/path/to/working_dir",
    pipes_context={"DAGSTER_PIPES_STEP_KEY": "my_step"},
    activation_script="/path/to/activate.sh"
)
```

--------------------------------

### Start Dagster in Production Mode

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/how-to/prepare_production_style_runs.md

Initiate Dagster in production mode for the real HPC system. For local integration testing, refer to the Docker mode documentation.

```bash
# for the real HPC system
pixi run start-production-supercomputer
```

--------------------------------

### Run Heterogeneous Jobs with Compute Resource

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/api/api_core.md

Launches a heterogeneous job with specified assets, resource requirements, and Ray launchers for training and inference.

```python
compute.run_hetjob(
    context,
    assets=[
        ("prep", "prep.py", {"nodes": 1, "cpus_per_task": 8, "mem": "32G"}),
        ("train", "train.py", {"nodes": 4, "cpus_per_task": 32, "mem": "128G", "gpus_per_node": 2}),
        ("infer", "infer.py", {"nodes": 8, "cpus_per_task": 16, "mem": "64G", "gpus_per_node": 1}),
    ],
    launchers={
        "train": RayLauncher(num_gpus_per_node=2),
        "infer": RayLauncher(num_gpus_per_node=1),
    }
)
```

--------------------------------

### Dagster Pipes Context for Workloads

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/slides/slides_multimodal.md

Example of using `PipesContext` within a Python script to access configuration and log information, intended to be run as part of a Dagster asset.

```python
def main():
    context = PipesContext.get()
    num_epochs = context.extras["epochs"]
    context.log.debug(f"Number of epochs: {num_epochs}")
    input_path = os.environ.get("INPUT_DATA")
    context.log.info(f"Input: {input_path}")
    # Your processing logic here (pytorch, ...)
    result = {"rows_processed": 1000}
    context.report_asset_materialization(
        metadata={
            "rows": result["rows_processed"],
            "processing_time": "10s",
        }
    )

if __name__ == "__main__":
    with open_dagster_pipes() as context:
        main()
```

--------------------------------

### ComputeResource Get Pipes Client

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/api/api_core.md

Retrieves the appropriate Pipes client (LocalPipesClient or SlurmPipesClient) based on the ComputeResource's execution mode. Allows overriding the default launcher.

```python
pipes_client = compute.get_pipes_client(context=context, launcher=my_custom_launcher)
```

--------------------------------

### Docling Model Warmup Logic

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/applications/document-preprocessing-docling.md

Illustrates the simplified flow within the `process_documents_docling.py` script, showing how model warmup is performed before Ray workers begin processing. This ensures models are downloaded once and cached.

```python
# Simplified flow inside process_documents_docling.py
def run_processing(...):
    # 1. Warmup runs BEFORE Ray processing
    warmup_model_cache(context=context)  # Downloads models once

    # 2. Then Ray workers start
    ds = rd.from_items([{"path": p} for p in files])
    result = ds.map_batches(
        mapper_document,  # Workers now use cached models
        batch_size=batch_size,
        concurrency=(num_workers, num_workers),
    )
```

--------------------------------

### Front Matter Example

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/tutorial-basics/markdown-features.mdx

Front matter is metadata at the top of Markdown files, enclosed by triple dashes. It's used for document identification, titles, descriptions, and custom URLs.

```text
---
id: my-doc-id
title: My document title
description: My document description
slug: /my-custom-url
---
```

--------------------------------

### Manual Model Pre-download for Air-gapped Clusters

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/applications/document-preprocessing-docling.md

For air-gapped or restricted HPC clusters, manually pre-download models on a node with internet access. This involves unpacking an environment and running a download script.

```bash
# One-time setup on HPC cluster (run on login node with internet)
# 1. Unpack the environment
cd /home/first.lastname/dagster_runs/env-cache/ID_HASH
./environment-workload-document-processing-linux-64-20260122-203856.sh

# 2. Activate and download models
source activate.sh
python /path/to/examples/scripts/download_docling_models.py
```

--------------------------------

### Configure ComputeResource for Slurm

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/how-to/environment-packaging.md

Configure ComputeResource to use Slurm mode with a BashLauncher and enable auto-detection of the platform. This setup ensures assets run correctly on the cluster by packaging the environment.

```python
compute = ComputeResource(
    mode="slurm",
    slurm=slurm_resource,
    default_launcher=BashLauncher(),
    auto_detect_platform=True,
)
```

--------------------------------

### RayLauncher for Distributed Ray Workloads

Source: https://context7.com/ascii-supply-networks/dagster-slurm/llms.txt

Generates a bash script to bootstrap a Ray cluster across Slurm nodes. Supports multi-node GPU clusters and local single-node mode. Configure ports, timeouts, and startup delays.

```python
from dagster_slurm import RayLauncher, ComputeResource


# Multi-node GPU Ray cluster
ray_launcher = RayLauncher(
    num_gpus_per_node=2,
    ray_port=6379,
    dashboard_port=8265,
    head_startup_timeout=120,        # seconds to wait for head node
    worker_startup_delay=5,          # seconds between worker launches
    object_store_memory_gb=20,       # Ray object store per node (GB)
    port_strategy="hash_jobid",      # avoid port collisions on shared nodes
    pre_start_commands=["ulimit -n 65536"],  # run before ray start
)

@dg.asset(group_name="distributed")
def distributed_training(
    context: dg.AssetExecutionContext,
    compute: ComputeResource,
    config: SlurmRunConfig,
):
    return compute.run(
        context=context,
        payload_path="workloads/train_ray.py",
        launcher=ray_launcher,
        extra_slurm_opts={
            "nodes": 4,
            "cpus_per_task": 32,
            "mem": "128G",
            "gpus_per_node": 2,
            "time_limit": "04:00:00",
        },
        config=config,
    ).get_results()
```

--------------------------------

### Initial Load and Incremental Update Demo

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/slides/slides_multimodal.md

This output shows the state of data processing over three rounds. Round 1 is the initial load. Round 2 demonstrates an incremental update where one row is changed, one is added, and one is removed. Round 3 shows the result when no changes are detected.

```text
========================================================================
ROUND 1: initial load
========================================================================

[root input]
┌────────────┬──────────────────┬───────┬──────────────────┐
│ sample_uid ┆ file_uri         ┆ value ┆ category         │
│ ---        ┆ ---              ┆ ---   ┆ ---              │
│ str        ┆ str              ┆ f64   ┆ str              │
╞════════════╪══════════════════╪═══════╪══════════════════╡
│ avatar_001 ┆ s3://anam/raw/av ┆ 10.0  ┆ video/full       │
│ avatar_002 ┆ s3://anam/raw/av ┆ 20.0  ┆ transcript/whisp │
│ avatar_003 ┆ s3://anam/raw/av ┆ 30.0  ┆ video/face_crop  │
└────────────┴──────────────────┴───────┴──────────────────┘

[processed latest rows]
┌────────────┬────────┬──────────────┐
│ sample_uid ┆ result ┆ value_bucket │
╞════════════╪════════╪══════════════╡
│ avatar_001 ┆ 100.0  ┆ lt_50        │
│ avatar_002 ┆ 400.0  ┆ lt_50        │
│ avatar_003 ┆ 900.0  ┆ lt_50        │
└────────────┴────────┴──────────────┘
[processed increment] new=3 stale=0 orphaned=0 processed=3

```

```text
========================================================================
ROUND 2: one changed + one added + one removed
========================================================================

[root input]
┌────────────┬──────────────────┬───────┬──────────────────┐
│ sample_uid ┆ file_uri         ┆ value ┆ category         │
╞════════════╪══════════════════╪═══════╪══════════════════╡
│ avatar_001 ┆ s3://anam/raw/av ┆ 10.0  ┆ video/full       │
│ avatar_002 ┆ s3://anam/raw/av ┆ 25.0  ┆ transcript/whisp │
│ avatar_004 ┆ s3://anam/raw/av ┆ 60.0  ┆ video/face_crop  │
└────────────┴──────────────────┴───────┴──────────────────┘

[processed latest rows]
┌────────────┬────────┬──────────────┐
│ sample_uid ┆ result ┆ value_bucket │
╞════════════╪════════╪══════════════╡
│ avatar_001 ┆ 100.0  ┆ lt_50        │
│ avatar_002 ┆ 625.0  ┆ lt_50        │
│ avatar_003 ┆ 900.0  ┆ lt_50        │
│ avatar_004 ┆ 3600.0 ┆ gte_50       │
└────────────┴────────┴──────────────┘
[processed increment] new=1 stale=1 orphaned=0 processed=2

```

```text
========================================================================
ROUND 3: no changes
========================================================================
[processed increment] new=0 stale=0 orphaned=0 processed=0

```

--------------------------------

### BashLauncher

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/api/api_core.md

Executes Python scripts via bash, utilizing a self-contained pixi environment.

```APIDOC
## class dagster_slurm.BashLauncher(**data)

Bases: `ComputeLauncher`

Executes Python scripts via bash.

Uses the self-contained pixi environment extracted at runtime.
Sources the activation script provided by pixi-pack.

### Parameters
  **data** (`Any`)

#### prepare_execution(payload_path, python_executable, working_dir, pipes_context, extra_env=None, allocation_context=None, activation_script=None)

Generate bash execution plan.

### Parameters
  - **payload_path** (`str`) – Path to Python script on remote host
  - **python_executable** (`str`) – Python from extracted environment
  - **working_dir** (`str`) – Working directory
  - **pipes_context** (`Dict`[`str`, `str`]) – Dagster Pipes environment variables
  - **extra_env** (`Optional`[`Dict`[`str`, `str`]]) – Additional environment variables
  - **allocation_context** (`Optional`[`Dict`[`str`, `Any`]]) – Slurm allocation info (for session mode)
  - **activation_script** (`Optional`[`str`]) – Path to activation script (provided by pixi-pack)
### Return type
`ExecutionPlan`
### Returns
ExecutionPlan with shell script
```

--------------------------------

### Initialize Git Submodules

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/SKILLS.md

Run this command after cloning the repository to ensure all submodules are updated and initialized.

```bash
git submodule update --init --recursive
```

--------------------------------

### Get Compute Resources based on Environment

Source: https://context7.com/ascii-supply-networks/dagster-slurm/llms.txt

Builds `ComputeResource` instances based on the `DAGSTER_DEPLOYMENT` environment variable. This function is used to wire resources in `definitions.py` without cluttering asset code.

```python
import os
import dagster as dg
from dagster_slurm_example.resources import get_resources

# Set the deployment target before calling get_resources():
#   DAGSTER_DEPLOYMENT=development                        → local mode
#   DAGSTER_DEPLOYMENT=staging_docker                     → per-asset sbatch via Docker Slurm
#   DAGSTER_DEPLOYMENT=production_docker                  → pre-deployed env, skip payload upload
#   DAGSTER_DEPLOYMENT=staging_supercomputer              → per-asset sbatch on real HPC
#   DAGSTER_DEPLOYMENT=production_supercomputer           → pre-deployed env on real HPC
#   DAGSTER_DEPLOYMENT=staging_supercomputer_session      → session mode
#   DAGSTER_DEPLOYMENT=staging_supercomputer_cluster_reuse→ session + Ray cluster reuse

os.environ["DAGSTER_DEPLOYMENT"] = "staging_docker"
os.environ["SLURM_EDGE_NODE_USER"] = "submitter"
os.environ["SLURM_EDGE_NODE_PASSWORD"] = "s3cr3t"

resource_defs = get_resources()
# Returns: {"compute": ComputeResource, "compute_ray": ComputeResource, "compute_spark": ComputeResource}

@dg.definitions
def defs():
    return dg.Definitions(
        assets=dg.load_assets_from_package_module(...),
        resources=resource_defs,
    )
```

--------------------------------

### Execute Asset with Session Resource Requirements

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/docs/api/api_core.md

Execute an asset in session mode, specifying resource requirements for cluster reuse. This example configures CPU, GPU, memory, and framework for a Ray cluster.

```python
# Session mode: specify resource requirements for cluster reuse
yield from compute.run(
    context,
    "script.py",
    launcher=RayLauncher(num_gpus_per_node=2),
    resource_requirements={"cpus": 32, "gpus": 2, "memory_gb": 128, "framework": "ray"}
)
```

--------------------------------

### Create Interactive Button with JSX

Source: https://github.com/ascii-supply-networks/dagster-slurm/blob/main/docs/blog/2021-08-01-mdx-blog-post.mdx

Use JSX within MDX to create interactive elements. This example shows a simple button that triggers an alert on click. Ensure React is configured for your MDX environment.

```js
<button onClick={() => alert('button clicked!')}>Click me!</button>
```