### Set up LangChain environment with uv

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/langchain_with_blablador

Creates a virtual environment named 'langchain' and installs the required LangChain packages (langchain_openai, langchain_community, langchain, chromadb) using uv. Run after installing uv with 'pip install uv'.

```bash
uv venv langchain
source langchain/bin/activate
uv pip install langchain_openai langchain_community langchain chromadb

```

--------------------------------

### Install pi-web-access package

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/blablador_pi

Run this command to install the pi-web-access package, which enables web browsing capabilities for your Blablador-powered Pi agent.

```bash
pi install npm:pi-web-access
```

--------------------------------

### Install vLLM environment on Jedi/JUPITER

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/vllm_on_jedi?q=

Creates a uv virtual environment with system site packages, installs required Python packages, removes CPU-only PyTorch, builds Triton, and installs pre-built wheels for flash-attn, xformers, megablocks, bitsandbytes, and vllm. Requires uv and the 2025 stage.

```bash
ml Python
pip install --user uv

ml PyTorch

uv venv --system-site-packages --relocatable vllm

source vllm/bin/activate

cd vllm

uv pip install \
        accelerate \
        deepspeed \
        openai \
        mistral_common \
        msgspec \
        peft \
        pyarrow \
        sentencepiece \
        tiktoken \
        transformers \
        trl \
        opencv-python


# Someone here installs a cpu-only version of pytorch. We don’t need it. 
rm -rf lib/python3.12/site-packages/torch*

uv pip install stanford-stk --no-deps

uv pip install \
        aioprometheus \
        fastapi \
        fschat[model_worker,webui] \
        gguf \
        lm-format-enforcer \
        llmcompressor \
        outlines \
        partial_json_parser \
        prometheus-fastapi-instrumentator \
        ray==2.34.0 \
        typer \
        uvicorn[standard]\
        zmq \
        ninja \
        wheel

uv pip uninstall  pynvml
uv pip install nvidia-ml-py

# Triton
ml CMake LLVM
git clone https://github.com/triton-lang/triton.git;
uv pip install  triton/python/ -vvv
python -c 'from torch.utils import _triton; print(f"{_triton.has_triton() =} ")'

wget https://static.abacus.ai/pypi/abacusai/gh200-llm/pytorch-2412-cuda126/flash_attn-2.7.2.post1-cp312-cp312-linux_aarch64.whl

wget https://static.abacus.ai/pypi/abacusai/gh200-llm/pytorch-2412-cuda126/vllm-0.6.6.post1%2Bcu126-cp312-cp312-linux_aarch64.whl

wget https://static.abacus.ai/pypi/abacusai/gh200-llm/pytorch-2412-cuda126/bitsandbytes-0.45.1.dev0-cp312-cp312-linux_aarch64.whl

wget https://static.abacus.ai/pypi/abacusai/gh200-llm/pytorch-2412-cuda126/megablocks-0.7.0-cp312-cp312-linux_aarch64.whl

wget https://static.abacus.ai/pypi/abacusai/gh200-llm/pytorch-2412-cuda126/xformers-0.0.30%2B46a02df6.d20250103-cp312-cp312-linux_aarch64.whl

uv pip install --no-deps --no-index --upgrade --find-links ./ flash-attn
uv pip install --no-deps --no-index --find-links ./  xformers
uv pip install --no-deps --no-index --find-links ./  megablocks
uv pip install --no-deps --no-index --find-links ./  bitsandbytes
uv pip install --no-deps --no-index --find-links ./  vllm
```

--------------------------------

### Run setup.sh to create the virtual environment

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setup_environment

After editing modules.sh and requirements.txt, run the setup script to create the virtual environment.

```bash
bash sc_venv_template/setup.sh

```

--------------------------------

### Activate the virtual environment

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setup_environment

Activate the virtual environment created by the setup script.

```bash
source sc_venv_template/activate.sh

```

--------------------------------

### Example OpenSSH public key

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setting-up-windows

Example of an OpenSSH public key file content. It should end with your computer user (e.g., alban@DESKTOP-3KTU7GR). This format is not produced by PuTTYgen. Copy this content for uploading to JuDoor.

```text
ssh-ed25519 OOPAC3NzaC1lMDI1NDD5AAAAIOpEi0ieVw39Fo/bDBSbee3FcFHfKAzKCLECwinFQ+gyfD alban@DESKTOP-3KTU7GR
```

--------------------------------

### Launch TensorBoard with root logdir

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/vizualize_training

Starts TensorBoard pointing to a root directory containing multiple experiment logs. Useful when managing several experiments under one root.

```bash
tensorboard --logdir runs --port 6006
```

--------------------------------

### Launch TensorBoard with logdir

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/vizualize_training

Starts TensorBoard pointing to a specific event file directory. Use this on a node that can access the training logs, typically a login or devel node.

```bash
tensorboard --logdir runs/exp_001/tb --port 6006
```

--------------------------------

### Clone repository with git clone

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/Run_a_github_code_on_the_SC

Navigates to the personal directory and clones the example repository using HTTPS. Always store code in the project space directory.

```bash
cd $USER
git clone https://github.com/HelmholtzAI-FZJ/2023-getting-started-with-ai-on-supercomputers.git
```

--------------------------------

### Example SSH key with IP restrictions

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setup_ssh?q=

Example of a complete SSH key entry with IP restrictions for JSC systems, including FZJ IP range.

```text
from="93.199.0.0/16,10.0.0.0/8,134.94.0.0/16" ssh-ed25519 AAAAC3NzaC1lZDE1NTA4AAAAIHaoOJF3gqXd7CV6wncoob0DL2OJNfvjgnHLKEniHV6F user@demonstration.fz-juelich.de
```

--------------------------------

### Add wandb to requirements.txt

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setup_environment

Add the line 'wandb' to the sc_venv_template/requirements.txt file to install Weights and Biases.

```text
wandb

```

--------------------------------

### VSCode settings: Change .vscode-server location

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setup_vscode_editor?q=

Example of the 'remote.SSH.serverInstallPath' setting to change where the VSCode server is installed on the remote system. Replace 'yourHostName' and the path with your values.

```json
# EXAMPLE
"remote.SSH.serverInstallPath": {
    "yourHostName": "/path/to/your/preferred/location/"
},

```

--------------------------------

### Copy ImageNet dataset with scp

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/Run_a_github_code_on_the_SC

Example scp command to copy the ImageNet dataset from a local machine to the scratch folder on the supercomputer. Replace placeholders with actual values.

```bash
bash scp -R imagenet-object-localization-challenge.zip <yourid>@juwels-booster.fz-juelich.de:/p/scratch/your_project_name/
```

--------------------------------

### Python example: copy Tiny-ImageNet-200 to /dev/shm

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/handling_large_datasets

Python code to create a tar archive of a dataset and extract it into /dev/shm, with cleanup and permission setting. Assumes single Python process; not for parallel use.

```python
import getpass
import os
from pathlib import Path
import shutil
import tarfile

data_path = Path('/p/project1/training2306/datasets/tiny-imagenet-200').expanduser()
tar_path = data_path.with_suffix('.tar')
shm_path = Path(f'/dev/shm/{getpass.getuser()}/{data_path.name}')

# Create a tar archive of the data if it doesn't exist.
if not tar_path.exists():
    with tarfile.open(tar_path, 'w') as tar:
        tar.add(data_path, arcname=data_path.name)

# Clean up any remains of previous jobs.
shutil.rmtree(shm_path, ignore_errors=True)
shm_path.mkdir(parents=True)

# Extract the tar file into `shm_path`.
with tarfile.open(tar_path, 'r') as tar:
    tar.extractall(shm_path.parent)
# Make our data private so future jobs of other users will not be able
# to access possibly sensitive data in this shared location.
shm_path.chmod(0o700)

# Read data from `shm_path`.

```

--------------------------------

### SSH config file for JUWELS Booster

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setting-up-windows

Example SSH configuration for connecting to the JUWELS Booster login node. Replace USERNAME with your JuDoor username and update the IdentityFile path to your OpenSSH private key. The file must be stored in the .ssh directory (e.g., C:/Users/myuser/.ssh/config).

```ssh
Host booster
        HostName juwels-booster.fz-juelich.de
        User USERNAME   
        AddressFamily inet
        IdentityFile C:/Users/me/.ssh/id_ed25519-JSC.openssh
    MACs hmac-sha2-512-etm@openssh.com

```

--------------------------------

### SLURM job status: squeue output (pending)

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setup_vscode_editor?q=

Example output of 'squeue -u $USER' showing a pending job. The job is in PD state and has not been scheduled to a node yet.

```text
JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
6462769     batch  example user PD       0:00      1 (Priority)

```

--------------------------------

### List Blablador models with curl

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/blablador_api_access?q=

The source provides a single code block that combines a curl command and its example JSON output. The command lists available models via the Blablador API endpoint; replace MY_API_KEY with your actual key. The output shows the response structure including model metadata and permissions.

```bash
curl --header "Authorization: Bearer ***" https://api.blablador.fz-juelich.de/v1/models

{
  "object": "list",
  "data": [
    {
      "id": "alias-code",
      "object": "model",
      "created": 1789947674,
      "owned_by": "vllm",
          "root": "Qwen/Qwen3.8-27B-FP8",
          "parent": null,
      "max_model_len": 262144,
      "permission": [
        {
          "id": "modelperm-98d463ba4933365e",
          "object": "model_permission",
          "created": 1789947674,
          "allow_create_engine": false,
          "allow_sampling": true,
          "allow_logprobs": true,
          "allow_search_indices": false,
          "allow_view": true,
          "allow_fine_tuning": false,
          "organization": "*",
          "group": null,
          "is_blocking": false
        }
      ]
    },
    {
      "id": "alias-fast",
      "object": "model",
      "created": 1789947674,
      "owned_by": "vllm",
          "root": "openai/gpt-oss-120b",
          "parent": null,
      "max_model_len": 131072,
      "permission": [
        {
          "id": "modelperm-953e87ceaf308332",
          "object": "model_permission",
          "created": 1789947674,
          "allow_create_engine": false,
          "allow_sampling": true,
          "allow_logprobs": true,
          "allow_search_indices": false,
          "allow_view": true,
          "allow_fine_tuning": false,
          "organization": "*",
          "group": null,
          "is_blocking": false
        }
      ]
    },
  ]
}%   

```

--------------------------------

### SLURM job status: squeue output (running)

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setup_vscode_editor?q=

Example output of 'squeue -u $USER' after the job is running. The job is in R state and scheduled to node jwc00n051. The long hostname is jwc00n051.juwels.

```text
JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
6462769     batch  example user  R       4:31      1 jwc00n051

```

--------------------------------

### Set up Virtual Environment with sc_venv_template

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/lightning_hydra_kickstart?q=

Commands to navigate to the sc_venv_template directory, edit requirements.txt to include lightning-hydra-template's requirements, comment out a module load line in modules.sh, and execute setup.sh to create the virtual environment.

```bash
# Navigate to sc_venv_template directory
cd sc_venv_template

# Edit requirements.txt of sc_venv_template to point to requirements.txt of lightning-hydra-template
# Add the line: `-r path/to/lightning-hydra-template/requirements.txt`

# Comment out the following line in modules.sh as it will be handled by the requirements.txt file:
# module load  PyTorch scikit-learn torchvision PyTorch-Lightning

# Execute setup.sh to create the virtual environment
bash setup.sh

```

--------------------------------

### Create and link personal directory

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/Run_a_github_code_on_the_SC

Creates a personal directory in the project space and links it to the home directory. Use this after logging into the login node to avoid limited home directory space.

```bash
mkdir -p /p/project1/your_project_name/$USER   # This command creates a directory within your project space.
ln -s /p/project1/your_project_name/$USER ~    # This creates a symbolic link, making your personal directory easily accessible from your home directory.
```

--------------------------------

### Download pretrained weights on login node

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/Run_a_github_code_on_the_SC

Commands to activate the virtual environment and run the training script on the login node to download pretrained model weights, since compute nodes lack internet access.

```bash
source sc_venv_template/activate.sh
python3 ddp_training.py
```

--------------------------------

### Create and mount a SquashFS of Tiny-ImageNet-200

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/handling_large_datasets

Creates a SquashFS archive of the dataset if it doesn't exist, mounts it at a path under /dev/shm using squashfuse_ll, and registers cleanup on exit. Requires mksquashfs, squashfuse_ll, and fusermount3 utilities.

```python
import atexit
import getpass
import os
from pathlib import Path
import shutil
import subprocess

data_path = Path('/p/project1/training2306/datasets/tiny-imagenet-200').expanduser()
sqsh_path = Path(f'{data_path}.sqsh')
mount_path = Path(f'/dev/shm/{getpass.getuser()}/sqsh/{data_path.name}')

# Create a SquashFS of the data if it doesn't exist.
if not sqsh_path.exists():
    subprocess.run(['mksquashfs', data_path, sqsh_path], check=True)

def clean_up_squashfuse(mount_path):
    if mount_path.is_dir():
        subprocess.run(['fusermount3', '-u', mount_path])
    shutil.rmtree(mount_path, ignore_errors=True)

# Clean up any remains of previous jobs.
clean_up_squashfuse(mount_path)
# Create our mount directory.
mount_path.mkdir(parents=True, exist_ok=True)
mount_path.chmod(0o700)

# Unmount the SquashFS when Python exits.
atexit.register(clean_up_squashfuse, mount_path)
# Mount the SquashFS at our mount path.
subprocess.run(['squashfuse_ll', sqsh_path, mount_path], check=True);

# Read data from `mount_path`.
```

--------------------------------

### Configure OpenCode to use Blablador

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/blablador_opencode?q=

Configuration for OpenCode to connect to Blablador. Place this in `$HOME/.config/opencode/opencode.json` on Linux and Mac. Replace `YOUR_BLABLADOR_API_KEY_HERE` with your actual Blablador token. The configuration sets the default and small models to `blablador/alias-fast`, enables context pruning via the `@tarquinen/opencode-dcp` plugin, and defines four Blablador models with context and output limits.

```json
{
  "$schema": "https://opencode.ai/config.json",
  // Plugin for pruning context
  "plugin": ["@tarquinen/opencode-dcp@latest"],
  // The small model is used to generate texts for action summaries
  "model": "blablador/alias-fast",
  "small_model": "blablador/alias-fast",

"compaction": {
    "auto": true,
    "prune": true,
    "reserved": 10000
  },

  "provider": {
    "blablador": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Blablador",

      "options": {
        "baseURL": "https://api.blablador.fz-juelich.de/v1",
        "apiKey": "YOUR_BLABLADOR_API_KEY_HERE"
      },

      "models": {
        "alias-code": {
          "name": "Blablador Code (alias-code)",
          "limit": {
            "context": 98304,
            "output": 32768
          }
        },
        "alias-huge": {
          "name": "Blablador Minimax (alias-huge)",
          "limit": {
            "context": 98304,
            "output": 32768
          }
        },
        "alias-fast": {
          "name": "Blablador Fast (alias-fast)",
           "limit": {
            "context": 98304,
            "output": 32768
          }
        },
        "alias-large": {
          "name": "Blablador Large (alias-large)",
           "limit": {
            "context": 98304,
            "output": 32768
          }
        }
      }
    }
  }
}
```

--------------------------------

### Create .env file with BlaBlador API key

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/blablador_codex_tutorial?q=

Creates ~/.codex/.env containing the BlaBlador API key. Replace YOUR_BLABLADOR_API_KEY with your actual key. The file must be at this exact path for Codex to read it.

```bash
echo 'LOCAL_LLM_API_KEY="YOUR_BLABLADOR_API_KEY"' > ~/.codex/.env
```

--------------------------------

### Link cache directories to SCRATCH

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/jsc_basics

Moves and links ~/.cache, ~/.apptainer, and ~/.triton to /p/scratch/my-project/$USER to free up HOME space. Run these commands after creating the project directory and linking it to HOME.

```bash
[ -d ~/.cache ] && mv ~/.cache /p/scratch/my-project/"$USER"/.cache; mkdir -p /p/scratch/my-project/"$USER"/.cache; ln -s /p/scratch/my-project/"$USER"/.cache ~

[ -d ~/.apptainer ] && mv ~/.apptainer /p/scratch/my-project/"$USER"/.apptainer; mkdir -p /p/scratch/my-project/"$USER"/.apptainer; ln -s /p/scratch/my-project/"$USER"/.apptainer ~

[ -d ~/.triton ] && mv ~/.triton /p/scratch/my-project/"$USER"/.triton; mkdir -p /p/scratch/my-project/"$USER"/.triton; ln -s /p/scratch/my-project/"$USER"/.triton ~

```

--------------------------------

### RAG with LangChain and Blablador

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/langchain_with_blablador

Demonstrates RAG with LangChain and Blablador. Sets OPENAI_API_KEY and OPENAI_API_BASE environment variables, downloads the State of the Union text, creates an index with OpenAIEmbeddings, and queries it with ChatOpenAI using the 'alias-fast' model. Replace the placeholder token with your actual Blablador token.

```python
from langchain_openai import ChatOpenAI
from langchain_community.document_loaders import TextLoader
from langchain_openai import OpenAIEmbeddings
from langchain_classic.indexes import VectorstoreIndexCreator

from urllib.request import urlretrieve

# You need to set them as environment variables, because the OpenAI API client uses them
# multiple times
os.environ["OPENAI_API_KEY"] = "ADD_YOUR_TOKEN_HERE"
os.environ["OPENAI_API_BASE"] = "https://api.blablador.fz-juelich.de/v1"

embedding = OpenAIEmbeddings(model="text-embedding-ada-002")

# Download the file to query from
urlretrieve(
    url="https://raw.githubusercontent.com/hwchase17/langchain/v0.0.200/docs/modules/state_of_the_union.txt",
    filename="./state_of_the_union.txt")

loader = TextLoader("state_of_the_union.txt")
index = VectorstoreIndexCreator(embedding=embedding).from_loaders([loader])
llm = ChatOpenAI(model="alias-fast")

questions = [
    "Who is the speaker",
    "What did the president say about Ketanji Brown Jackson",
    "What are the threats to America",
    "Who are mentioned in the speech",
    "Who is the vice president",
    "How many projects were announced",
]

for query in questions:
    print("Query:", query)
    print("Answer:", index.query(query, llm=llm))

```

--------------------------------

### Enable W&B online mode

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/vizualize_training

Sets W&B to upload live. Use on devel nodes that have internet access.

```bash
export WANDB_MODE=online
```

--------------------------------

### Download data with wget

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/Run_a_github_code_on_the_SC

Direct download command for fetching data from the internet on the supercomputer.

```bash
wget [URL]
```

--------------------------------

### Add web search skill instructions

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/blablador_hermes?q=

Paste these instructions into the chat to create a web search skill using the searxng engine at the given URL.

```text
Add a new skill for web search. It will use the searxng search engine with the following URL: https://search.blablador.fz-juelich.de - and this will be your MAIN AND ONLY web search. You should name it blablador-search- Test it afterwards.
```

--------------------------------

### Activate vLLM environment

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/vllm_on_jedi?q=

Loads the PyTorch module and activates the vLLM virtual environment to run vLLM.

```bash
ml PyTorch
source vllm/bin/activate
```

--------------------------------

### Relocate and link .cache directory

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/Run_a_github_code_on_the_SC

Relocates the .cache directory to the project space and links it back to the home directory. Run after setting up the personal directory to keep cache data off the limited home space.

```bash
[ -d ~/.cache ] && mv ~/.cache /p/project1/your_project_name/$USER/.cache; mkdir -p /p/project1/your_project_name/$USER/.cache; ln -s /p/project1/your_project_name/$USER/.cache ~
```

--------------------------------

### Authenticate W&B client

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/vizualize_training

Logs in to Weights & Biases from a node with internet access. Do this once per environment or when authentication expires.

```bash
wandb login
```

--------------------------------

### SSH login to JUWELS Booster

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setup_ssh

Use this command to log in to the JUWELS Booster system. Replace <yourid> with your JuDoor user name.

```bash
ssh -i ~/.ssh/id_ed25519 <yourid>@juwels-booster.fz-juelich.de
```

--------------------------------

### squashfs.sbatch template for FUSE-mounted SquashFS

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/handling_large_datasets

Template sbatch script for using a FUSE-mounted SquashFS. Handles creation, mounting, and cleanup across nodes. Requires adaptation of TODO values. Uses squashfuse_ll and fusermount3. Mount path must be in /dev/shm or /tmp on Jülich machines.

```bash
#!/usr/bin/env bash

# This `sbatch` script provides a template for using a FUSE-mounted
# SquashFS.
#
# Places that need adaptation depending on your needs are marked with
# "TODO".
#
# The advantage of using a SquashFS is good multiprocessing
# performance with arbitrarily sized data.
# The disadvantage is that FUSE is noticably slower than reading
# directly from RAM. If your data fits into RAM, prefer `shm.sbatch`
# instead.
# Another disadvantage is that a SquashFS is read-only. If your data
# changes, you will have to re-create the SquashFS every time.
# Similarly, you cannot write to the same location where you read data
# from.

#SBATCH --nodes=# TODO
#SBATCH --ntasks-per-node=# TODO
#SBATCH --account=# TODO
#SBATCH --partition=# TODO

[...]  # TODO

# Path to your data.
export DATA_PATH="..."  # TODO
# Where to create the SquashFS.
export SQSH_PATH="$DATA_PATH.sqsh"
# Where to put the virtual file system that accesses the SquashFS.
# Has to be in `/dev/shm` or `/tmp` on Jülich machines.
export MOUNT_PATH="/dev/shm/$(whoami)/sqsh/$(basename "$DATA_PATH")"

# Create the SquashFS if it doesn't exist yet.
[ -e "$SQSH_PATH" ] || mksquashfs "$DATA_PATH" "$SQSH_PATH"
# Optionally, to use a larger block size (may increase performance when
# you have larger files), replace the above line with:
# [ -e "$SQSH_PATH" ] || mksquashfs "$DATA_PATH" "$SQSH_PATH" -b 1M

unmount_squashfuse() {
    # Do nothing on tasks with node-local rank other than 0.
    ((SLURM_LOCALID)) && return 0
    [ -d "$MOUNT_PATH" ] && fusermount3 -u "$MOUNT_PATH"
    rm -rf "$MOUNT_PATH"
}
export -f unmount_squashfuse

mount_squashfuse() {
    # Do nothing on tasks with node-local rank other than 0.
    ((SLURM_LOCALID)) && return 0

    [ -d "$MOUNT_PATH" ] && ls -l "$MOUNT_PATH"
    # We unmount and completely remove `MOUNT_PATH` to prevent dirty
    # data (maybe there are still some remains from previous jobs).
    [ -d "$MOUNT_PATH" ] && fusermount3 -u "$MOUNT_PATH" || true
    rm -rf "$MOUNT_PATH"
    # Create a private `MOUNT_PATH`.
    mkdir -p "$MOUNT_PATH"
    chmod 700 "$MOUNT_PATH"

    # Register a clean-up handler for `MOUNT_PATH` that will run once this
    # job finishes, is interrupted, times out, or receives `scancel`.
    trap 'bash -c unmount_squashfuse' EXIT SIGINT SIGTERM SIGCONT
    # Try to make the SquashFS available at `MOUNT_PATH`. If that
    # fails, exit.
    squashfuse_ll "$SQSH_PATH" "$MOUNT_PATH" || exit 1

    # Enter an infinite loop so we can still receive signals in this
    # process. (The `srun` that calls this doesn't exit anyway.)
    while true; do
        # 25 hours
        sleep 90000
    done
}
export -f mount_squashfuse

wait_for_mount() {
    # Get the process ID of the most recent mount process.
    mount_pid="$(pgrep -n -f -u "$(whoami)" -- ' -c mount_squashfuse$')"
    # Check whether the mount process died or whether the mount is
    # complete.
    while ps -p "$mount_pid" > /dev/null \
            && ! mountpoint -q "$MOUNT_PATH"; do
        sleep 1
    done
}
export -f wait_for_mount

# Mount the SquashFS at `MOUNT_PATH`.
#
# This `srun` will never stop running (barring errors) due to the FUSE
# mount. We put it in the background and allow other jobs to overlap
# with its resources.
srun --overlap bash -c mount_squashfuse &

# Make sure the mount is complete on all nodes before continuing. This
# is required because we had to put the previous `srun` in the
# background.
srun bash -c wait_for_mount

# Access the SquashFS at `MOUNT_PATH`.
srun python main.py --input_path="$MOUNT_PATH"  # TODO

```

--------------------------------

### Migrate cache links to another project

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/jsc_basics

Moves cache directories from the old project's SCRATCH to a new project's SCRATCH and updates the symlinks in HOME. Run these when the original project ends.

```bash
mkdir -p /p/scratch/my-other-project/"$USER"; mv /p/scratch/my-project/"$USER"/.cache /p/scratch/my-other-project/"$USER"/.cache; rm ~/.cache; ln -s /p/scratch/my-other-project/"$USER"/.cache ~

mkdir -p /p/scratch/my-other-project/"$USER"; mv /p/scratch/my-project/"$USER"/.apptainer /p/scratch/my-other-project/"$USER"/.apptainer; rm ~/.apptainer; ln -s /p/scratch/my-other-project/"$USER"/.apptainer ~

mkdir -p /p/scratch/my-other-project/"$USER"; mv /p/scratch/my-project/"$USER"/.triton /p/scratch/my-other-project/"$USER"/.triton; rm ~/.triton; ln -s /p/scratch/my-other-project/"$USER"/.triton ~

```

--------------------------------

### Configure Codex with BlaBlador provider in config.toml

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/blablador_codex_tutorial?q=

Configures ~/.codex/config.toml to use the BlaBlador provider with the alias-code model. The base_url points to the BlaBlador API, env_key references the LOCAL_LLM_API_KEY environment variable, and wire_api is set to responses for compatibility.

```toml
model = "alias-code"
model_provider = "blablador"

[model_providers.blablador]
name = "BlaBlador"
base_url = "https://api.blablador.fz-juelich.de/v1"
env_key = "LOCAL_LLM_API_KEY"
wire_api = "responses"
```

--------------------------------

### Create SSH config file

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setup_ssh

Creates the SSH configuration file ~/.ssh/config if it doesn't exist.

```bash
touch ~/.ssh/config
```

--------------------------------

### Connect using SSH config alias

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setup_ssh

Use this command to connect to JUWELS Booster using the configured alias.

```bash
ssh juwels_booster
```

--------------------------------

### Configure Continue.Dev config.yaml for Blablador

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/blablador_on_vscode?q=

Configuration for the Continue.Dev extension in Visual Studio Code. Replace ADD_BLABLADOR_TOKEN_HERE with your actual Blablador token. The model is set to AUTODETECT and the API base is https://api.blablador.fz-juelich.de/v1. All Blablador models are subject to change without notice, but aliases are kept so updates are automatic.

```yaml
name: Blablador Helmholtz
version: 1.0.0
schema: v1
models: 
  - name: Blablador
    provider: openai
    model: AUTODETECT
    apiKey: ADD_BLABLADOR_TOKEN_HERE
    apiBase: https://api.blablador.fz-juelich.de/v1

context:
  - provider: code
  - provider: docs
  - provider: diff
  - provider: terminal
  - provider: problems
  - provider: folder
  - provider: codebase

```

--------------------------------

### SSH login to JURECA (standard access)

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setup_ssh

Use this command for standard access to JURECA. Replace <yourid> with your JuDoor user name.

```bash
ssh -i ~/.ssh/id_ed25519 <yourid>@jureca.fz-juelich.de
```

--------------------------------

### CLAUDE.md behavioral guidelines

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/blablador_hermes?q=

Create this file at $HOME/.hermes/CLAUDE.md to provide behavioral guidelines that reduce common LLM mistakes.

```markdown
# CLAUDE.md

Behavioral guidelines to reduce common LLM coding mistakes. Merge with project-specific instructions as needed.

**Tradeoff:** These guidelines bias toward caution over speed. For trivial tasks, use judgment.

## 1. Think Before Coding

**Don't assume. Don't hide confusion. Surface tradeoffs.**

Before implementing:
- State your assumptions explicitly. If uncertain, ask.
- If multiple interpretations exist, present them - don't pick silently.
- If a simpler approach exists, say so. Push back when warranted.
- If something is unclear, stop. Name what's confusing. Ask.

## 2. Simplicity First

**Minimum code that solves the problem. Nothing speculative.**

- No features beyond what was asked.
- No abstractions for single-use code.
- No "flexibility" or "configurability" that wasn't requested.
- No error handling for impossible scenarios.
- If you write 200 lines and it could be 50, rewrite it.

Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify.

## 3. Surgical Changes

**Touch only what you must. Clean up only your own mess.**

When editing existing code:
- Don't "improve" adjacent code, comments, or formatting.
- Don't refactor things that aren't broken.
- Match existing style, even if you'd do it differently.
- If you notice unrelated dead code, mention it - don't delete it.

When your changes create orphans:
- Remove imports/variables/functions that YOUR changes made unused.
- Don't remove pre-existing dead code unless asked.

The test: Every changed line should trace directly to the user's request.

## 4. Goal-Driven Execution

**Define success criteria. Loop until verified.**

Transform tasks into verifiable goals:
- "Add validation" → "Write tests for invalid inputs, then make them pass"
- "Fix the bug" → "Write a test that reproduces it, then make it pass"
- "Refactor X" → "Ensure tests pass before and after"

For multi-step tasks, state a brief plan:
1. [Step] → verify: [check]
2. [Step] → verify: [check]
3. [Step] → verify: [check]

Strong success criteria let you loop independently. Weak criteria ("make it work") require constant clarification.

---

**These guidelines are working if:** fewer unnecessary changes in diffs, fewer rewrites due to overcomplication, and clarifying questions come before implementation rather than after mistakes.
```

--------------------------------

### Profile dataset from FUSE-mounted SquashFS

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/handling_large_datasets

Asserts that the mount path exists and is mounted, then calls profile_dataset on the mount path to measure performance when reading from the SquashFS.

```python
assert mount_path.exists(), 'dataset directory does not exist'
assert (mount_path / 'val').exists(), 'dataset directory is not mounted'

profile_dataset(mount_path)
```

--------------------------------

### Profile dataset from /dev/shm

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/handling_large_datasets

Calls profile_dataset on the shm_path to measure performance when reading directly from /dev/shm.

```python
profile_dataset(shm_path)
```

--------------------------------

### Bash sbatch template for in-memory dataset (shm.sbatch)

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/handling_large_datasets

Template script for SLURM jobs that creates a tar archive and extracts it into /dev/shm once per node. Includes TODO placeholders for SLURM directives and paths. Use only when data fits in RAM; prefer squashfs.sbatch otherwise.

```bash
#!/usr/bin/env bash

# This `sbatch` script provides a template for using an in-memory
# dataset by creating and extracting a tar archive into `/dev/shm`.
#
# Places that need adaptation depending on your needs are marked with
# "TODO".
#
# The advantage of using an in-memory dataset is blazing performance.
# The disadvantage is that your data has to fit into RAM next to the
# program state. If your data does not fit into RAM, prefer
# `squashfs.sbatch` instead.
# Another disadvantage is that if your data changes, you will have to
# add it to the tar archive manually. You should also avoid writing to
# the same location where you read data from since the read location
# is node-local and temporary; you will not be able to access the data
# after the job ends.

#SBATCH --nodes=# TODO
#SBATCH --ntasks-per-node=# TODO
#SBATCH --account=# TODO
#SBATCH --partition=# TODO

[...]  # TODO

# Path to your data.
export DATA_PATH="..."  # TODO
export TAR_PATH="$DATA_PATH.tar"
# Where you will access your data from.
export SHM_PATH="/dev/shm/$(whoami)/$(basename "$DATA_PATH")"

# Create the tar file if it doesn't exist yet.
if ! [ -e "$TAR_PATH" ]; then
   tar cf "$TAR_PATH" -C "$(dirname "$DATA_PATH")" "$(basename "$DATA_PATH")"
fi

extract_to_shm() {
    # Do nothing on tasks with node-local rank other than 0.
    ((SLURM_LOCALID)) && return 0

    # We completely remove `SHM_PATH` before doing anything else to
    # prevent dirty data (maybe there are still some remains from
    # previous jobs).
    rm -rf "$SHM_PATH"

    mkdir -p "$SHM_PATH"
    # Extract data to `SHM_PATH`.
    tar xf "$TAR_PATH" -C "$(dirname "$SHM_PATH")"
    # Make `SHM_PATH` private.
    chmod 700 "$SHM_PATH"
}
export -f extract_to_shm

# Extract the data to `SHM_PATH`; only once per node.
srun bash -c extract_to_shm
# Use the data at `SHM_PATH`.
srun python main.py --input_path="$SHM_PATH"  # TODO

```

--------------------------------

### Profile dataset from standard filesystem

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/handling_large_datasets

Calls profile_dataset on the original data path to measure performance when reading from the standard file system.

```python
profile_dataset(data_path)
```

--------------------------------

### Create Slurm submission script

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/Run_a_github_code_on_the_SC

Slurm submission script for running a Python script on the booster partition. It requests 1 node, 4 tasks, 24 CPUs per task, and 4 GPUs, sets the CUDA_VISIBLE_DEVICES environment variable, activates the virtual environment, and runs the script with srun.

```bash
#!/bin/bash
#SBATCH --account=training2338           # Account details
#SBATCH --nodes=1                        # Number of compute nodes required
#SBATCH --ntasks-per-node=4              # Number of tasks per node
#SBATCH --cpus-per-task=24               # CPU cores per task
#SBATCH --gres=gpu:4                     # Use the 4 GPUs available
#SBATCH --output=output.%j               # File for standard output
#SBATCH --error=error.%j                 # File for standard error
#SBATCH --time=02:00:00                  # Maximum runtime
#SBATCH --partition=booster              # Specified machine partition

export CUDA_VISIBLE_DEVICES=0,1,2,3      # Very important to make the GPUs visible
export SRUN_CPUS_PER_TASK="$SLURM_CPUS_PER_TASK"  # Get number of cpu per task in the script

source sc_venv_template/activate.sh      # Command to activate the environment
srun python main.py      # Command to run your Python script
```

--------------------------------

### Add files to GitLab repository

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/recipes/hyperparameter_tuning

Commands to add files to a GitLab repository or push an existing Git repository. Run these commands from the command line to set up the remote and push the main branch.

```bash
cd existing_repo
git remote add origin https://gitlab.jsc.fz-juelich.de/AI_Recipe_Book/recipes/hyperparameter_tuning.git
git branch -M main
git push -uf origin main
```

--------------------------------

### List available models via curl

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/blablador_codex_tutorial?q=

Queries the BlaBlador API to list available models. Uses the LOCAL_LLM_API_KEY environment variable for authentication.

```bash
curl -H "Authorization: Bearer $LOCAL_LLM_API_KEY" https://api.blablador.fz-juelich.de/v1/models
```

--------------------------------

### SSH login to JURECA (IPv4 access)

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setup_ssh

Use this command to explicitly use IPv4 when accessing JURECA. Replace <yourid> with your JuDoor user name.

```bash
ssh -i ~/.ssh/id_ed25519 <yourid>@jureca-ipv4.fz-juelich.de
```

--------------------------------

### Configure MFA session persistence for JUWELS Booster

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setup_ssh

Add these settings to ~/.ssh/config to persist your SSH session and avoid repeated MFA prompts. Replace [USERNAME] and [OTHER SETTINGS ...] as needed. Note: session persistence only works for one login-node at a time.

```ssh-config
Host juwels_booster
    User [USERNAME]
    [OTHER SETTINGS ...]
    ControlPath ~/.ssh/control-%h-%p-%r
    ControlMaster auto
    ControlPersist 4h
```

--------------------------------

### Configure Blablador provider in models.json

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/blablador_pi

Add this JSON to ~/.pi/agent/models.json to register Blablador as a provider. Replace YOUR_BLABLADOR_API_TOKEN with your actual token. The models listed (alias-code, alias-huge, alias-fast, alias-large) are available for selection.

```json
{
  "providers": {
    "blablador": {
      "baseUrl": "https://api.helmholtz-blablador.fz-juelich.de/v1",
      "api": "openai-completions",
      "apiKey": "YOUR_BLABLADOR_API_TOKEN",
      "models": [
        { "id": "alias-code" },
        { "id": "alias-huge" },
        { "id": "alias-fast" },
        { "id": "alias-large" }
      ]
    }
  }
}
```

--------------------------------

### Sync all offline W&B runs

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/vizualize_training

Syncs all offline runs under a W&B directory to the cloud. Useful when you have multiple runs to upload.

```bash
wandb sync /path/to/WANDB_DIR/wandb
```

--------------------------------

### Clone the sc_venv_template repository

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setup_environment

Navigate to your project directory and clone the sc_venv_template repository from GitLab.

```bash
cd [YOUR PROJECT'S DIRECTORY]
git clone https://gitlab.jsc.fz-juelich.de/kesselheim1/sc_venv_template

```

--------------------------------

### SSH tunnel to view TensorBoard locally

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/vizualize_training

Creates an SSH tunnel from your local machine to the node running TensorBoard, allowing you to view it in your browser at http://localhost:6006. Ensure the hostname and port are correct and not already in use.

```bash
ssh -N -L 6006:localhost:6006 user@<login-or-devel-host>
```

--------------------------------

### Check W&B offline run directories

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/vizualize_training

Verifies the location of offline run directories. Use these commands to ensure you are pointing to the correct W&B directory.

```bash
ls -lah /path/to/WANDB_DIR
find /path/to/WANDB_DIR -maxdepth 3 -type d -name "offline-run-*" | head
```

--------------------------------

### Sync specific offline W&B run

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/vizualize_training

Syncs a specific offline run directory to W&B cloud from a node with internet. Replace /path/to/WANDB_DIR with the actual path.

```bash
wandb sync /path/to/WANDB_DIR/wandb/offline-run-*
```

--------------------------------

### Clone lightning-hydra-template and sc_venv_template

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/lightning_hydra_kickstart?q=

Commands to clone the lightning-hydra-template and sc_venv_template repositories from GitHub and GitLab respectively.

```bash
# Clone `lightning-hydra-template`
git clone https://github.com/ashleve/lightning-hydra-template

# Clone `sc_venv_template` for HPC systems
git clone https://gitlab.jsc.fz-juelich.de/kesselheim1/sc_venv_template.git

```

--------------------------------

### SSH config: Add login node as jump host

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setup_vscode_editor?q=

Adds the JUWELS login node as a jump host in the SSH config file. Replace placeholders with your username and private key path. This is the first step to enable VSCode to reach compute nodes through the login node.

```ssh
Host juwels_jump
    HostName juwels.fz-juelich.de
    User <username on JUWELS>
    IdentityFile ~/.ssh/id_ed_juwels_key

```

--------------------------------

### Configure OpenAI Sublime Text plugin for Blablador

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/blablador_on_vscode?q=

Settings for the OpenAI Sublime Text plugin to use Blablador. Replace ADD_BLABLADOR_TOKEN_HERE with your token. The prompt_mode can be panel, append, insert, or replace. These parameters work for the author but not always perfectly; you may add multiple assistants for different models or prompt modes.

```json
{
      "url": "https://api.blablador.fz-juelich.de/",
      "token": "ADD_BLABLADOR_TOKEN_HERE",
      "assistants": [
            {
                  "name": "Blablador Code",
                  "prompt_mode": "append",
                  "chat_model": "alias-code",
                  "assistant_role": "You are a senior code assistant",
                  "placeholder": "[PLACEHOLDER]",
            }
      ]
}

```

--------------------------------

### SSH key format for JuDoor

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setup_ssh

Format required when managing SSH keys in JuDoor under 'Systems' > 'Manage SSH-keys'. Replace placeholders with your IP range and your public key.

```text
from="[Your IP address or hostname]" ssh-ed25519 [Your SSH key]

```