AI@JSC

https://sdlaml.pages.jsc.fz-juelich.de/ai/
Admin
AI@JSC provides guides, recipes, and consulting services for running applied machine learning and...
Tokens:53,504
Snippets:434
Trust Score:10
Update:3 weeks ago
Show doc for...
Tokens:
Raw
### Set up LangChain environment with uv

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/langchain_with_blablador

Creates a virtual environment named 'langchain' and installs the required LangChain packages (langchain_openai, langchain_community, langchain, chromadb) using uv. Run after installing uv with 'pip install uv'.

```bash
uv venv langchain
source langchain/bin/activate
uv pip install langchain_openai langchain_community langchain chromadb

```

--------------------------------

### Install pi-web-access package

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/blablador_pi

Run this command to install the pi-web-access package, which enables web browsing capabilities for your Blablador-powered Pi agent.

```bash
pi install npm:pi-web-access
```

--------------------------------

### Install vLLM environment on Jedi/JUPITER

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/vllm_on_jedi?q=

Creates a uv virtual environment with system site packages, installs required Python packages, removes CPU-only PyTorch, builds Triton, and installs pre-built wheels for flash-attn, xformers, megablocks, bitsandbytes, and vllm. Requires uv and the 2025 stage.

```bash
ml Python
pip install --user uv

ml PyTorch

uv venv --system-site-packages --relocatable vllm

source vllm/bin/activate

cd vllm

uv pip install \
        accelerate \
        deepspeed \
        openai \
        mistral_common \
        msgspec \
        peft \
        pyarrow \
        sentencepiece \
        tiktoken \
        transformers \
        trl \
        opencv-python


# Someone here installs a cpu-only version of pytorch. We don’t need it. 
rm -rf lib/python3.12/site-packages/torch*

uv pip install stanford-stk --no-deps

uv pip install \
        aioprometheus \
        fastapi \
        fschat[model_worker,webui] \
        gguf \
        lm-format-enforcer \
        llmcompressor \
        outlines \
        partial_json_parser \
        prometheus-fastapi-instrumentator \
        ray==2.34.0 \
        typer \
        uvicorn[standard]\
        zmq \
        ninja \
        wheel

uv pip uninstall  pynvml
uv pip install nvidia-ml-py

# Triton
ml CMake LLVM
git clone https://github.com/triton-lang/triton.git;
uv pip install  triton/python/ -vvv
python -c 'from torch.utils import _triton; print(f"{_triton.has_triton() =} ")'

wget https://static.abacus.ai/pypi/abacusai/gh200-llm/pytorch-2412-cuda126/flash_attn-2.7.2.post1-cp312-cp312-linux_aarch64.whl

wget https://static.abacus.ai/pypi/abacusai/gh200-llm/pytorch-2412-cuda126/vllm-0.6.6.post1%2Bcu126-cp312-cp312-linux_aarch64.whl

wget https://static.abacus.ai/pypi/abacusai/gh200-llm/pytorch-2412-cuda126/bitsandbytes-0.45.1.dev0-cp312-cp312-linux_aarch64.whl

wget https://static.abacus.ai/pypi/abacusai/gh200-llm/pytorch-2412-cuda126/megablocks-0.7.0-cp312-cp312-linux_aarch64.whl

wget https://static.abacus.ai/pypi/abacusai/gh200-llm/pytorch-2412-cuda126/xformers-0.0.30%2B46a02df6.d20250103-cp312-cp312-linux_aarch64.whl

uv pip install --no-deps --no-index --upgrade --find-links ./ flash-attn
uv pip install --no-deps --no-index --find-links ./  xformers
uv pip install --no-deps --no-index --find-links ./  megablocks
uv pip install --no-deps --no-index --find-links ./  bitsandbytes
uv pip install --no-deps --no-index --find-links ./  vllm
```

--------------------------------

### Run setup.sh to create the virtual environment

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setup_environment

After editing modules.sh and requirements.txt, run the setup script to create the virtual environment.

```bash
bash sc_venv_template/setup.sh

```

--------------------------------

### Activate the virtual environment

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setup_environment

Activate the virtual environment created by the setup script.

```bash
source sc_venv_template/activate.sh

```

--------------------------------

### Example OpenSSH public key

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setting-up-windows

Example of an OpenSSH public key file content. It should end with your computer user (e.g., alban@DESKTOP-3KTU7GR). This format is not produced by PuTTYgen. Copy this content for uploading to JuDoor.

```text
ssh-ed25519 OOPAC3NzaC1lMDI1NDD5AAAAIOpEi0ieVw39Fo/bDBSbee3FcFHfKAzKCLECwinFQ+gyfD alban@DESKTOP-3KTU7GR
```

--------------------------------

### Launch TensorBoard with root logdir

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/vizualize_training

Starts TensorBoard pointing to a root directory containing multiple experiment logs. Useful when managing several experiments under one root.

```bash
tensorboard --logdir runs --port 6006
```

--------------------------------

### Launch TensorBoard with logdir

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/vizualize_training

Starts TensorBoard pointing to a specific event file directory. Use this on a node that can access the training logs, typically a login or devel node.

```bash
tensorboard --logdir runs/exp_001/tb --port 6006
```

--------------------------------

### Clone repository with git clone

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/Run_a_github_code_on_the_SC

Navigates to the personal directory and clones the example repository using HTTPS. Always store code in the project space directory.

```bash
cd $USER
git clone https://github.com/HelmholtzAI-FZJ/2023-getting-started-with-ai-on-supercomputers.git
```

--------------------------------

### VSCode settings: Change .vscode-server location

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setup_vscode_editor?q=

Example of the 'remote.SSH.serverInstallPath' setting to change where the VSCode server is installed on the remote system. Replace 'yourHostName' and the path with your values.

```json
# EXAMPLE
"remote.SSH.serverInstallPath": {
    "yourHostName": "/path/to/your/preferred/location/"
},

```

--------------------------------

### Set up Virtual Environment with sc_venv_template

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/lightning_hydra_kickstart?q=

Commands to navigate to the sc_venv_template directory, edit requirements.txt to include lightning-hydra-template's requirements, comment out a module load line in modules.sh, and execute setup.sh to create the virtual environment.

```bash
# Navigate to sc_venv_template directory
cd sc_venv_template

# Edit requirements.txt of sc_venv_template to point to requirements.txt of lightning-hydra-template
# Add the line: `-r path/to/lightning-hydra-template/requirements.txt`

# Comment out the following line in modules.sh as it will be handled by the requirements.txt file:
# module load  PyTorch scikit-learn torchvision PyTorch-Lightning

# Execute setup.sh to create the virtual environment
bash setup.sh

```

--------------------------------

### Example SSH key with IP restrictions

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setup_ssh?q=

Example of a complete SSH key entry with IP restrictions for JSC systems, including FZJ IP range.

```text
from="93.199.0.0/16,10.0.0.0/8,134.94.0.0/16" ssh-ed25519 AAAAC3NzaC1lZDE1NTA4AAAAIHaoOJF3gqXd7CV6wncoob0DL2OJNfvjgnHLKEniHV6F user@demonstration.fz-juelich.de
```

--------------------------------

### Create and link personal directory

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/Run_a_github_code_on_the_SC

Creates a personal directory in the project space and links it to the home directory. Use this after logging into the login node to avoid limited home directory space.

```bash
mkdir -p /p/project1/your_project_name/$USER   # This command creates a directory within your project space.
ln -s /p/project1/your_project_name/$USER ~    # This creates a symbolic link, making your personal directory easily accessible from your home directory.
```

--------------------------------

### Download pretrained weights on login node

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/Run_a_github_code_on_the_SC

Commands to activate the virtual environment and run the training script on the login node to download pretrained model weights, since compute nodes lack internet access.

```bash
source sc_venv_template/activate.sh
python3 ddp_training.py
```

--------------------------------

### Add wandb to requirements.txt

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setup_environment

Add the line 'wandb' to the sc_venv_template/requirements.txt file to install Weights and Biases.

```text
wandb

```

--------------------------------

### Create and mount a SquashFS of Tiny-ImageNet-200

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/handling_large_datasets

Creates a SquashFS archive of the dataset if it doesn't exist, mounts it at a path under /dev/shm using squashfuse_ll, and registers cleanup on exit. Requires mksquashfs, squashfuse_ll, and fusermount3 utilities.

```python
import atexit
import getpass
import os
from pathlib import Path
import shutil
import subprocess

data_path = Path('/p/project1/training2306/datasets/tiny-imagenet-200').expanduser()
sqsh_path = Path(f'{data_path}.sqsh')
mount_path = Path(f'/dev/shm/{getpass.getuser()}/sqsh/{data_path.name}')

# Create a SquashFS of the data if it doesn't exist.
if not sqsh_path.exists():
    subprocess.run(['mksquashfs', data_path, sqsh_path], check=True)

def clean_up_squashfuse(mount_path):
    if mount_path.is_dir():
        subprocess.run(['fusermount3', '-u', mount_path])
    shutil.rmtree(mount_path, ignore_errors=True)

# Clean up any remains of previous jobs.
clean_up_squashfuse(mount_path)
# Create our mount directory.
mount_path.mkdir(parents=True, exist_ok=True)
mount_path.chmod(0o700)

# Unmount the SquashFS when Python exits.
atexit.register(clean_up_squashfuse, mount_path)
# Mount the SquashFS at our mount path.
subprocess.run(['squashfuse_ll', sqsh_path, mount_path], check=True);

# Read data from `mount_path`.
```

--------------------------------

### Copy ImageNet dataset with scp

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/Run_a_github_code_on_the_SC

Example scp command to copy the ImageNet dataset from a local machine to the scratch folder on the supercomputer. Replace placeholders with actual values.

```bash
bash scp -R imagenet-object-localization-challenge.zip <yourid>@juwels-booster.fz-juelich.de:/p/scratch/your_project_name/
```

--------------------------------

### Configure OpenCode to use Blablador

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/blablador_opencode?q=

Configuration for OpenCode to connect to Blablador. Place this in `$HOME/.config/opencode/opencode.json` on Linux and Mac. Replace `YOUR_BLABLADOR_API_KEY_HERE` with your actual Blablador token. The configuration sets the default and small models to `blablador/alias-fast`, enables context pruning via the `@tarquinen/opencode-dcp` plugin, and defines four Blablador models with context and output limits.

```json
{
  "$schema": "https://opencode.ai/config.json",
  // Plugin for pruning context
  "plugin": ["@tarquinen/opencode-dcp@latest"],
  // The small model is used to generate texts for action summaries
  "model": "blablador/alias-fast",
  "small_model": "blablador/alias-fast",

"compaction": {
    "auto": true,
    "prune": true,
    "reserved": 10000
  },

  "provider": {
    "blablador": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Blablador",

      "options": {
        "baseURL": "https://api.blablador.fz-juelich.de/v1",
        "apiKey": "YOUR_BLABLADOR_API_KEY_HERE"
      },

      "models": {
        "alias-code": {
          "name": "Blablador Code (alias-code)",
          "limit": {
            "context": 98304,
            "output": 32768
          }
        },
        "alias-huge": {
          "name": "Blablador Minimax (alias-huge)",
          "limit": {
            "context": 98304,
            "output": 32768
          }
        },
        "alias-fast": {
          "name": "Blablador Fast (alias-fast)",
           "limit": {
            "context": 98304,
            "output": 32768
          }
        },
        "alias-large": {
          "name": "Blablador Large (alias-large)",
           "limit": {
            "context": 98304,
            "output": 32768
          }
        }
      }
    }
  }
}
```

--------------------------------

### SLURM job status: squeue output (pending)

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setup_vscode_editor?q=

Example output of 'squeue -u $USER' showing a pending job. The job is in PD state and has not been scheduled to a node yet.

```text
JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
6462769     batch  example user PD       0:00      1 (Priority)

```

--------------------------------

### Python example: copy Tiny-ImageNet-200 to /dev/shm

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/handling_large_datasets

Python code to create a tar archive of a dataset and extract it into /dev/shm, with cleanup and permission setting. Assumes single Python process; not for parallel use.

```python
import getpass
import os
from pathlib import Path
import shutil
import tarfile

data_path = Path('/p/project1/training2306/datasets/tiny-imagenet-200').expanduser()
tar_path = data_path.with_suffix('.tar')
shm_path = Path(f'/dev/shm/{getpass.getuser()}/{data_path.name}')

# Create a tar archive of the data if it doesn't exist.
if not tar_path.exists():
    with tarfile.open(tar_path, 'w') as tar:
        tar.add(data_path, arcname=data_path.name)

# Clean up any remains of previous jobs.
shutil.rmtree(shm_path, ignore_errors=True)
shm_path.mkdir(parents=True)

# Extract the tar file into `shm_path`.
with tarfile.open(tar_path, 'r') as tar:
    tar.extractall(shm_path.parent)
# Make our data private so future jobs of other users will not be able
# to access possibly sensitive data in this shared location.
shm_path.chmod(0o700)

# Read data from `shm_path`.

```

--------------------------------

### Create .env file with BlaBlador API key

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/blablador_codex_tutorial?q=

Creates ~/.codex/.env containing the BlaBlador API key. Replace YOUR_BLABLADOR_API_KEY with your actual key. The file must be at this exact path for Codex to read it.

```bash
echo 'LOCAL_LLM_API_KEY="YOUR_BLABLADOR_API_KEY"' > ~/.codex/.env
```

--------------------------------

### SLURM job status: squeue output (running)

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/setup_vscode_editor?q=

Example output of 'squeue -u $USER' after the job is running. The job is in R state and scheduled to node jwc00n051. The long hostname is jwc00n051.juwels.

```text
JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
6462769     batch  example user  R       4:31      1 jwc00n051

```

--------------------------------

### Link cache directories to SCRATCH

Source: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/jsc_basics

Moves and links ~/.cache, ~/.apptainer, and ~/.triton to /p/scratch/my-project/$USER to free up HOME space. Run these commands after creating the project directory and linking it to HOME.

```bash
[ -d ~/.cache ] && mv ~/.cache /p/scratch/my-project/"$USER"/.cache; mkdir -p /p/scratch/my-project/"$USER"/.cache; ln -s /p/scratch/my-project/"$USER"/.cache ~

[ -d ~/.apptainer ] && mv ~/.apptainer /p/scratch/my-project/"$USER"/.apptainer; mkdir -p /p/scratch/my-project/"$USER"/.apptainer; ln -s /p/scratch/my-project/"$USER"/.apptainer ~

[ -d ~/.triton ] && mv ~/.triton /p/scratch/my-project/"$USER"/.triton; mkdir -p /p/scratch/my-project/"$USER"/.triton; ln -s /p/scratch/my-project/"$USER"/.triton ~

```
Tokens:
Raw