AMD ROCm

https://rocm.docs.amd.com/en/latest/
Admin
ROCm is an open-source software platform optimized to extract HPC and AI workload performance from...
Tokens:334,594
Snippets:4,621
Trust Score:10
Update:5 months ago
Show doc for...
Tokens:
Raw
### Setup JAX MaxText Benchmark Environment

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/training/benchmark-docker/jax-maxtext.html?model=jax_maxtext_train_llama-2-7b

Run the setup script to install necessary libraries and datasets for benchmarking within the Docker container.

```bash
./jax-maxtext_benchmark_setup.sh -m Llama-3.1-70B
```

--------------------------------

### Setup JAX MaxText Benchmark Environment

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/training/benchmark-docker/jax-maxtext.html

Run the setup script to install necessary libraries and download datasets for JAX MaxText benchmarking. Specify the model to be used.

```bash
./jax-maxtext_benchmark_setup.sh -m Llama-2-70B
```

--------------------------------

### Setup JAX MaxText Benchmark Environment

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/training/benchmark-docker/jax-maxtext.html

Run the setup script to install necessary libraries and datasets for benchmarking the Llama-3.1-8B model within the Docker container.

```bash
./jax-maxtext_benchmark_setup.sh -m Llama-3.1-8B
```

--------------------------------

### Run JAX MaxText Benchmark Setup Script

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/training/benchmark-docker/jax-maxtext.html

Execute the setup script to install necessary libraries and datasets for benchmarking the specified model.

```bash
./jax-maxtext_benchmark_setup.sh -m Llama-2-7B
```

--------------------------------

### Run JAX MaxText Benchmark Setup Script

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/training/benchmark-docker/jax-maxtext.html

Executes the setup script to install required libraries and datasets for benchmarking the DeepSeek-V2-lite model.

```bash
./jax-maxtext_benchmark_setup.sh -m DeepSeek-V2-lite
```

--------------------------------

### Run PyTorch Benchmark Setup Script

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/training/benchmark-docker/pytorch-training.html

Execute the setup script to install necessary libraries and download datasets for PyTorch benchmarking. This script handles dependencies for various models.

```bash
./pytorch_benchmark_setup.sh
```

--------------------------------

### Setup Environment and Download Model

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference/benchmark-docker/sglang-mori-distributed.html

Set up a Python virtual environment, install the Hugging Face CLI, and download a specified model to a shared NFS mount point. Replace `<your_hf_token>` and the model name/path as needed.

```bash
# Set up a virtual environment and install the Hugging Face CLI
sudo apt update && sudo apt install -y python3-venv
python3 -m venv ~/venvs/hf
source ~/venvs/hf/bin/activate
pip install huggingface_hub

# Download the model to the shared NFS mount point
# Replace 'deepseek-ai/DeepSeek-R1-0528' with your desired model
hf download --token <your_hf_token> \
    deepseek-ai/DeepSeek-R1-0528 \
    --local-dir /mount/point/models/DeepSeek-R1

```

--------------------------------

### Run JAX MaxText Benchmark Setup Script

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/training/benchmark-docker/jax-maxtext.html

Execute the setup script to install necessary libraries and download datasets for benchmarking the Mixtral-8x7B model. This script is run from within the Docker container.

```bash
./jax-maxtext_benchmark_setup.sh -m Mixtral-8x7B
```

--------------------------------

### Install torchtune

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/fine-tuning/multi-gpu-fine-tuning-and-inference.html

Install the torchtune library, a PyTorch-native tool for LLM fine-tuning and inference on single and multi-GPU systems.

```bash
# Install torchtune with PyTorch release 2.2.2+
pip install torchtune
```

--------------------------------

### Install Optimum-AMD from Source

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference/hugging-face-models.html

Clone the Optimum-AMD repository and install it from source using pip for advanced customization or development.

```bash
git clone https://github.com/huggingface/optimum-amd.git
cd optimum-amd
pip install -e .

```

--------------------------------

### Install Dependencies

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/training/benchmark-docker/primus-megatron.html

Installs the necessary Python packages from the requirements file before running training.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install RDMA Performance Tools

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference/benchmark-docker/vllm-mori-distributed.html

Builds and installs the ROCm-optimized `rdma-perftest` suite from source. Requires development libraries and ROCm installation.

```bash
sudo apt install -y libibumad-dev libpci-dev libibverbs-dev librdmacm-dev ibverbs-utils libtool
git clone https://github.com/ROCm/rdma-perftest
cd rdma-perftest/
./autogen.sh
./configure --enable-rocm --with-rocm=/opt/rocm
make -j$(nproc)
sudo make install
```

--------------------------------

### Start and Attach to Docker Training Environment

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/training/benchmark-docker/primus-pytorch.html

Commands to start a stopped Docker container and then attach a bash shell to it.

```bash
docker start training_env
docker exec -it training_env bash
```

--------------------------------

### Compile and Install RoCE Library

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/system-setup/multi-node-setup.html

Compiles and installs the RoCE library, configures shared library paths, and copies the driver file.

```bash
sh autogen.sh
./configure
make
find /usr/lib64/ /usr/lib -name "libbnxt_re-rdmav*.so" -exec mv {} {}.inbox \;
make install all
sh -c "echo /usr/local/lib >> /etc/ld.so.conf"
ldconfig
cp -f bnxt_re.driver /etc/libibverbs.d/
find . -name "*.so" -exec md5sum {} \;
BUILT_MD5SUM=$(find . -name "libbnxt_re-rdmav*.so" -exec md5sum {} \; | cut -d " " -f 1)
```

--------------------------------

### Run Fine-tuning Benchmark (Qwen3-8B)

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/training/benchmark-docker/pytorch-training.html

Use this command to start the fine-tuning benchmark for the Qwen3-8B model. Supports full weight fine-tuning with BF16 and FP8, or LoRA fine-tuning with BF16.

```bash
./pytorch_benchmark_report.sh -t $training_mode \
    -m Qwen3-8B \
    -p $datatype \
    -s $sequence_length

```

--------------------------------

### Install Flash Attention 2 with Triton backend

Source: https://rocm.docs.amd.com/en/latest/how-to/llm-fine-tuning-optimization/model-acceleration-libraries.html

Installs Flash Attention 2 using the OpenAI Triton backend. This requires setting the FLASH_ATTENTION_TRITON_AMD_ENABLE environment variable to TRUE before running the setup script. Ensure you have cloned the repository and installed ninja.

```bash
git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention/
pip install ninja

# To install the Triton backend flash attention
FLASH_ATTENTION_TRITON_AMD_ENABLE="TRUE" python setup.py install
```

--------------------------------

### Run Qwen 2.5 7B Pretraining Benchmark (MI355X/MI350X)

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/training/benchmark-docker/primus-megatron.html

Start a pretraining benchmark for Qwen 2.5 7B on MI355X/MI350X accelerators.

```bash
bash runner/primus-cli direct \
  --log_file /tmp/primus_qwen2.5_7B.log \
  -- train pretrain \
  --config examples/megatron/configs/MI355X/qwen2.5_7B-BF16-pretrain.yaml
```

--------------------------------

### Install Flash Attention 2 with both CK and Triton backends

Source: https://rocm.docs.amd.com/en/latest/how-to/llm-fine-tuning-optimization/model-acceleration-libraries.html

Installs Flash Attention 2 with both Composable Kernel (CK) and OpenAI Triton backends. This is achieved by setting both FLASH_ATTENTION_TRITON_AMD_ENABLE to TRUE and FLASH_ATTENTION_SKIP_CK_BUILD to FALSE before running the setup script. Ensure you have cloned the repository and installed ninja.

```bash
git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention/
pip install ninja

# To install both CK and Triton backend flash attention
FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE && FLASH_ATTENTION_SKIP_CK_BUILD=FALSE python setup.py install
```

--------------------------------

### Install CK xFormers from source

Source: https://rocm.docs.amd.com/en/latest/how-to/llm-fine-tuning-optimization/model-acceleration-libraries.html

Installs the Composable Kernel (CK) version of xFormers from its source repository. This involves cloning the repository, initializing submodules, and running the setup script. Specify the ROCm architecture if needed, such as for Instinct MI300-series.

```bash
# Install from source
git clone https://github.com/ROCm/xformers.git
cd xformers/
git submodule update --init --recursive
PYTORCH_ROCM_ARCH=gfx942 python setup.py install #Instinct MI300-series
```

--------------------------------

### Initial Training Command Example

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/training/benchmark-docker/jax-maxtext.html?model=jax_maxtext_train_llama-2-7b

This command initiates a pretraining process using a specified configuration file and SLURM for distributed execution.

```bash
./primus-cli --config my_maxtext_config.yaml slurm srun -N 8 \
  -- train pretrain \
  --config examples/maxtext/configs/MI300X/mixtral_8x7B-pretrain.yaml
```

--------------------------------

### Run Fine-tuning Benchmark (Qwen2-1.5B)

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/training/benchmark-docker/pytorch-training.html

Start the fine-tuning benchmark for the Qwen2-1.5B model. Supports full weight fine-tuning (BF16, FP8) and LoRA fine-tuning (BF16).

```bash
./pytorch_benchmark_report.sh -t $training_mode \
    -m Qwen2-1.5B \
    -p $datatype \
    -s $sequence_length

```

--------------------------------

### Run PyTorch ROCm Docker Image

Source: https://rocm.docs.amd.com/en/latest/conceptual/ai-pytorch-inception.html

Execute this command to start the PyTorch ROCm-based Docker container. Ensure you have Docker installed and the necessary ROCm drivers.

```bash
docker run -it -v $HOME:/data --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --device=/dev/kfd --device=/dev/dri --group-add video --ipc=host --shm-size 8G rocm/pytorch:latest

```

--------------------------------

### Build Documentation (Linux/WSL)

Source: https://rocm.docs.amd.com/en/latest/contribute/building.html

Builds the HTML documentation using Sphinx from the project root on Linux or WSL. Ensure the virtual environment is activated and dependencies are installed.

```bash
.venv/bin/python -m sphinx -T -E -b html -d _build/doctrees -D language=en docs _build/html
```

--------------------------------

### Install ROCm Software Stack

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference/benchmark-docker/vllm-mori-distributed.html

Install ROCm 7.0.2 on Ubuntu 24.04 by downloading the .deb package, installing it, updating apt, installing Python dependencies, adding the user to necessary groups, and finally installing ROCm.

```bash
wget https://repo.radeon.com/amdgpu-install/7.0.2/ubuntu/noble/amdgpu-install_7.0.2.70002-1_all.deb
sudo apt install ./amdgpu-install_7.0.2.70002-1_all.deb
sudo apt update
sudo apt install python3-setuptools python3-wheel
sudo usermod -a -G render,video $LOGNAME # Add the current user to the render and video groups
sudo apt install rocm

```

--------------------------------

### Run Qwen 2.5 72B Pretraining Benchmark (MI355X/MI350X)

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/training/benchmark-docker/primus-megatron.html

Start a pretraining benchmark for Qwen 2.5 72B on MI355X/MI350X accelerators.

```bash
bash runner/primus-cli direct \
  --log_file /tmp/primus_qwen2.5_72B.log \
  -- train pretrain \
  --config examples/megatron/configs/MI355X/qwen2.5_72B-BF16-pretrain.yaml
```

--------------------------------

### Verify TensorFlow Installation

Source: https://rocm.docs.amd.com/en/latest/conceptual/ai-pytorch-inception.html

Print the installed TensorFlow version to confirm successful installation.

```python
print(tf._version__) r
```

--------------------------------

### Install pipx

Source: https://rocm.docs.amd.com/en/latest/how-to/system-optimization/rdna3-5.html

Installs pipx, a tool for managing Python applications. Ensure pipx is installed before proceeding with AMD debug tools.

```bash
sudo apt install pipx
pipx ensurepath
```

--------------------------------

### Run Fine-tuning Benchmark (Qwen3-32)

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/training/benchmark-docker/pytorch-training.html

Execute this command for the Qwen3-32 model fine-tuning benchmark. Supports LoRA fine-tuning with BF16.

```bash
./pytorch_benchmark_report.sh -t $training_mode \
    -m Qwen3-32 \
    -p $datatype \
    -s $sequence_length

```

--------------------------------

### Run Fine-tuning Benchmark (Qwen2.5-32B)

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/training/benchmark-docker/pytorch-training.html

Initiate the fine-tuning benchmark for the Qwen2.5-32B model. This command supports LoRA fine-tuning with BF16.

```bash
./pytorch_benchmark_report.sh -t $training_mode \
    -m Qwen2.5-32B \
    -p $datatype \
    -s $sequence_length

```

--------------------------------

### Install Optimum-AMD

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference/hugging-face-models.html

Install the Optimum-AMD library for interfacing Hugging Face libraries with the ROCm software stack. Use pip for installation.

```bash
pip install --upgrade --upgrade-strategy eager optimum[amd]

```

--------------------------------

### Run Qwen 2.5 7B Pretraining Benchmark (MI300X)

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/training/benchmark-docker/primus-megatron.html

Execute a pretraining benchmark for Qwen 2.5 7B on MI300X. Includes performance tuning variables.

```bash
# Set the variables for better performance
# only on MI325X and MI300X
export PRIMUS_TURBO_ATTN_V3_ATOMIC_FP32=1
export NVTE_CK_IS_V3_ATOMIC_FP32=1

bash runner/primus-cli direct \
  --log_file /tmp/primus_qwen2.5_7B.log \
  -- train pretrain \
  --config examples/megatron/configs/MI300X/qwen2.5_7B-BF16-pretrain.yaml
```

--------------------------------

### Verify bitsandbytes Installation

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/model-quantization.html

After installation, use `pip show bitsandbytes` to confirm the package details. The output should display the installed version information.

```bash
pip show bitsandbytes

```

--------------------------------

### Build Documentation (Windows)

Source: https://rocm.docs.amd.com/en/latest/contribute/building.html

Builds the HTML documentation using Sphinx from the project root on Windows. Ensure the virtual environment is activated and dependencies are installed.

```powershell
.venv\Scripts\python.exe -m sphinx -T -E -b html -d _build/doctrees -D language=en docs _build/html
```

--------------------------------

### Install AutoGPTQ from Source for ROCm

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference/hugging-face-models.html

Install AutoGPTQ from source for AMD GPUs supporting ROCm. Set the ROCM_VERSION environment variable before installation.

```bash
ROCM_VERSION=6.1 pip install -vvv --no-build-isolation -e .

```

--------------------------------

### Setup Multi-node Training Environment

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/training/benchmark-docker/primus-megatron.html

Clone the Primus repository, checkout a specific commit, update submodules, and set necessary environment variables for multi-node training. This includes Docker image, Hugging Face token, and network interface configurations.

```bash
git clone --recurse-submodules https://github.com/AMD-AGI/Primus.git
cd Primus/
git checkout 44f780d
git submodule update --init --recursive
export DOCKER_IMAGE=rocm/primus:v26.2
export HF_TOKEN=<your_HF_token>
export NCCL_IB_HCA=<your_NCCL_IB_HCA> # specify which RDMA interfaces to use for communication
export NCCL_SOCKET_IFNAME=<your_NCCL_SOCKET_IFNAME> # your Network Interface
export GLOO_SOCKET_IFNAME=<your_GLOO_SOCKET_IFNAME> # your Network Interface
export NCCL_IB_GID_INDEX=3 # Set InfiniBand GID index for NCCL communication. Default is 3 for ROCE

# Set the variables for better performance
# only on MI325X and MI300X
export PRIMUS_TURBO_ATTN_V3_ATOMIC_FP32=1
export NVTE_CK_IS_V3_ATOMIC_FP32=1
```

--------------------------------

### Install AutoGPTQ from Pip

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/model-quantization.html

Install the latest stable release of the AutoGPTQ library from pip. This command installs pre-built wheels for a specific ROCm version.

```bash
# This will install pre-built wheel for a specific ROCm version.

pip install auto-gptq --no-build-isolation --extra-index-url https://huggingface.github.io/autogptq-index/whl/rocm573/


```

--------------------------------

### Install AMD GPU Driver (amdgpu)

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference/benchmark-docker/vllm-mori-distributed.html

Installs the AMD GPU Driver (ROCm 7.0.2) on Ubuntu 24.04. Ensure you have the correct kernel headers and modules installed.

```bash
wget https://repo.radeon.com/amdgpu-install/7.0.2/ubuntu/noble/amdgpu-install_7.0.2.70002-1_all.deb
sudo apt install ./amdgpu-install_7.0.2.70002-1_all.deb
sudo apt update
sudo apt install "linux-headers-$(uname -r)" "linux-modules-extra-$(uname -r)"
sudo apt install amdgpu-dkms
```

--------------------------------

### Install bitsandbytes for ROCm

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/model-quantization.html

Follow these commands to clone the bitsandbytes repository, checkout the ROCm-enabled branch, install dependencies, configure the build with target GPU architecture, compile, and install the library.

```bash
# Clone the github repo
git clone --recurse https://github.com/ROCm/bitsandbytes.git
cd bitsandbytes
git checkout rocm_enabled_multi_backend

# Install dependencies
pip install -r requirements-dev.txt

# Use -DBNB_ROCM_ARCH to specify target GPU arch
cmake -DBNB_ROCM_ARCH="gfx942" -DCOMPUTE_BACKEND=hip -S .

# Compile the project
make

# Install
python setup.py install

```

--------------------------------

### Install ROCm Validation Suite (RVS)

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/system-setup/system-health-check.html

Installs the ROCm Validation Suite on Ubuntu systems with ROCm already installed. This suite includes tests for GPU stress and memory bandwidth.

```bash
sudo apt update
sudo apt install rocm-validation-suite
```

--------------------------------

### Setup Primus Multi-node Training Environment

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/training/benchmark-docker/primus-megatron.html?model=primus_pyt_megatron_lm_train_llama-3.3-70b

Clone the Primus repository, checkout a specific commit, update submodules, and set essential Docker and network environment variables for multi-node training.

```bash
git clone --recurse-submodules https://github.com/AMD-AGI/Primus.git
cd Primus/
git checkout 44f780d
git submodule update --init --recursive
export DOCKER_IMAGE=rocm/primus:v26.2
export HF_TOKEN=<your_HF_token>
export NCCL_IB_HCA=<your_NCCL_IB_HCA> # specify which RDMA interfaces to use for communication
export NCCL_SOCKET_IFNAME=<your_NCCL_SOCKET_IFNAME> # your Network Interface
export GLOO_SOCKET_IFNAME=<your_GLOO_SOCKET_IFNAME> # your Network Interface
export NCCL_IB_GID_INDEX=3 # Set InfiniBand GID index for NCCL communication. Default is 3 for ROCE
```

--------------------------------

### Run Qwen 2.5 7B FP8 Pretraining Benchmark (MI355X/MI350X)

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/training/benchmark-docker/primus-megatron.html

Initiate a pretraining benchmark for Qwen 2.5 7B using FP8 precision on MI355X/MI350X accelerators.

```bash
bash runner/primus-cli direct \
  --log_file /tmp/primus_qwen2.5_7B_fp8.log \
  -- train pretrain \
  --config examples/megatron/configs/MI355X/qwen2.5_7B-FP8-pretrain.yaml
```

--------------------------------

### Install Flash Attention 2 with CK backend

Source: https://rocm.docs.amd.com/en/latest/how-to/llm-fine-tuning-optimization/model-acceleration-libraries.html

Installs Flash Attention 2 using the Composable Kernel (CK) backend, which is the default. Ensure you have cloned the repository and installed ninja.

```bash
git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention/
pip install ninja

# To install the CK backend flash attention
python setup.py install
```

--------------------------------

### Example Output During Distributed Fine-Tuning

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/fine-tuning/multi-gpu-fine-tuning-and-inference.html

This output indicates successful initialization of distributed training components, including FSDP, model instantiation, and dataset loading. It shows progress with loss information.

```text
INFO:torchtune.utils.logging:FSDP is enabled. Instantiating Model on CPU for Rank 0 ...
INFO:torchtune.utils.logging:Model instantiation took 7.32 secs
INFO:torchtune.utils.logging:Memory Stats after model init:
{'peak_memory_active': 9.478172672, 'peak_memory_alloc': 8.953868288, 'peak_memory_reserved': 11.112808448}
INFO:torchtune.utils.logging:Optimizer and loss are initialized.
INFO:torchtune.utils.logging:Dataset and Sampler are initialized.
INFO:torchtune.utils.logging:Learning rate scheduler is initialized.
1|111|Loss: 1.5790324211120605:   7%|█                                          | 114/1618

```

--------------------------------

### Install AMD GPU Driver (amdgpu) ROCm 7.1.1

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference/benchmark-docker/sglang-mori-distributed.html

Installs the AMD GPU Driver (amdgpu) for ROCm 7.1.1 on Ubuntu 22.04. This includes installing kernel headers and modules required for the driver.

```bash
wget https://repo.radeon.com/amdgpu-install/7.1.1/ubuntu/jammy/amdgpu-install_7.1.1.70101-1_all.deb
sudo apt install ./amdgpu-install_7.1.1.70101-1_all.deb
sudo apt update
sudo apt install "linux-headers-$(uname -r)" "linux-modules-extra-$(uname -r)"
sudo apt install amdgpu-dkms

```

--------------------------------

### Start Primus Docker Container

Source: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/training/benchmark-docker/primus-pytorch.html?model=primus_pyt_train_llama-3.1-8b

Starts a previously stopped Docker container named 'training_env'.

```bash
docker start training_env
```
Tokens:
Raw