AMD ROCm

https://rocm.docs.amd.com/en/docs-6.4.3
Admin
AMD ROCm is an open-source software platform optimized for HPC and AI workloads on AMD Instinct...
Tokens:438,011
Snippets:6,344
Trust Score:10
Update:3 months ago
Show doc for...
Tokens:
Raw
### Run Benchmark Setup Script

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/training/benchmark-docker/previous-versions/pytorch-training-v25.4.html

Execute the setup script to install necessary libraries and download datasets required for benchmarking.

```bash
./pytorch_benchmark_setup.sh

```

--------------------------------

### Run JAX MaxText Benchmark Setup

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/training/benchmark-docker/jax-maxtext.html?model=jax_maxtext_train_llama-3.3-70b

Execute the setup script to install necessary libraries and datasets for benchmarking. Replace `Llama-2-7B` with the desired model.

```bash
./jax-maxtext_benchmark_setup.sh -m Llama-2-7B
```

--------------------------------

### Run JAX MaxText Benchmark Setup

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/training/benchmark-docker/jax-maxtext.html

Execute the setup script to install necessary libraries and datasets for JAX MaxText benchmarking. This command installs dependencies for the Llama-2-7B model.

```bash
./jax-maxtext_benchmark_setup.sh -m Llama-2-7B

```

--------------------------------

### Setup JAX MaxText Benchmark

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/training/benchmark-docker/jax-maxtext.html?model=jax_maxtext_train_llama-3.3-70b

Execute the setup script to install necessary libraries and download datasets for benchmarking the Llama-3.1-70B model.

```bash
./jax-maxtext_benchmark_setup.sh -m Llama-3.1-70B

```

--------------------------------

### Run JAX MaxText Benchmark Setup

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/training/benchmark-docker/jax-maxtext.html?model=jax_maxtext_train_llama-3.3-70b

Executes the setup script to install necessary libraries and datasets for benchmarking the Llama-2-70B model.

```bash
./jax-maxtext_benchmark_setup.sh -m Llama-2-70B

```

--------------------------------

### Run JAX MaxText Benchmark Setup

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/training/benchmark-docker/jax-maxtext.html?model=jax_maxtext_train_llama-3.3-70b

Execute the setup script to install necessary libraries and datasets for JAX MaxText benchmarking. The `-m` flag specifies the model to prepare for.

```bash
./jax-maxtext_benchmark_setup.sh -m DeepSeek-V2-lite
```

--------------------------------

### Start Fine-tuning Benchmark

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/training/benchmark-docker/previous-versions/pytorch-training-v25.3.html

Execute this command to start a fine-tuning benchmark. This example uses Llama 2 70B with the WikiText dataset and BF16 precision.

```bash
./pytorch_benchmark_report.sh -t {finetune_fw, finetune_lora} -p BF16 -m Llama-3.1-70B
```

--------------------------------

### Run JAX MaxText Benchmark Setup Script

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/training/benchmark-docker/jax-maxtext.html

Execute the setup script to install necessary libraries and download datasets for benchmarking the Llama-3.3-70B model.

```bash
./jax-maxtext_benchmark_setup.sh -m Llama-3.3-70B

```

--------------------------------

### Run JAX MaxText Benchmark Setup Script

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/training/benchmark-docker/jax-maxtext.html?model=jax_maxtext_train_llama-3.3-70b

Execute the setup script to install necessary libraries and download datasets for JAX MaxText benchmarking. Use the `-m` flag to specify the model, e.g., Llama-3.3-70B.

```bash
./jax-maxtext_benchmark_setup.sh -m Llama-3.3-70B
```

--------------------------------

### Run JAX MaxText Benchmark Setup

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/training/benchmark-docker/jax-maxtext.html

Execute the setup script to install necessary libraries and datasets for JAX MaxText benchmarking. Replace 'DeepSeek-V2-lite' with your target model.

```bash
./jax-maxtext_benchmark_setup.sh -m DeepSeek-V2-lite

```

--------------------------------

### Run JAX MaxText Benchmark Setup Script

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/training/benchmark-docker/jax-maxtext.html?model=jax_maxtext_train_llama-3.3-70b

Execute the setup script to install required libraries and datasets for benchmarking JAX MaxText models. Specify the model to be used.

```bash
./jax-maxtext_benchmark_setup.sh -m Mixtral-8x7B

```

--------------------------------

### Install torchtune and check installation

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/fine-tuning/multi-gpu-fine-tuning-and-inference.html

Install the torchtune library using pip. Use `tune --help` to verify the installation and see available commands.

```bash
pip install torchtune
```

```bash
tune --help
```

--------------------------------

### Install Optimum-AMD from Source

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/inference/hugging-face-models.html

Install Optimum-AMD from its source repository. This involves cloning the repository, navigating into the directory, and then performing a local installation.

```bash
git clone https://github.com/huggingface/optimum-amd.git
cd optimum-amd
pip install -e .

```

--------------------------------

### Start vLLM Serving Benchmark

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/inference/benchmark-docker/vllm.html

Start the vLLM server for benchmarking with specified model and configuration parameters, optimizing for serving performance.

```bash
model=mistralai/Mixtral-8x22B-Instruct-v0.1
tp=8
dtype=auto
kv_cache_dtype=auto
max_num_seqs=256
max_seq_len_to_capture=65536
max_num_batched_tokens=65536
max_model_len=8192

vllm serve $model \
    -tp $tp \
    --dtype $dtype \
    --kv-cache-dtype $kv_cache_dtype \
    --max-num-seqs $max_num_seqs \
    --max-seq-len-to-capture $max_seq_len_to_capture \
    --max-num-batched-tokens $max_num_batched_tokens \
    --max-model-len $max_model_len \
    --no-enable-prefix-caching \
    --swap-space 16 \
    --disable-log-requests \
    --trust-remote-code \
    --gpu-memory-utilization 0.9
```

--------------------------------

### Install Documentation Dependencies

Source: https://rocm.docs.amd.com/en/docs-6.4.3/contribute/building.html

Installs the necessary Python packages for building the documentation from the requirements file within the virtual environment.

```bash
.venv/bin/python     -m pip install -r docs/sphinx/requirements.txt
```

--------------------------------

### Start vLLM Serving

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/inference/benchmark-docker/vllm.html

Start the vLLM serving engine with specified model and configuration parameters. This command is optimized for Qwen3 30B A3B.

```bash
model=Qwen/Qwen3-30B-A3B
tp=1
dtype=auto
kv_cache_dtype=auto
max_num_seqs=256
max_seq_len_to_capture=32768
max_num_batched_tokens=32768
max_model_len=8192

vllm serve $model \
    -tp $tp \
    --dtype $dtype \
    --kv-cache-dtype $kv_cache_dtype \
    --max-num-seqs $max_num_seqs \
    --max-seq-len-to-capture $max_seq_len_to_capture \
    --max-num-batched-tokens $max_num_batched_tokens \
    --max-model-len $max_model_len \
    --no-enable-prefix-caching \
    --swap-space 16 \
    --disable-log-requests \
    --trust-remote-code \
    --gpu-memory-utilization 0.9

```

--------------------------------

### Install Dependencies and Set Environment Variables

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/training/benchmark-docker/primus-megatron.html

Installs project dependencies and sets necessary environment variables for training. Ensure you are in the `/workspace/Primus` directory before running.

```bash
pip install -r requirements.txt
export HSA_NO_SCRATCH_RECLAIM=1
export NVTE_CK_USES_BWD_V3=1

```

--------------------------------

### Install RDMA Driver Packages

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/training/benchmark-docker/previous-versions/jax-maxtext-v25.5.html

Installs essential packages for building and installing the RDMA driver. Ensure these are installed on all nodes in a multi-node setup.

```bash
sudo apt install iproute2 -y
sudo apt install -y linux-headers-"$(uname-r)" libelf-dev
sudo apt install -y gcc make libtool autoconf librdmacm-dev rdmacm-utils infiniband-diags ibverbs-utils perftest ethtool libibverbs-dev rdma-core strace libibmad5 libibnetdisc5 ibverbs-providers libibumad-dev libibumad3 libibverbs1 libnl-3-dev libnl-route-3-dev

```

--------------------------------

### Install CK xFormers from Source

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/inference-optimization/model-acceleration-libraries.html

Clone the xFormers repository and install it using the provided setup script. Specify the ROCm architecture for your Instinct MI300-series GPU.

```bash
# Install from source
git clone https://github.com/ROCm/xformers.git
cd xformers/
git submodule update --init --recursive
PYTORCH_ROCM_ARCH=gfx942 python setup.py install #Instinct MI300-series

```

--------------------------------

### Find and Link ROCm Libraries (MIOpen Example)

Source: https://rocm.docs.amd.com/en/docs-6.4.3/conceptual/cmake-packages.html

Use this snippet to find and link ROCm libraries like MIOpen. Requires CMake version 3.5 or higher. This example assumes MIOpen is installed and discoverable by CMake.

```cmake
cmake_minimum_required(VERSION 3.5) # find_package(miopen) requires 3.5
cmake_policy(VERSION 3.5...3.27)
project(MyProj LANGUAGES CXX)
find_package(miopen)
add_library(MyLib ...)
target_link_libraries(MyLib PUBLIC MIOpen)
```

--------------------------------

### Example: FLUX.1-dev Pretraining Benchmark

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/training/benchmark-docker/previous-versions/pytorch-training-v25.3.html

Benchmark FLUX.1-dev with BF16 precision using FluxBenchmark for pretraining.

```bash
./pytorch_benchmark_report.sh -t pretrain -p BF16 -m Flux
```

--------------------------------

### Get Predictions from Exported Model

Source: https://rocm.docs.amd.com/en/docs-6.4.3/conceptual/ai-pytorch-inception.html

Uses the exported model to generate predictions for a list of new text examples. Ensure the model is compiled and ready for prediction.

```python
examples = [
"The movie was great!",
"The movie was okay.",
"The movie was terrible..."
]

export_model.predict(examples)

```

--------------------------------

### Example: Torchtune Full Weight Fine-tuning

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/training/benchmark-docker/previous-versions/pytorch-training-v25.3.html

Benchmark torchtune full weight fine-tuning with Llama 3.1 70B using BF16 precision.

```bash
./pytorch_benchmark_report.sh -t finetune_fw -p BF16 -m Llama-3.1-70B
```

--------------------------------

### Run GPT2 Model with Hugging Face Transformers

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/inference/hugging-face-models.html

Example of loading and running the GPT2 model using the Hugging Face Transformers library. Ensure you have the library installed.

```python
from transformers import GPT2Tokenizer, GPT2Model

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

model = GPT2Model.from_pretrained('gpt2')

text = "Replace me with any text you'd like."

encoded_input = tokenizer(text, return_tensors='pt')

output = model(**encoded_input)

```

--------------------------------

### Start vLLM Serving Benchmark

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/inference/benchmark-docker/vllm.html

Start the vLLM server for benchmarking. Configure parameters such as model, tensor parallelism, sequence lengths, and GPU memory utilization.

```bash
model=meta-llama/Llama-3.1-70B-Instruct
tp=8
dtype=auto
kv_cache_dtype=auto
max_num_seqs=256
max_seq_len_to_capture=131072
max_num_batched_tokens=131072
max_model_len=8192

vllm serve $model \
    -tp $tp \
    --dtype $dtype \
    --kv-cache-dtype $kv_cache_dtype \
    --max-num-seqs $max_num_seqs \
    --max-seq-len-to-capture $max_seq_len_to_capture \
    --max-num-batched-tokens $max_num_batched_tokens \
    --max-model-len $max_model_len \
    --no-enable-prefix-caching \
    --swap-space 16 \
    --disable-log-requests \
    --trust-remote-code \
    --gpu-memory-utilization 0.9

```

--------------------------------

### Benchmark Torchtune QLoRA Fine-tuning (Llama 3.3 70B)

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/training/benchmark-docker/previous-versions/pytorch-training-v25.5.html

Example command for torchtune QLoRA fine-tuning with Llama 3.3 70B.

```bash
./pytorch_benchmark_report.sh -t finetune_qlora -p BF16 -m Llama-3.3-70B
```

--------------------------------

### Install AutoGPTQ from Pip

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/inference-optimization/model-quantization.html

Install the latest stable release of AutoGPTQ from pip, specifying a ROCm version. This command installs pre-built wheels for a specific ROCm version.

```bash
pip install auto-gptq --no-build-isolation --extra-index-url https://huggingface.github.io/autogptq-index/whl/rocm573/
```

--------------------------------

### Throughput Benchmark Example

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/inference/benchmark-docker/previous-versions/vllm-0.8.3-20250415.html

Run the throughput benchmark for the QwQ-32B model on eight GPUs with float16 precision.

```bash
./vllm_benchmark_report.sh -s throughput -m Qwen/QwQ-32B -g 8 -d float16

```

--------------------------------

### Install AutoGPTQ from Source for ROCm

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/inference/hugging-face-models.html

Install AutoGPTQ from source for AMD accelerators supporting ROCm. Set the `ROCM_VERSION` environment variable before running the pip install command.

```bash
ROCM_VERSION=6.1 pip install -vvv --no-build-isolation -e .

```

--------------------------------

### Start Pretraining Benchmark

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/training/benchmark-docker/previous-versions/pytorch-training-v25.3.html

Use this command to initiate a pretraining benchmark. Configure options such as training mode, model repository, data type, and sequence length.

```bash
./pytorch_benchmark_report.sh -t $training_mode -m $model_repo -p $datatype -s $sequence_length
```

--------------------------------

### Install AutoGPTQ for ROCm 5.7.3

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/inference/hugging-face-models.html

Install the AutoGPTQ library with ROCm 5.7.3 support using pip. This command installs pre-built wheels and is recommended for ease of use.

```bash
pip install auto-gptq --no-build-isolation --extra-index-url https://huggingface.github.io/autogptq-index/whl/rocm573/

```

--------------------------------

### Install ROCm Validation Suite on Ubuntu

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/system-health-check.html

Installs the ROCm Validation Suite (RVS) on an Ubuntu system where ROCm is already installed. This package includes various tests and benchmarks for system validation.

```bash
sudo apt update
sudo apt install rocm-validation-suite

```

--------------------------------

### Start vLLM Serving Benchmark

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/inference/benchmark-docker/vllm.html

Command to start the vLLM server for serving benchmarks. It configures parameters related to sequence handling, caching, and GPU memory utilization.

```bash
model=meta-llama/Llama-3.1-8B-Instruct
tp=1
dtype=auto
kv_cache_dtype=auto
max_num_seqs=256
max_seq_len_to_capture=131072
max_num_batched_tokens=131072
max_model_len=8192

vllm serve $model \
    -tp $tp \
    --dtype $dtype \
    --kv-cache-dtype $kv_cache_dtype \
    --max-num-seqs $max_num_seqs \
    --max-seq-len-to-capture $max_seq_len_to_capture \
    --max-num-batched-tokens $max_num_batched_tokens \
    --max-model-len $max_model_len \
    --no-enable-prefix-caching \
    --swap-space 16 \
    --disable-log-requests \
    --trust-remote-code \
    --gpu-memory-utilization 0.9
```

--------------------------------

### Start vLLM Serving Benchmark

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/inference/benchmark-docker/vllm.html

Launch the vLLM serving process with specified model and configuration parameters. This command is used to benchmark the serving capabilities of the model, including settings for tensor parallelism, data types, sequence lengths, and GPU memory utilization.

```bash
model=amd/Llama-3.1-8B-Instruct-FP8-KV
tp=1
dtype=auto
kv_cache_dtype=fp8
max_num_seqs=256
max_seq_len_to_capture=131072
max_num_batched_tokens=131072
max_model_len=8192

vllm serve $model \
    -tp $tp \
    --dtype $dtype \
    --kv-cache-dtype $kv_cache_dtype \
    --max-num-seqs $max_num_seqs \
    --max-seq-len-to-capture $max_seq_len_to_capture \
    --max-num-batched-tokens $max_num_batched_tokens \
    --max-model-len $max_model_len \
    --no-enable-prefix-caching \
    --swap-space 16 \
    --disable-log-requests \
    --trust-remote-code \
    --gpu-memory-utilization 0.9

```

--------------------------------

### Install AMD Quark

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/inference-optimization/model-quantization.html

Install the latest release of AMD Quark using pip.

```bash
pip install amd-quark

```

--------------------------------

### Install AutoGPTQ from Source

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/inference-optimization/model-quantization.html

Install AutoGPTQ from source for a specific ROCm version. This involves cloning the repository, navigating to the directory, and installing using pip, optionally speeding up compilation by setting PYTORCH_ROCM_ARCH and ROCM_VERSION.

```bash
# Clone the source code.
git clone https://github.com/AutoGPTQ/AutoGPTQ.git
cd AutoGPTQ

# Speed up the compilation by specifying PYTORCH_ROCM_ARCH to target device.
PYTORCH_ROCM_ARCH=gfx942 ROCM_VERSION=6.1 pip install .

# Show the package after the installation

```

--------------------------------

### Start vLLM Serving Benchmark

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/inference/benchmark-docker/vllm.html

Start the vLLM server for benchmarking with specified model and configuration parameters, including tensor parallelism, sequence lengths, and GPU memory utilization.

```bash
model=meta-llama/Llama-3.1-405B-Instruct
tp=8
dtype=auto
kv_cache_dtype=auto
max_num_seqs=256
max_seq_len_to_capture=131072
max_num_batched_tokens=131072
max_model_len=8192

vllm serve $model \
    -tp $tp \
    --dtype $dtype \
    --kv-cache-dtype $kv_cache_dtype \
    --max-num-seqs $max_num_seqs \
    --max-seq-len-to-capture $max_seq_len_to_capture \
    --max-num-batched-tokens $max_num_batched_tokens \
    --max-model-len $max_model_len \
    --no-enable-prefix-caching \
    --swap-space 16 \
    --disable-log-requests \
    --trust-remote-code \
    --gpu-memory-utilization 0.9

```

--------------------------------

### Example: Llama 3.1 70B Pretraining Benchmark

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/training/benchmark-docker/previous-versions/pytorch-training-v25.3.html

Benchmark Llama 3.1 70B with BF16 precision using torchtitan for pretraining.

```bash
./pytorch_benchmark_report.sh -t pretrain -p BF16 -m Llama-3.1-70B -s 8192
```

--------------------------------

### Install KVM Hypervisor Packages on Ubuntu

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/system-optimization/w6000-v620.html

Installs necessary packages for the Kernel-based Virtual Machine (KVM) hypervisor on Ubuntu. Ensure you enable the default virtual network using `virsh net-start default` after installation.

```bash
sudo apt-get -y install qemu-kvm qemu-utils bridge-utils virt-manager gir1.2-spiceclientgtk* gir1.2-spice-client-gtk* libvirt-daemon-system dnsmasq-base
sudo virsh net-start default /*to enable Virtual network by default
```

--------------------------------

### Install Triton Flash Attention 2 from Source

Source: https://rocm.docs.amd.com/en/docs-6.4.3/how-to/rocm-for-ai/inference-optimization/model-acceleration-libraries.html

Install the OpenAI Triton implementation of Flash Attention 2 from its source repository. This involves cloning the Triton repository and installing it with specific GPU architecture support.

```bash
pip uninstall pytorch-triton-rocm triton -y
git clone https://github.com/ROCm/triton.git
cd triton/python
GPU_ARCHS=gfx942 python setup.py install #MI300 series
pip install matplotlib pandas
```

--------------------------------

### Install Sphinx Requirements

Source: https://rocm.docs.amd.com/en/docs-6.4.3/contribute/contributing.html

Installs the necessary Python packages for building the documentation locally. This command only needs to be run once.

```bash
pip3 install -r sphinx/requirements.txt  # You only need to run this command once
```
Tokens:
Raw