### Install from Local Clone
Source: https://github.com/vllm-project/guidellm/blob/main/docs/getting-started/install.md
Install GuideLLM from a local clone of the repository by navigating to the directory and running pip install.
```bash
pip install .
```
--------------------------------
### Install Dependencies
Source: https://github.com/vllm-project/guidellm/blob/main/DEVELOPING.md
Run this command to install all necessary project dependencies before starting development.
```bash
npm install
```
--------------------------------
### Install from Source (Main Branch)
Source: https://github.com/vllm-project/guidellm/blob/main/docs/getting-started/install.md
Install the latest development version of GuideLLM by cloning and installing directly from the main branch on GitHub.
```bash
pip install git+https://github.com/vllm-project/guidellm.git
```
--------------------------------
### Install GuideLLM with Vision Support
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/multimodal/video.md
Command to install GuideLLM with the necessary components for vision and recommended features.
```bash
pip install guidellm[vision,recommended]
```
--------------------------------
### Install Latest Release from PyPI
Source: https://github.com/vllm-project/guidellm/blob/main/docs/getting-started/install.md
Install the latest stable release of GuideLLM with recommended dependencies using pip.
```bash
pip install guidellm[recommended]
```
--------------------------------
### Install GuideLLM with Audio Support
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/multimodal/audio.md
Install GuideLLM with the necessary audio and recommended dependencies. This command should be run on the same instance as your server or another machine that can reach it.
```bash
pip install guidellm[audio,recommended]
```
--------------------------------
### Install vLLM
Source: https://github.com/vllm-project/guidellm/blob/main/docs/getting-started/server.md
Install the vLLM library using pip. This is the first step to setting up the recommended server backend.
```bash
pip install vllm
```
--------------------------------
### Install GuideLLM with Multimodal Extras
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/multimodal/index.md
Install GuideLLM with the necessary extras for all multimodal features or specific modalities like vision or audio.
```bash
pip install guidellm[vision,audio]
```
```bash
pip install guidellm[vision]
```
```bash
pip install guidellm[audio]
```
--------------------------------
### Install GuideLLM Dependencies
Source: https://github.com/vllm-project/guidellm/blob/main/docs/examples/practice_on_vllm_simulator.md
Clones the GuideLLM repository and installs the package using pip. Ensure you are in the correct directory after cloning.
```bash
git clone https://github.com/vllm-project/guidellm.git
cd guidellm
pip install guidellm
```
--------------------------------
### Install from Local Clone (Editable Mode)
Source: https://github.com/vllm-project/guidellm/blob/main/docs/getting-started/install.md
Install GuideLLM from a local clone in editable mode for development. Changes to the source code are reflected immediately without reinstallation.
```bash
pip install -e .
```
--------------------------------
### Install from Specific Git Branch
Source: https://github.com/vllm-project/guidellm/blob/main/docs/getting-started/install.md
Install GuideLLM from a specific branch (e.g., 'feature-branch') by providing the branch name in the git+https URL.
```bash
pip install git+https://github.com/vllm-project/guidellm.git@feature-branch
```
--------------------------------
### Install Specific Version from PyPI
Source: https://github.com/vllm-project/guidellm/blob/main/docs/getting-started/install.md
Install a specific version of GuideLLM by specifying the version number during pip installation.
```bash
pip install guidellm==0.2.0
```
--------------------------------
### Start Development Server
Source: https://github.com/vllm-project/guidellm/blob/main/DEVELOPING.md
Starts the local development server for the Next.js UI. Access the UI at http://localhost:3000.
```bash
npm run dev
```
--------------------------------
### Verify GuideLLM Installation
Source: https://github.com/vllm-project/guidellm/blob/main/docs/examples/practice_on_vllm_simulator.md
Checks if the GuideLLM installation was successful by running the --help command. This confirms the CLI is accessible.
```bash
guidellm --help
```
--------------------------------
### Example GuideLLM Dataset Preprocessing
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/datasets.md
An example of preprocessing a dataset, specifying input and output paths, the 'gpt2' processor, and configuration for prompt and output token sizes.
```bash
guidellm preprocess dataset \
"path/to/input_dataset.jsonl" \
"path/to/processed_dataset.jsonl" \
--processor "gpt2" \
--config "prompt_tokens=512,output_tokens=256,prefix_tokens_max=100"
```
--------------------------------
### Install Development Dependencies
Source: https://github.com/vllm-project/guidellm/blob/main/DEVELOPING.md
Install the necessary dependencies for the GuideLLM package in editable mode, including development-specific tools. The `-e` flag allows for live code changes without reinstallation.
```bash
pip install -e ./[dev]
```
--------------------------------
### Serve Qwen3-VL Video Model with vLLM
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/multimodal/video.md
Command to start vLLM serving the Qwen3-VL model for Video QA and Captioning tasks.
```bash
vllm serve Qwen/Qwen3-VL-2B-Instruct
```
--------------------------------
### Clone GuideLLM Repository
Source: https://github.com/vllm-project/guidellm/blob/main/CONTRIBUTING.md
Clone the official GuideLLM repository to your local machine to start contributing.
```bash
git clone https://github.com/vllm-project/guidellm.git
cd guidellm
```
--------------------------------
### Upgrade Pip
Source: https://github.com/vllm-project/guidellm/blob/main/docs/getting-started/install.md
Ensure you have the latest version of pip installed before proceeding with GuideLLM installation.
```bash
python -m pip install --upgrade pip
```
--------------------------------
### Start vLLM Server
Source: https://github.com/vllm-project/guidellm/blob/main/docs/getting-started/server.md
Start a vLLM server with a quantized Llama 3.1 8B model. This command launches an OpenAI-compatible server at http://localhost:8000.
```bash
vllm serve "neuralmagic/Meta-Llama-3.1-8B-Instruct-quantized.w4a16"
```
--------------------------------
### Start llama.cpp Server
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/backends.md
Initialize a llama.cpp server with a specified model and context size. An alias is used for model metadata retrieval.
```bash
llama-server -hf ggml-org/gpt-oss-20b-GGUF --alias gpt-oss-20b --ctx-size 0 --jinja -ub 2048 -b 2048
```
--------------------------------
### Install Pre-Commit Hooks
Source: https://github.com/vllm-project/guidellm/blob/main/DEVELOPING.md
Run this command to install pre-commit hooks, which automate quality checks like linting and formatting before each commit. This ensures code quality is maintained.
```bash
pre-commit install
```
--------------------------------
### Serve vLLM with a Conversational Model
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/multiturn.md
Start a vLLM inference server with a conversational model. Ensure the server is accessible by GuideLLM.
```bash
vllm serve meta-llama/Llama-3.1-8B-Instruct
```
--------------------------------
### Start Ultravox Audio Chat Server
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/multimodal/audio.md
Use this command to start a vLLM inference server with the Ultravox model for audio chat completions. This server is compatible with OpenAI-compatible chat/completions endpoints.
```bash
vllm serve fixie-ai/ultravox-v0_5-llama-3_2-1b
```
--------------------------------
### Benchmark with HuggingFace Dataset and Custom Processor
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/datasets.md
This example demonstrates how to run a benchmark using a dataset from HuggingFace Hub, mapping a specific column to 'prompt', and specifying a custom processor with arguments. Use this for throughput testing with external datasets.
```bash
guidellm benchmark \
--target "http://localhost:8000" \
--profile "kind=throughput" \
--max-requests 1000 \
--data "kind=huggingface,source=my/dataset" \
--data-column-mapper '{"column_mappings": {"text_column": "prompt"}}' \
--processor "path/to/processor" \
--processor-args '{"arg1": "value1"}' \
--data-sampler "random"
```
--------------------------------
### Start TGI Server with Docker
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/backends.md
Launch a Text Generation Inference (TGI) server using Docker. Configure model, GPU usage, and token limits.
```bash
docker run --gpus 1 -ti --shm-size 1g --ipc=host --rm -p 8080:80 \
-e MODEL_ID=meta-llama/Meta-Llama-3.1-8B-Instruct \
-e NUM_SHARD=1 \
-e MAX_INPUT_TOKENS=4096 \
-e MAX_TOTAL_TOKENS=6000 \
-e HF_TOKEN=$(cat ~/.cache/huggingface/token) \
ghcr.io/huggingface/text-generation-inference:2.2.0
```
--------------------------------
### Start vLLM Server with Embeddings Model
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/embeddings.md
Start a vLLM server with a specified embeddings model. This command is used to set up the endpoint for benchmarking.
```bash
# Start vLLM server with an embeddings model
vllm serve BAAI/bge-small-en-v1.5 --port 8000
```
--------------------------------
### GuideLLM Console Output: Benchmarks Metadata
Source: https://github.com/vllm-project/guidellm/blob/main/docs/getting-started/analyze.md
Displays high-level summary of the benchmark run, including server configuration, data configuration, profile arguments, and any extra metadata. Useful for verifying benchmark setup.
```text
Benchmarks Metadata
------------------
Args: {"backend_type": "openai", "target": "http://localhost:8000", "model": "Meta-Llama-3.1-8B-Instruct-quantized", ...}
Worker: {"type_": "generative", "backend_type": "openai", "backend_args": {"timeout": 120.0, ...}, ...}
Request Loader: {"type_": "generative", "data_args": {"prompt_tokens": 256, "output_tokens": 128, ...}, ...}
Extras: {}
```
--------------------------------
### Benchmark Advanced Prefix Distribution
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/multiturn.md
This example demonstrates benchmarking with advanced prefix distributions using JSON configuration. It allows for weighted distribution of different prefix lengths and counts, running for a specified duration.
```bash
guidellm benchmark run \
--target "http://localhost:8000" \
--model "meta-llama/Llama-3.1-8B-Instruct" \
--request-format /v1/chat/completions \
--profile kind=constant \
--rate 1.5 \
--max-seconds 60 \
--data '{
"kind": "synthetic_text",
"prompt_tokens": 180,
"output_tokens": 90,
"turns": 3,
"prefix_buckets": [
{"bucket_weight": 60, "prefix_count": 5, "prefix_tokens": 100},
{"bucket_weight": 40, "prefix_count": 1, "prefix_tokens": 0}
]
}'
```
--------------------------------
### Start Whisper ASR/Translation Server
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/multimodal/audio.md
Use this command to start a vLLM inference server with a Whisper model for ASR and translation tasks. Ensure the model is compatible with OpenAI-compatible servers.
```bash
vllm serve openai/whisper-small
```
--------------------------------
### Preprocessing with Token Distribution and Short Prompt Strategy
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/datasets.md
This example shows how to preprocess a dataset with specified token count distributions (mean and standard deviation) and a strategy for handling short prompts by concatenation. A random seed is used for reproducibility.
```bash
guidellm preprocess dataset \
"dataset.jsonl" \
"processed.jsonl" \
--processor "gpt2" \
--config "prompt_tokens=512,prompt_tokens_stdev=50,output_tokens=256,output_tokens_stdev=25" \
--short-prompt-strategy "concatenate" \
--concat-delimiter "\n\n" \
--random-seed 123
```
--------------------------------
### Run All Tests with Tox
Source: https://github.com/vllm-project/guidellm/blob/main/AGENTS.md
Execute all defined test suites using tox. Ensure `tox` is installed and configured.
```bash
# Run all tests
tox -e tests
```
--------------------------------
### Text File Dataset Example
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/datasets.md
Each line in a text file represents a separate prompt. Suitable for simple prompt lists.
```text
Hello, how are you?
What is your name?
```
--------------------------------
### Start vLLM Simulator API
Source: https://github.com/vllm-project/guidellm/blob/main/docs/examples/practice_on_vllm_simulator.md
Launches the vLLM simulator in a Docker container, exposing an OpenAI-compatible API. Specify the model and LoRA modules to load.
```bash
docker pull ghcr.io/llm-d/llm-d-inference-sim:v0.4.0
docker run --rm --publish 8000:8000 \
ghcr.io/llm-d/llm-d-inference-sim:v0.4.0 \
--port 8000 \
--model "Qwen/Qwen2.5-1.5B-Instruct" \
--lora-modules '{"name":"tweet-summary-0"}' '{"name":"tweet-summary-1"}'
```
--------------------------------
### Example JSONL File for Multiturn Data
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/multiturn.md
This is an example of a JSONL file format for multiturn conversations. Each line represents a conversation with optional prefixes and turn-based prompts and output token counts.
```json
{"prefix": "You are a helpful assistant.", "prompt_0": "What is Python?", "output_tokens_count_0": 50, "prompt_1": "How do I install it?", "output_tokens_count_1": 40}
{"prefix": "You are a coding expert.", "prompt_0": "Explain functions", "output_tokens_count_0": 60, "prompt_1": "Give me an example", "output_tokens_count_1": 45}
```
--------------------------------
### Start vLLM Server for Geospatial Models
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/geospatial.md
Serves the Prithvi geospatial model using vLLM. Requires specific flags for multimodal embeddings and IO processor plugins.
```bash
vllm serve \
ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11 \
--enforce-eager \
--skip-tokenizer-init \
--enable-mm-embeds \
--io-processor-plugin \
terratorch_segmentation
```
--------------------------------
### Map Video Column from Second Dataset
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/multimodal/video.md
Example of mapping a video column from a specific dataset (index 1) using --data-column-mapper.
```bash
--data-column-mapper '{"column_mappings": {"1.video_column": "url"}}'
```
--------------------------------
### Specify Output Formats
Source: https://github.com/vllm-project/guidellm/blob/main/docs/getting-started/analyze.md
Use the `--outputs` argument to specify which formats to generate. This example generates JSON and CSV outputs.
```bash
guidellm benchmark --target "http://localhost:8000" --outputs json csv
```
--------------------------------
### Example JSON for Assistant Turn with Reasoning
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/multiturn.md
Illustrates the JSON structure for an assistant's turn when reasoning is included in the conversation history using think tags.
```json
{"role": "assistant", "content": "step-by-step reasoning...Final answer."}
```
--------------------------------
### Trace Replay Benchmarking Example
Source: https://github.com/vllm-project/guidellm/blob/main/docs/getting-started/benchmark.md
Benchmark by replaying trace events from a JSONL file. This provides realistic load testing by using actual request timings and token lengths.
```json
{"timestamp": 1234500.0, "input_length": 256, "output_length": 128}
{"timestamp": 1234500.5, "input_length": 512, "output_length": 64}
```
--------------------------------
### Benchmark with In-Memory List Dataset
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/datasets.md
Example of benchmarking guidellm using an in-memory dataset provided as a list of dictionaries. The `benchmark_generative_text` function accepts the data directly. Ensure the data format is consistent.
```python
from guidellm.benchmark import benchmark_generative_text
data = [
{"prompt": "Hello", "output": "Hi"},
{"prompt": "How are you?", "output": "I'm fine."}
]
benchmark_generative_text(data=data, ...)
```
--------------------------------
### Run Benchmark with TurnPivot Preprocessor
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/multiturn.md
Use the `guidellm benchmark run` command with multiple `--data` arguments and the `turn_pivot` preprocessor to create synthetic datasets for multiturn conversations. Each `--data` argument defines a turn with potentially different distributions.
```bash
guidellm benchmark run \
--target "http://localhost:8000" \
--model "meta-llama/Llama-3.1-8B-Instruct" \
--request-format /v1/chat/completions \
--profile kind=concurrent \
--rate 10 \
--max-requests 150 \
--data "kind=synthetic_text,prefix_tokens=512,prompt_tokens=128,output_tokens=256" \
--data "kind=synthetic_text,prompt_tokens=256,prompt_token_stdev=32,output_tokens=128" \
--data "kind=synthetic_text,prompt_tokens=64,output_tokens=128,output_tokens_stdev=16" \
--data-preprocessors "turn_pivot"
```
--------------------------------
### GuideLLM Preprocessing Config: Key-Value Pairs
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/datasets.md
Configure dataset preprocessing using simple key=value pairs for token sizes and prefix limits.
```bash
--config "prompt_tokens=512,output_tokens=256,prefix_tokens_max=100"
```
--------------------------------
### GuideLLM Preprocessing Config: File Path
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/datasets.md
Configure dataset preprocessing by referencing a configuration file containing token size and prefix limit settings.
```bash
--config "path/to/config.json"
```
--------------------------------
### Using TurnPivot in CLI
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/multiturn.md
Shows how to configure the TurnPivot preprocessor when running GuideLLM from the command line. Ensure to specify all necessary preprocessors as `--data-preprocessors` overrides others.
```bash
--data "kind=huggingface,source=dataset0.jsonl" \
--data "kind=huggingface,source=dataset1.jsonl" \
--data-preprocessors "encode_media,turn_pivot"
```
--------------------------------
### Benchmark Run with Synthetic Data and Tool Calls
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/tool_calling.md
Run a GuideLLM benchmark using synthetic data with a specified number of tool call turns. This configures the benchmark to generate data with 2 tool call turns.
```bash
guidellm benchmark run \
--target "http://localhost:8000" \
--model "Qwen/Qwen3-0.6B" \
--request-format /v1/chat/completions \
--data '{"prompt_tokens": 200, "output_tokens": 100, "turns": 3, "tool_call_turns": 2}' \
--max-requests 30 \
--profile kind=constant \
--rate 1
```
--------------------------------
### Map Video Column in Dataset
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/multimodal/video.md
Example of using --data-column-mapper to specify the video column in a dataset, mapping it to 'url'.
```bash
--data-column-mapper '{"column_mappings": {"video_column": "url"}}'
```
--------------------------------
### Include Extra Body Parameters in Request
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/multimodal/video.md
Example of using --request-formatter-kwargs to add extra arguments, such as 'temperature', to the request body.
```bash
--request-formatter-kwargs '{"extras": {"body": {"temperature": 0.7}}}'
```
--------------------------------
### Build for Production
Source: https://github.com/vllm-project/guidellm/blob/main/DEVELOPING.md
Generates a production-ready build of the UI, outputting files to the 'out' directory.
```bash
npm run build
```
--------------------------------
### JSON Lines File Dataset Example
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/datasets.md
Each line in a JSON Lines file is a JSON object. The object must contain a 'prompt' field.
```json
{"prompt": "Hello, how are you?", "output_tokens_count": 5, "additional_column": "foo", "additional_column2": "bar"}
{"prompt": "What is your name?", "output_tokens_count": 3, "additional_column": "baz", "additional_column2": "qux"}
```
--------------------------------
### GuideLLM Preprocessing Config: JSON String
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/datasets.md
Configure dataset preprocessing using a JSON string for token sizes and prefix limits.
```bash
--config '{"prompt_tokens": 512, "output_tokens": 256, "prefix_tokens_max": 100}'
```
--------------------------------
### CSV File Dataset Example
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/datasets.md
CSV files use rows for entries and the first row for column names. The 'prompt' column is mandatory.
```csv
prompt,output_tokens_count,additional_column,additional_column2
Hello, how are you?,5,foo,bar
What is your name?,3,baz,qux
```
--------------------------------
### Benchmark with JSON File Dataset
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/datasets.md
Use this command to benchmark guidellm with a dataset stored in a JSON file. Replace `path/to/dataset.json` with the actual path to your file. The `--data-column-mapper` can be used to specify column mappings, such as the prompt column.
```bash
guidellm benchmark \
--target "http://localhost:8000" \
--profile "kind=throughput" \
--max-requests 1000 \
--data "kind=json_file,path=path/to/dataset.json" \
--data-column-mapper '{"column_mappings": {"column_mappings": {"text_column": "prompt"}}}'
```
--------------------------------
### Specify Output Directory
Source: https://github.com/vllm-project/guidellm/blob/main/docs/getting-started/analyze.md
Use the `--output-dir` argument to change the directory where all output files are saved. This example saves outputs to the 'results/' directory.
```bash
guidellm benchmark --target "http://localhost:8000" --output-dir results/
```
--------------------------------
### Run GuideLLM Benchmark
Source: https://github.com/vllm-project/guidellm/blob/main/README.md
Execute a GuideLLM benchmark sweep to determine maximum performance and rates. Configure the target server, profile, maximum duration, and synthetic data generation parameters.
```bash
guidellm benchmark \
--target "http://localhost:8000" \
--profile kind=sweep \
--max-seconds 30 \
--data "kind=synthetic_text,prompt_tokens=256,output_tokens=128"
```
--------------------------------
### Basic Benchmark Command
Source: https://github.com/vllm-project/guidellm/blob/main/docs/getting-started/benchmark.md
Use this command to run a benchmark against a local vLLM server with default settings. It connects to the server, generates synthetic data, and determines optimal performance points.
```bash
guidellm benchmark \
--target "http://localhost:8000" \
--data "kind=synthetic_text,prompt_tokens=256,output_tokens=128" \
--max-seconds 60
```
--------------------------------
### Run GuideLLM Container
Source: https://github.com/vllm-project/guidellm/blob/main/README.md
Execute the latest GuideLLM container from ghcr.io, mounting a results directory and setting environment variables for target, profile, duration, and data generation.
```bash
podman run \
--rm -it \
-v "./results:/results:rw" \
-e GUIDELLM_TARGET=http://localhost:8000 \
-e GUIDELLM_PROFILE=sweep \
-e GUIDELLM_MAX_SECONDS=30 \
-e GUIDELLM_DATA="kind=synthetic_text,prompt_tokens=256,output_tokens=128" \
ghcr.io/vllm-project/guidellm:latest
```
--------------------------------
### Configure Output Formats with Environment Variables
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/outputs.md
When using environment variables, specify the desired output formats as a comma-separated string for the GUIDELLM_OUTPUTS variable. This is useful in containerized environments.
```bash
-e GUIDELLM_OUTPUTS=json,csv,html
```
--------------------------------
### Combine API Key and Sampling Parameters
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/backends.md
Demonstrates how to simultaneously configure an API key and sampling parameters using the `--backend-kwargs` option. The `api_key` is set directly, while sampling parameters are passed within the `extras.body`.
```bash
guidellm benchmark \
--target "http://localhost:8000/v1" \
--model "meta-llama/Meta-Llama-3.1-8B-Instruct" \
--data "kind=synthetic_text,prompt_tokens=256,output_tokens=128" \
--backend-kwargs '{"api_key": "sk-...", "extras": {"body": {"temperature": 0.8, "top_p": 0.9}}}'
```
--------------------------------
### Audio Translation Benchmark Output
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/multimodal/audio.md
This is an example of the output generated after running the audio translation benchmark. It includes statistics on input tokens, samples, seconds, and bytes processed.
```bash
✔ OpenAIHTTPBackend backend validated with model openai/whisper-small
......
......
✔ Setup complete, starting benchmarks...
......
......
ℹ Audio Metrics Statistics (Completed Requests)
|=============|=======|========|========|========|=========|=========|==========|==========|======|=======|======|======|=========|==========|==========|==========|
| Benchmark | Input Tokens |||| Input Samples |||| Input Seconds |||| Input Bytes ||||
| Strategy | Per Request || Per Second || Per Request || Per Second || Per Request || Per Second || Per Request || Per Second ||
| | Mdn | p95 | Mdn | Mean | Mdn | p95 | Mdn | Mean | Mdn | p95 | Mdn | Mean | Mdn | p95 | Mdn | Mean |
|-------------|-------|--------|--------|--------|---------|---------|----------|----------|------|-------|------|------|---------|----------|----------|----------|
| synchronous | 642.0 | 1688.0 | 7483.6 | 7563.5 | 16000.0 | 16000.0 | 133404.0 | 146385.0 | 6.4 | 16.8 | 74.5 | 75.2 | 52172.0 | 135692.0 | 603620.5 | 611706.4 |
|=============|=======|========|========|========|=========|=========|==========|==========|======|=======|======|======|=========|==========|==========|==========|
......
......
✔ Benchmarking complete, generated 1 benchmark(s)
… json : /Users/markkurtz/code/github/vllm-project/guidellm/benchmarks.json
… csv : /Users/markkurtz/code/github/vllm-project/guidellm/benchmarks.csv
… html : /Users/markkurtz/code/github/vllm-project/guidellm/benchmarks.html
```
--------------------------------
### Benchmark Chat Completions API
Source: https://github.com/vllm-project/guidellm/blob/main/README.md
Configures the benchmark to test the chat completions API using a custom JSON dataset file. GuideLLM automatically formats requests for the chat completions schema.
```bash
guidellm benchmark \
--target http://localhost:8000 \
--request-type chat_completions \
--data "kind=json_file,path=path/to/data.json"
```
--------------------------------
### JSON File Dataset Example
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/datasets.md
JSON files contain a JSON array of objects nested under a specific key. The `--data-column-mapper` argument is required to specify this key.
```json
{
"version": "1.0",
"data": [
{"prompt": "Hello, how are you?", "output_tokens_count": 5, "additional_column": "foo", "additional_column2": "bar"},
{"prompt": "What is your name?", "output_tokens_count": 3, "additional_column": "baz", "additional_column2": "qux"}
]
}
```
--------------------------------
### Tagging Tests with Jest Runner Groups
Source: https://github.com/vllm-project/guidellm/blob/main/DEVELOPING.md
Example of how to tag test files using Jest Runner Groups in a docblock. This helps categorize tests for selective execution.
```javascript
/**
* Admin dashboard tests
*
* @group smoke
* @group sanity
* @group regression
*/
```
--------------------------------
### Download Tokenizer
Source: https://github.com/vllm-project/guidellm/blob/main/docs/examples/practice_on_vllm_simulator.md
Instructions to download tokenizer files for a specific model. These files are necessary for the benchmark process.
```bash
ls ./Qwen2.5-1.5B-Instruct
merges.txt tokenizer.json tokenizer_config.json vocab.json
```
--------------------------------
### Run GuideLLM Benchmark
Source: https://github.com/vllm-project/guidellm/blob/main/docs/examples/practice_on_vllm_simulator.md
Executes a benchmark test using the guidellm CLI. Specify the target API endpoint, model, tokenizer path, and benchmark parameters.
```bash
guidellm benchmark \
--target "http://localhost:8000/" \
--model "tweet-summary-0" \
--processor "${local_path}/Qwen2.5-1.5B-Instruct" \
--profile kind=sweep \
--max-seconds 10 \
--max-requests 10 \
--data "kind=synthetic_text,prompt_tokens=128,output_tokens=56"
```
--------------------------------
### Include Extra Arguments for Audio Requests
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/multimodal/audio.md
Use the `extras` argument within `--request-formatter-kwargs` to include endpoint-specific parameters. This example sets the language to French for an audio translation request.
```bash
--request-formatter-kwargs '{"extras": {"body": {"language": "fr"}}}'
```
--------------------------------
### Clone Forked GuideLLM Repository
Source: https://github.com/vllm-project/guidellm/blob/main/CONTRIBUTING.md
Clone your forked GuideLLM repository to your local machine if you plan to contribute changes back.
```bash
git clone https://github.com//guidellm.git
cd guidellm
```
--------------------------------
### Run GuideLLM with Environment Variables in Docker
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/outputs.md
This command runs GuideLLM within a Docker container, configuring benchmark parameters and output formats (JSON, CSV, HTML) via environment variables. Results are saved to a mounted volume.
```bash
podman run --rm -it --network=host \
-v "/tmp/results:/results:z" \
-e GUIDELLM_TARGET=http://localhost:8000 \
-e GUIDELLM_PROFILE=sweep \
-e GUIDELLM_MAX_SECONDS=30 \
-e GUIDELLM_DATA="kind=synthetic_text,prompt_tokens=256,output_tokens=128" \
-e GUIDELLM_OUTPUT_DIR=/results \
-e GUIDELLM_OUTPUTS=json,csv,html \
-e GUIDELLM_SAMPLE_REQUESTS=20 \
ghcr.io/vllm-project/guidellm:latest
```
--------------------------------
### Configure Audio Encoder for Chat Completions
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/multimodal/audio.md
For chat completion tasks, audio encoding arguments must be nested under an `audio` key within `encode_kwargs`. This example truncates audio to 30 seconds.
```bash
--request-formatter-kwargs '{"encode_kwargs": {"audio": {"max_duration": 30.0}}}'
```
--------------------------------
### Specify Output File Names
Source: https://github.com/vllm-project/guidellm/blob/main/docs/getting-started/analyze.md
The `--outputs` argument can also accept full file names to customize output paths and names. This example saves JSON and CSV to specific files.
```bash
guidellm benchmark --target "http://localhost:8000" --outputs results/benchmarks.json results/summary.csv
```
--------------------------------
### Run Production-like Benchmark
Source: https://github.com/vllm-project/guidellm/blob/main/AGENTS.md
Execute a production-like benchmark with a specific dataset, rate limits, duration, and output files. Requires an active model server.
```bash
# Production-like benchmark with specific dataset
uv run guidellm benchmark run \
--target http://localhost:8000 \
--profile kind=constant \
--rate 10,20 \
--data "kind=huggingface,source=openai/gsm8k" \
--max-seconds 300 \
--outputs "benchmark.json,report.csv"
```
--------------------------------
### Map Audio Data Column from Second Dataset
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/multimodal/audio.md
When combining datasets, prepend the column name with the dataset index (starting at 0) or alias followed by a dot to map the audio column from a specific dataset.
```bash
--data-column-mapper '{"column_mappings": {"1.audio_column": "speech_data"}}'
```
--------------------------------
### Configure Constant Benchmark Parameters
Source: https://github.com/vllm-project/guidellm/blob/main/docs/getting-started/benchmark.md
Configure the constant benchmark with `rate` and `rampup_duration`. Multiple rates can be specified for successive streams.
```bash
guidellm benchmark --profile '{"kind": "constant","rate": [16,32]}'
```
```bash
guidellm benchmark --profile '{"kind": "constant","rate": [16,32],"rampup_duration": 10}'
```
--------------------------------
### Basic GuideLLM Dataset Preprocessing Command
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/datasets.md
Use this command to preprocess datasets. Specify the input data, output path, processor, and configuration for token size adjustments.
```bash
guidellm preprocess dataset \
\
\
--processor \
--config
```
--------------------------------
### Audio Transcription Benchmark Output
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/multimodal/audio.md
This is an example of the output generated after running the audio transcription benchmark. It includes statistics on input tokens, samples, seconds, and bytes, categorized by median, p95, and mean values.
```bash
✔ OpenAIHTTPBackend backend validated with model openai/whisper-small
......
......
✔ Setup complete, starting benchmarks...
......
......
ℹ Audio Metrics Statistics (Completed Requests)
|=============|=======|========|========|========|=========|=========|==========|==========|======|=======|======|======|=========|==========|==========|==========|
| Benchmark | Input Tokens |||| Input Samples |||| Input Seconds |||| Input Bytes ||||
| Strategy | Per Request || Per Second || Per Request || Per Second || Per Request || Per Second || Per Request || Per Second ||
| | Mdn | p95 | Mdn | Mean | Mdn | p95 | Mdn | Mean | Mdn | p95 | Mdn | Mean | Mdn | p95 | Mdn | Mean |
|-------------|-------|--------|--------|--------|---------|---------|----------|----------|------|-------|------|------|---------|----------|----------|----------|
| synchronous | 642.0 | 1688.0 | 7565.1 | 7329.1 | 16000.0 | 16000.0 | 129722.1 | 141848.5 | 6.4 | 16.8 | 75.3 | 72.9 | 52172.0 | 135692.0 | 610195.0 | 592749.4 |
|=============|=======|========|========|========|=========|=========|==========|==========|======|=======|------|------|---------|----------|----------|----------|
......
......
✔ Benchmarking complete, generated 1 benchmark(s)
… json : /Users/markkurtz/code/github/vllm-project/guidellm/benchmarks.json
… csv : /Users/markkurtz/code/github/vllm-project/guidellm/benchmarks.csv
… html : /Users/markkurtz/code/github/vllm-project/guidellm/benchmarks.html
```
--------------------------------
### Configure Audio Encoder for Transcription
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/multimodal/audio.md
Use `--request-formatter-kwargs` to specify audio encoding parameters for transcription tasks. This example sets the audio format to WAV and the target sample rate to 16000 Hz.
```bash
--request-formatter-kwargs '{"encode_kwargs": {"audio_format": "wav", "encode_sample_rate": 16000}}'
```
--------------------------------
### Run Benchmark with OpenAI HTTP Backend
Source: https://github.com/vllm-project/guidellm/blob/main/docs/guides/geospatial.md
Executes a benchmark against a vLLM server using the OpenAI HTTP backend. Specify the model, data source, and request format.
```bash
guidellm benchmark \
--target http://localhost:8000 \
--model ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11 \
--backend openai_http \
--data "kind=huggingface,source=flood_detection_dataset.jsonl" \
--request-format /pooling \
--data-column-mapper pooling_column_mapper \
--max-requests 100 \
--output-path results.json
```
--------------------------------
### Benchmark Different Models
Source: https://github.com/vllm-project/guidellm/blob/main/docs/getting-started/analyze.md
Run benchmarks to compare the performance of different models or hardware configurations. Specify the target server and an output directory for results.
```bash
guidellm benchmark --target "http://server1:8000" --output-dir model1/
```
```bash
guidellm benchmark --target "http://server2:8000" --output-dir model2/
```
--------------------------------
### Run Replay Benchmark Profile
Source: https://github.com/vllm-project/guidellm/blob/main/docs/getting-started/benchmark.md
Use the replay profile to benchmark by replaying trace events with timestamps from a synthetic dataset. Configure the time scale for intervals between trace events.
```bash
guidellm benchmark --profile kind=replay
```