GuideLLM

https://vllm-project.github.io/guidellm/stable/
Admin
GuideLLM is an evaluation and benchmarking platform for measuring and optimizing the performance,...
Tokens:1,245,368
Snippets:5,881
Trust Score:9.7
Update:1 month ago
Show doc for...
Tokens:
Raw
### MockServer usage example

Source: https://vllm-project.github.io/guidellm/stable/reference/guidellm/mock_server/server?q=

Shows how to instantiate and run the MockServer with a ServerConfig. This is the minimal setup for starting the mock server for testing.

```python
config = ServerConfig(model="test-model", port=8080)
server = MockServer(config)
server.run()
```

--------------------------------

### Install GuideLLM with multimodal extras

Source: https://vllm-project.github.io/guidellm/stable/guides/multimodal

Install GuideLLM with the required extras for multimodal features. Use 'pip install guidellm[vision,audio]' for all modalities, or install specific extras for images/video or audio.

```bash
# For all multimodal features
pip install guidellm[vision,audio]

# For specific modalities
pip install guidellm[vision]  # Images and Video
pip install guidellm[audio]   # Audio

```

--------------------------------

### Verify GuideLLM installation

Source: https://vllm-project.github.io/guidellm/stable/examples/practice_on_vllm_simulator?q=

Displays the GuideLLM command-line help to confirm the installation.

```bash
guidellm --help

```

--------------------------------

### Full example usage with multiple data options

Source: https://vllm-project.github.io/guidellm/stable/guides/datasets

Complete example combining backend, profile, constraint, data, column mapper, tokenizer, and data loader in a single guidellm run command.

```bash
guidellm run \
  --backend kind=openai_http,target=http://localhost:8000 \
  --profile kind=throughput \
  --constraint kind=max_requests,count=1000 \
  --data '{"kind":"huggingface","source":"my/dataset","load_kwargs":{"split":"test"}}' \
  --data-column-mapper '{"kind":"generative_column_mapper","column_mappings":{"text_column":"prompt"}}' \
  --tokenizer '{"kind":"huggingface_auto","model":"path/to/processor","load_kwargs":{"use_fast":false}}' \
  --data-loader kind=pytorch,shuffle=true

```

--------------------------------

### Example: Using InterProcessMessagingQueue

Source: https://vllm-project.github.io/guidellm/stable/reference/guidellm/utils/messaging?q=

Shows how to instantiate InterProcessMessagingQueue with pickle serialization and a max pending size of 100, then start, put, get with a timeout, and stop the messaging. This is the documented usage example for the InterProcessMessaging abstract base.

```python
from guidellm.utils.messaging import InterProcessMessagingQueue

messaging = InterProcessMessagingQueue(
    serialization="pickle",
    max_pending_size=100
)

await messaging.start()
await messaging.put(request_data)
response = await messaging.get(timeout=5.0)
await messaging.stop()
```

--------------------------------

### Start a mock server for testing

Source: https://vllm-project.github.io/guidellm/stable/reference/guidellm/cli

Starts a mock server on the specified host and port using the guidellm mock-server command. Useful for development and testing without a real model backend.

```bash
# Start a mock server for testing
guidellm mock-server --host 0.0.0.0 --port 8080
```

--------------------------------

### Full GuideLLM run example

Source: https://vllm-project.github.io/guidellm/stable/guides/datasets?q=

Complete example of a GuideLLM run combining backend, profile, constraint, data, column mapper, tokenizer, and data loader options. Uses a HuggingFace dataset with a generative column mapper and a custom tokenizer.

```bash
guidellm run \
  --backend kind=openai_http,target=http://localhost:8000 \
  --profile kind=throughput \
  --constraint kind=max_requests,count=1000 \
  --data '{"kind":"huggingface","source":"my/dataset","load_kwargs":{"split":"test"}}' \
  --data-column-mapper '{"kind":"generative_column_mapper","column_mappings":{"text_column":"prompt"}}' \
  --tokenizer '{"kind":"huggingface_auto","model":"path/to/processor","load_kwargs":{"use_fast":false}}' \
  --data-loader kind=pytorch,shuffle=true
```

--------------------------------

### Start vLLM server with a conversational model

Source: https://vllm-project.github.io/guidellm/stable/guides/multiturn

Starts a vLLM server serving the meta-llama/Llama-3.1-8B-Instruct model, which is used as the inference server for the benchmarking examples.

```bash
vllm serve meta-llama/Llama-3.1-8B-Instruct
```

--------------------------------

### Start vLLM server for Qwen3-VL

Source: https://vllm-project.github.io/guidellm/stable/guides/multimodal/video?q=

Starts a vLLM server for the Qwen3-VL model, which is used as the OpenAI-compatible inference server for the video benchmarking examples.

```bash
vllm serve Qwen/Qwen3-VL-2B-Instruct
```

--------------------------------

### Install from local clone

Source: https://vllm-project.github.io/guidellm/stable/getting-started/install

Install GuideLLM from a local repository clone. Use the editable mode for development purposes to reflect source code changes immediately.

```bash
pip install .
```

```bash
pip install -e .
```

--------------------------------

### SchedulingStrategy.get_processes_start_time

Source: https://vllm-project.github.io/guidellm/stable/reference/guidellm/scheduler/strategies?q=

Gets the synchronized start time, waiting if not yet set. Blocks until the main process sets the start time via init_processes_start.

```APIDOC
## get_processes_start_time

### Description
Gets the synchronized start time, waiting if not yet set. Blocks until the main process sets the start time via init_processes_start.

### Method
get_processes_start_time

### Returns
- **float** - Unix timestamp when request processing began

### Exceptions
- **RuntimeError** - If called before init_processes_timings
```

--------------------------------

### Install GuideLLM from source

Source: https://vllm-project.github.io/guidellm/stable/examples/practice_on_vllm_simulator?q=

Clones the GuideLLM repository and installs the package via pip.

```bash
git clone https://github.com/vllm-project/guidellm.git
cd guidellm
pip install guidellm

```

--------------------------------

### Start vLLM server for Whisper and Ultravox

Source: https://vllm-project.github.io/guidellm/stable/guides/multimodal/audio?q=

Commands to start vLLM serving a Whisper model for transcription/translation and an Ultravox model for audio chat. These are sample commands for benchmarking examples.

```bash
# Whisper ASR/Translation
vllm serve openai/whisper-small

# Ultravox Audio Chat
vllm serve fixie-ai/ultravox-v0_5-llama-3_2-1b

```

--------------------------------

### Install GuideLLM with audio support

Source: https://vllm-project.github.io/guidellm/stable/guides/multimodal/audio?q=

Installs GuideLLM with audio and recommended dependencies using pip.

```bash
pip install guidellm[audio,recommended]

```

--------------------------------

### Install GuideLLM with vision support

Source: https://vllm-project.github.io/guidellm/stable/guides/multimodal/video?q=

Installs GuideLLM with vision support and recommended dependencies. Run this on the same instance or another machine that can reach the inference server.

```bash
pip install guidellm[vision,recommended]
```

--------------------------------

### WorkerProcess instantiation and run example

Source: https://vllm-project.github.io/guidellm/stable/reference/guidellm/scheduler/worker?q=

Shows how to create a WorkerProcess instance with required parameters and start it with the run() method. The example includes a worker_index, messaging interface, backend, timing strategy, concurrency limits, and synchronization events for coordination and shutdown.

```python
worker = WorkerProcess(
    worker_index=0,
    messaging=messaging_interface,
    backend=backend_instance,
    strategy=timing_strategy,
    async_limit=10,
    fut_scheduling_time_limit=5.0,
    startup_barrier=barrier,
    requests_generated_event=generated_event,
    constraint_reached_event=constraint_event,
    shutdown_event=shutdown,
    error_event=error,
)
worker.run()
```

--------------------------------

### Initialize and use OpenAIHTTPBackend

Source: https://vllm-project.github.io/guidellm/stable/reference/guidellm/backends/openai/http

This example demonstrates how to configure the backend with server details, initialize it, process requests, and shut it down.

```python
backend_args = OpenAIHTTPBackendArgs(
    target="http://localhost:8000",
    model="gpt-3.5-turbo",
    api_key="your-api-key",
)
backend = OpenAIHTTPBackend(backend_args)

await backend.process_startup()
async for response, request_info in backend.resolve(request, info):
    process_response(response)
await backend.process_shutdown()
```

--------------------------------

### EndlessTextCreator usage example

Source: https://vllm-project.github.io/guidellm/stable/reference/guidellm/utils/text?q=

Shows how to instantiate EndlessTextCreator with a source file and generate text sequences by calling create_text with start and length parameters.

```python
creator = EndlessTextCreator("path/to/source.txt")
generated = creator.create_text(start=0, length=100)
more_text = creator.create_text(start=50, length=200)
```

--------------------------------

### GenerativeRequestStats usage example

Source: https://vllm-project.github.io/guidellm/stable/reference/guidellm/schemas/request_stats

Shows how to instantiate `GenerativeRequestStats` with request metadata and usage metrics, then access the `output_tokens_per_second` computed property to get throughput.

```python
stats = GenerativeRequestStats(
    request_id="req_123",
    info=request_info,
    input_metrics=input_usage,
    output_metrics=output_usage
)
throughput = stats.output_tokens_per_second
```

--------------------------------

### Backend registration and usage example

Source: https://vllm-project.github.io/guidellm/stable/reference/guidellm/backends/backend

Shows how to register a custom backend with @Backend.register and create an instance using Backend.create. The example defines MyBackend with an async process_startup method and initializes it with MyBackendArgs.

```python
@Backend.register("my_backend")
class MyBackend(Backend):
    def __init__(self, args: MyBackendArgs):
        super().__init__(args)
        self.api_key = args.api_key

    async def process_startup(self):
        self.client = MyAPIClient(self.api_key)

args = MyBackendArgs(api_key="secret")
backend = Backend.create(args)
```

--------------------------------

### available_models() async method

Source: https://vllm-project.github.io/guidellm/stable/reference/guidellm/backends/openai/websocket

Lists model IDs from GET /v1/models on the HTTP target. Raises RuntimeError if the client is not started or the response is invalid.

```python
async def available_models(self) -> list[str]:
    """
    List model IDs from ``GET /v1/models`` on the HTTP target.

    :return: Model identifiers from the OpenAI-style payload.
    :raises RuntimeError: If the client is not started or the response is invalid.
    """
    if self._async_client is None:
        raise RuntimeError("Backend not started up for process.")
    target = f"{self._args.target}/v1/models"
    response = await self._async_client.get(
        target, headers=build_headers(self._args.api_key)
    )
    response.raise_for_status()
    return [item["id"] for item in response.json()["data"]]
```

--------------------------------

### Install from main branch

Source: https://vllm-project.github.io/guidellm/stable/getting-started/install

Install the latest development version of GuideLLM directly from the main branch of the repository.

```bash
pip install git+https://github.com/vllm-project/guidellm.git
```

--------------------------------

### Audio translation benchmark output

Source: https://vllm-project.github.io/guidellm/stable/guides/multimodal/audio?q=

Example output from the audio translation benchmark command, showing backend validation, setup, metrics table, and generated benchmark files.

```text
✔ OpenAIHTTPBackend backend validated with model openai/whisper-small

......
......
✔ Setup complete, starting benchmarks...

......
......

ℹ Audio Metrics Statistics (Completed Requests)
|=============|=======|========|========|========|=========|=========|==========|==========|======|=======|======|======|=========|==========|==========|==========|
| Benchmark   | Input Tokens                  |||| Input Samples                        |||| Input Seconds           |||| Input Bytes                           |||
| Strategy    | Per Request   || Per Second     || Per Request      || Per Second         || Per Request || Per Second || Per Request       || Per Second         ||
|             | Mdn   | p95    | Mdn    | Mean   | Mdn     | p95     | Mdn      | Mean     | Mdn  | p95   | Mdn  | Mean | Mdn     | p95      | Mdn      | Mean     |
|-------------|-------|--------|--------|--------|---------|---------|----------|----------|------|-------|------|-------|---------|----------|----------|----------|
| synchronous | 642.0 | 1688.0 | 7483.6 | 7563.5 | 16000.0 | 16000.0 | 133404.0 | 146385.0 | 6.4  | 16.8  | 74.5 | 75.2 | 52172.0 | 135692.0 | 603620.5 | 611706.4 |
|=============|=======|========|========|========|=========|=========|==========|==========|======|=======|======|======|=========|==========|==========|==========|

......
......

✔ Benchmarking complete, generated 1 benchmark(s)
…   json    : /Users/markkurtz/code/github/vllm-project/guidellm/benchmarks.json
…   csv     : /Users/markkurtz/code/github/vllm-project/guidellm/benchmarks.csv
…   html    : /Users/markkurtz/code/github/vllm-project/guidellm/benchmarks.html
```

--------------------------------

### Install from specific branch

Source: https://vllm-project.github.io/guidellm/stable/getting-started/install

Install GuideLLM from a specific branch by appending the branch name to the repository URL.

```bash
pip install git+https://github.com/vllm-project/guidellm.git@feature-branch
```

--------------------------------

### ConsoleUpdateStep usage example

Source: https://vllm-project.github.io/guidellm/stable/reference/guidellm/utils/console

Shows how to use ConsoleUpdateStep with a Console instance. Create a Console, then use the with statement to start a progress step, update its status, and finish with a success message.

```python
console = Console()
with console.print_update_step("Processing data") as step:
    step.update("Loading files", "info")
    # ... do work ...
    step.finish("Completed successfully", status_level="success")
```

--------------------------------

### OpenAIWebSocketBackend usage example

Source: https://vllm-project.github.io/guidellm/stable/reference/guidellm/backends/openai/websocket

Shows how to instantiate the OpenAIWebSocketBackend with OpenAIWebSocketBackendArgs, then start it, iterate over resolved responses, and shut it down. The backend connects to a vLLM-style /v1/realtime WebSocket and streams PCM16 audio chunks.

```python
args = OpenAIWebSocketBackendArgs(
    target="http://localhost:8000",
    model="my-model",
)
backend = OpenAIWebSocketBackend(args)

await backend.process_startup()
async for response, request_info in backend.resolve(request, info):
    ...
await backend.process_shutdown()
```

--------------------------------

### process_startup() async

Source: https://vllm-project.github.io/guidellm/stable/reference/guidellm/scheduler/schemas?q=

Performs backend initialization and startup procedures. Raises an exception on failure.

```APIDOC
## `process_startup()` `async`

### Description
Perform backend initialization and startup procedures.

### Method
async

### Parameters
None

### Request Example
N/A

### Response
#### Success Response
None

#### Response Example
N/A

### Error Handling
- **Exception** - Implementation-specific exceptions for startup failures

### Source Code
```python
async def process_startup(self) -> None:
    """
    Perform backend initialization and startup procedures.

    :raises Exception: Implementation-specific exceptions for startup failures
    """

```
```

--------------------------------

### WorkerProcessGroup usage example

Source: https://vllm-project.github.io/guidellm/stable/reference/guidellm/scheduler/worker_group

Shows the basic usage of WorkerProcessGroup: instantiate with requests, backend, strategy, and constraints, then create processes, start with a timestamp, iterate over request_updates to handle responses, and finally shut down.

```python
from guidellm.scheduler.worker_group import WorkerProcessGroup

group = WorkerProcessGroup(
    requests=request_iterable,
    backend=backend_instance,
    strategy=scheduling_strategy,
    constraints={"max_time": time_constraint}
)

await group.create_processes()
await group.start(time.time())

async for response, request, info, state in group.request_updates():
    if response is not None:
        # Process completed request
        handle_response(response)

await group.shutdown()
```

--------------------------------

### Run GuideLLM with local Hugging Face dataset directory

Source: https://vllm-project.github.io/guidellm/stable/guides/datasets

Example command to run GuideLLM with a local dataset directory in Hugging Face format.

```bash
guidellm run \
  --backend kind=openai_http,target=http://localhost:8000 \
  --profile kind=throughput \
  --constraint kind=max_requests,count=1000 \
  --data kind=huggingface,source=path/to/dataset

```

--------------------------------

### Dataset with tools column

Source: https://vllm-project.github.io/guidellm/stable/guides/tool_calling?q=

Command-line example for running a GuideLLM benchmark using a Hugging Face dataset that contains tool definitions. Requires enabling the tool_calling_message_extractor preprocessor explicitly.

```bash
guidellm run \
  --backend kind=openai_http,target=http://localhost:8000 \
  --data kind=huggingface,source=madroid/glaive-function-calling-openai \
  --data-column-mapper kind=generative_column_mapper,column_mappings.text_column=messages,column_mappings.tools_column=tools \
  --data-preprocessor kind=tool_calling_message_extractor \
  --data-preprocessor kind=encode_media \
  --constraint kind=max_requests,count=50 \
  --profile kind=constant,rate=1

```
Tokens:
Raw