### Setup genai-bench Development Environment

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/developer-guide/index.md

Installs dependencies and builds the genai-bench wheel using make commands. It also shows how to activate the virtual environment managed by uv.

```shell
# check out commands genai-bench supports
make help
#activate virtual env managed by uv
source .venv/bin/activate
# install dependencies
make install
```

--------------------------------

### Manage genai-bench Documentation

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/developer-guide/index.md

Commands to install documentation dependencies, serve the documentation locally for preview, and build the static documentation files.

```bash
# Install docs dependencies
make docs

# Serve docs locally
make docs-serve

# Build docs
make docs-build
```

--------------------------------

### SGLang or vLLM OpenAI-Compatible API Example

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-auth-storage.md

Shows how to benchmark using SGLang or vLLM, which provide OpenAI-compatible APIs. This example includes optional API key authentication and requires the server endpoint.

```bash
genai-bench benchmark \
  --api-backend sglang \
  --api-base http://localhost:8000 \
  --api-key optional-key \
  --api-model-name meta-llama/Llama-2-7b-hf \
  --model-tokenizer meta-llama/Llama-2-7b-hf \
  --task text-to-text \
  --max-requests-per-run 100 \
  --max-time-per-run 10
```

--------------------------------

### Build and Deploy genai-bench Wheel

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/developer-guide/index.md

Builds the genai-bench wheel package and demonstrates how to transfer it to a remote machine using rsync for installation.

```shell
# build a .whl under genai-bench/dist
make build
# send the wheel to your remote machine if applies
rsync --delete -avz ~/genai-bench/dist/<.wheel> <remote-user>@<remote-ip>:<dest-addr>
```

--------------------------------

### Azure OpenAI API Key Authentication Example

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-auth-storage.md

Demonstrates how to authenticate with Azure OpenAI using an API key for benchmarking. Requires API key and Azure-specific endpoint and deployment details.

```bash
genai-bench benchmark \
  --api-backend azure-openai \
  --api-base https://myresource.openai.azure.com \
  --azure-endpoint https://myresource.openai.azure.com \
  --azure-deployment my-gpt-4-deployment \
  --model-api-key YOUR_AZURE_API_KEY \
  --api-model-name gpt-4 \
  --model-tokenizer gpt2 \
  --task text-to-text \
  --max-requests-per-run 100 \
  --max-time-per-run 10
```

--------------------------------

### Install genai-bench Wheel on Remote Machine

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/developer-guide/index.md

Installs the genai-bench wheel package on a remote machine using pip after it has been transferred.

```bash
pip install <dest-addr>/<.wheel>
```

--------------------------------

### Install GenAI Bench

Source: https://context7.com/sgl-project/genai-bench/llms.txt

Installs the GenAI Bench tool from PyPI. Basic installation includes the core functionality, while the 'multi-cloud' extra installs support for AWS, Azure, and GCP. Verification commands are also provided.

```bash
pip install genai-bench

pip install genai-bench[multi-cloud]

genai-bench --version
genai-bench --help
```

--------------------------------

### GCP Vertex AI Service Account Authentication Example

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-auth-storage.md

Illustrates benchmarking with GCP Vertex AI using a service account key file. Requires GCP project ID, location, and the path to the service account JSON file.

```bash
genai-bench benchmark \
  --api-backend gcp-vertex \
  --api-base https://us-central1-aiplatform.googleapis.com \
  --gcp-project-id my-project-123 \
  --gcp-location us-central1 \
  --gcp-credentials-path /path/to/service-account.json \
  --api-model-name gemini-1.5-pro \
  --model-tokenizer google/gemini \
  --task text-to-text \
  --max-requests-per-run 100 \
  --max-time-per-run 10
```

--------------------------------

### Clone genai-bench Repository

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/developer-guide/index.md

Clones the genai-bench repository from GitHub and navigates into the project directory. This is the first step for setting up the development environment.

```bash
git clone https://github.com/sgl-project/genai-bench.git
cd genai-bench
```

--------------------------------

### JSON Dataset Configuration Example

Source: https://context7.com/sgl-project/genai-bench/llms.txt

An example of a JSON configuration file for specifying a HuggingFace dataset, including the source type, path, optional HuggingFace arguments, and the prompt column.

```json
// examples/dataset_configs/huggingface_simple.json
{
  "source": {
    "type": "huggingface",
    "path": "squad",
    "huggingface_kwargs": {
      "split": "train"
    }
  },
  "prompt_column": "question"
}
```

--------------------------------

### Start Inference Server with Docker

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/run-benchmark-using-docker.md

Starts an inference server (e.g., SGLang) within a Docker container, connected to the 'benchmark-network'. It maps GPUs, shared memory, and model volumes.

```shell
docker run -itd \
    --gpus \"device=0,1,2,3\" \
    --shm-size 10g  -v /raid/models:/models \
    --ulimit nofile=65535:65535   --network benchmark-network \
    --name sglang-v0.4.7.post1-llama4-scout-tp4 \
    lmsysorg/sglang:v0.4.7.post1-cu124 \
    python3 -m sglang.launch_server \
    --model-path=/models/meta-llama/Llama-4-Scout-17B-16E-Instruct \
    --tp 4 \
    --port=8080 \
    --host 0.0.0.0 \
    --context-length=131072
```

--------------------------------

### Cross-Cloud Benchmarking (OCI GenAI to OCI Object Storage)

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-auth-storage.md

Benchmark OCI GenAI models (Grok) and store results in OCI Object Storage. This example uses OCI configuration files and specifies the storage bucket and namespace.

```bash
genai-bench benchmark \
  --api-backend oci-genai \
  --api-base https://inference.generativeai.us-ashburn-1.oci.oraclecloud.com \
  --auth security_token \
  --config-file ~/.oci/config \
  --profile DEFAULT \
  --api-model-name xai.grok-3-mini-fast \
  --model-tokenizer gpt2 \
  --task text-to-text \
  --additional-request-params '{"compartmentId": "ocid1.compartment.oc1..example", "servingType": "ON_DEMAND"}' \
  --max-requests-per-run 100 \
  --max-time-per-run 10 \
  --upload-results \
  --storage-provider oci \
  --storage-bucket oci-genai-benchmarks \
  --namespace my-namespace
```

--------------------------------

### Multi-Modal Tasks (Image-text-to-text with GCP Vertex AI)

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-auth-storage.md

Perform image-text-to-text benchmarking using GCP Vertex AI with Gemini models. This example specifies GCP project details, model, and dataset path.

```bash
genai-bench benchmark \
  --api-backend gcp-vertex \
  --api-base https://us-central1-aiplatform.googleapis.com \
  --gcp-project-id my-project \
  --gcp-location us-central1 \
  --gcp-credentials-path /path/to/service-account.json \
  --api-model-name gemini-1.5-pro-vision \
  --model-tokenizer google/gemini \
  --task image-text-to-text \
  --dataset-path /path/to/image/dataset \
  --max-requests-per-run 50 \
  --max-time-per-run 10
```

--------------------------------

### Cross-Cloud Benchmarking and Storage

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-quick-reference.md

These examples show advanced multi-cloud scenarios, combining benchmarking of one cloud provider's API with uploading results to another's storage. The first example benchmarks OpenAI and stores results in AWS S3, while the second benchmarks AWS Bedrock and stores results in Azure Blob Storage.

```bash
export MODEL_API_KEY=sk-...
export AWS_PROFILE=default

genai-bench benchmark \
  --api-backend openai \
  --api-base https://api.openai.com/v1 \
  --api-model-name gpt-4 \
  --model-tokenizer gpt2 \
  --task text-to-text \
  --max-requests-per-run 100 \
  --max-time-per-run 10 \
  --upload-results \
  --storage-provider aws \
  --storage-bucket openai-benchmarks \
  --storage-aws-region us-east-1
```

```bash
export AWS_PROFILE=bedrock-user
export AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=..."

genai-bench benchmark \
  --api-backend aws-bedrock \
  --api-base https://bedrock-runtime.us-east-1.amazonaws.com \
  --aws-region us-east-1 \
  --api-model-name anthropic.claude-3-sonnet-20240229-v1:0 \
  --model-tokenizer Anthropic/claude-3-sonnet \
  --task text-to-text \
  --max-requests-per-run 100 \
  --max-time-per-run 10 \
  --upload-results \
  --storage-provider azure \
  --storage-bucket bedrock-benchmarks
```

--------------------------------

### Azure OpenAI Azure AD Authentication Example

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-auth-storage.md

Shows how to authenticate with Azure OpenAI using Azure Active Directory tokens. This method requires an Azure AD token and Azure-specific endpoint and deployment details.

```bash
genai-bench benchmark \
  --api-backend azure-openai \
  --api-base https://myresource.openai.azure.com \
  --azure-endpoint https://myresource.openai.azure.com \
  --azure-deployment my-gpt-4-deployment \
  --azure-ad-token YOUR_AAD_TOKEN \
  --api-model-name gpt-4 \
  --model-tokenizer gpt2 \
  --task text-to-text \
  --max-requests-per-run 100 \
  --max-time-per-run 10
```

--------------------------------

### AWS Bedrock Environment Variables Setup

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-auth-storage.md

Sets up AWS credentials and region using environment variables for benchmarking AWS Bedrock models. This follows the AWS SDK's default behavior for credential resolution.

```bash
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_DEFAULT_REGION=us-east-1

genai-bench benchmark --api-backend aws-bedrock ...
```

--------------------------------

### Distributed Benchmarking Setup

Source: https://context7.com/sgl-project/genai-bench/llms.txt

Instructions for setting up distributed benchmarking to scale load across multiple worker processes when single-process performance is insufficient. This is typically triggered by CPU usage warnings.

```bash
# When you see: "CPU usage above 90%! This may constrain your throughput..."
```

--------------------------------

### Run genai-bench with Multiple Text Scenarios

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/scenario-definition.md

Example CLI command to run genai-bench with multiple text traffic scenarios. This allows benchmarking with different token distributions (normal and deterministic).

```bash
genai-bench \
  --task text-to-text \
  --dataset-path prompts.txt \
  --traffic-scenario "N(480,240)/(300,150)" \
  --traffic-scenario "D(100,1000)" \
  --api-backend openai \
  --api-base http://localhost:8000 \
  --api-model-name gpt-test \
  --model-tokenizer gpt2
```

--------------------------------

### GCP Cloud Storage Upload

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-auth-storage.md

Upload benchmark results to GCP Cloud Storage. This example demonstrates authentication using a service account key file and specifies the project ID and bucket name.

```bash
genai-bench benchmark \
  ... \
  --upload-results \
  --storage-provider gcp \
  --storage-bucket my-benchmark-results \
  --storage-prefix experiments/2024 \
  --storage-gcp-project-id my-project-123 \
  --storage-gcp-credentials-path /path/to/service-account.json
```

--------------------------------

### Cross-Cloud Benchmarking (AWS Bedrock to Azure Blob)

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-auth-storage.md

Benchmark AWS Bedrock models and store results in Azure Blob Storage. This example configures AWS Bedrock access and Azure Blob connection string.

```bash
genai-bench benchmark \
  --api-backend aws-bedrock \
  --api-base https://bedrock-runtime.us-east-1.amazonaws.com \
  --aws-profile bedrock-user \
  --aws-region us-east-1 \
  --api-model-name anthropic.claude-3-sonnet-20240229-v1:0 \
  --model-tokenizer Anthropic/claude-3-sonnet \
  --task text-to-text \
  --max-requests-per-run 100 \
  --max-time-per-run 10 \
  --upload-results \
  --storage-provider azure \
  --storage-bucket benchmarks \
  --storage-azure-connection-string "DefaultEndpointsProtocol=..."
```

--------------------------------

### GenAI-Bench Dataset Configuration (JSON)

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/run-benchmark-using-docker.md

Example JSON configuration file for specifying a dataset to be used with GenAI-Bench. It defines the data source, prompt column, and image column.

```json
{
  "source": {
    "type": "huggingface",
    "path": "lmms-lab/llava-bench-in-the-wild",
    "huggingface_kwargs": {
      "split": "train"
    }
  },
  "prompt_column": "question",
  "image_column": "image"
}
```

--------------------------------

### Configure OCI Cohere Authentication for Genai-bench (User Principal)

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-auth-storage.md

This example shows how to configure genai-bench to use OCI Cohere models with user principal authentication. It requires specifying the OCI configuration file, profile, and other model parameters.

```bash
genai-bench benchmark \
  --api-backend oci-cohere \
  --api-base https://inference.generativeai.us-chicago-1.oci.oraclecloud.com \
  --auth user_principal \
  --config-file ~/.oci/config \
  --profile DEFAULT \
  --api-model-name cohere.command-r-plus \
  --model-tokenizer Cohere/command-r-plus \
  --task text-to-text \
  --max-requests-per-run 100 \
  --max-time-per-run 10
```

--------------------------------

### Benchmark Specific Tokens: SGLang vs. GenAI Bench

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/getting-started/migrating-from-sglang-vllm.md

This example demonstrates how to benchmark with a fixed number of input and output tokens, mimicking SGLang's `--random-range-ratio 1.0`. GenAI Bench uses a deterministic traffic scenario for this purpose.

```bash
python -m sglang.bench_serving \
    --backend sglang-oai-chat \
    --num-prompts 500 \
    --random-input-len 512 \
    --random-output-len 512 \
    --random-range-ratio 1.0 \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --base-url http://127.0.0.1:8080
```

```bash
genai-bench benchmark \
    --api-backend sglang \
    --api-base http://127.0.0.1:8080 \
    --api-model-name sglang-model \
    --model-tokenizer meta-llama/Llama-3.1-8B-Instruct \
    --task text-to-text \
    --max-requests-per-run 500 \
    --max-time-per-run 30 \
    --traffic-scenario "D(512,512)" \
    --num-concurrency 1 \
    --server-engine "SGLang"
```

--------------------------------

### Run Benchmark with Cross-Cloud Storage (Bash)

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/index.md

This command demonstrates how to run a benchmark against the OpenAI API and store the results in an AWS S3 bucket. It requires specifying the API backend, API key, and storage configuration.

```bash
genai-bench benchmark \
  --api-backend openai \
  --api-key $OPENAI_KEY \
  --upload-results \
  --storage-provider aws \
  --storage-bucket my-results
```

--------------------------------

### Simple CLI Dataset Selection in Shell

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/run-benchmark.md

Provides examples of selecting datasets for benchmarking using the genai-bench CLI. It covers local CSV and text files, as well as HuggingFace datasets with specified prompt columns.

```shell
# Local CSV file
--dataset-path /path/to/data.csv \
--dataset-prompt-column "prompt"

# HuggingFace dataset with simple options
--dataset-path squad \
--dataset-prompt-column "question"

# Local text file (default)
--dataset-path /path/to/prompts.txt
```

--------------------------------

### Run Tests for genai-bench

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/developer-guide/index.md

Executes tests for the genai-bench project using pytest. Includes options for running all tests, specific files, with coverage, or specific test cases.

```bash
# Run all tests with coverage
make test

# Or run pytest directly
pytest

# Run specific test file
pytest tests/auth/test_unified_factory.py

# Run with coverage
pytest --cov=genai_bench

# Run specific test
pytest -k "test_openai_auth"
```

--------------------------------

### Run GenAI-Bench Benchmark with Docker

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/run-benchmark-using-docker.md

Runs the GenAI-Bench container to perform benchmarks against a running inference server. It requires a dataset configuration file and specifies various benchmark parameters.

```shell
docker run \
    -tid \
    --shm-size 5g \
    --ulimit nofile=65535:65535 \
    --env HF_TOKEN="<your_HF_TOKEN>" \
    --network <your-network> \
    -v <path-to-your-local-model>:/models \
    -v $(pwd)/llava-config.json:/genai-bench/llava-config.json \
    --name llama-4-scout-benchmark \
    genai-bench:dev \
    benchmark \
    --api-backend sglang \
    --api-base http://localhost:8080 \
    --api-key your_api_key \
    --api-model-name /models/meta-llama/Llama-4-Scout-17B-16E-Instruct \
    --model-tokenizer /models/meta-llama/Llama-4-Scout-17B-16E-Instruct \
    --task image-text-to-text \
    --max-time-per-run 10 \
    --max-requests-per-run 100 \
    --server-engine "SGLang" \
    --server-gpu-type "H100" \
    --server-version "v0.4.7.post1" \
    --server-gpu-count 4 \
    --traffic-scenario "I(512,512)" \
    --traffic-scenario "I(2048,2048)" \
    --num-concurrency 1 \
    --num-concurrency 2 \
    --num-concurrency 4 \
    --dataset-config /genai-bench/llava-config.json
```

--------------------------------

### Run genai-bench in Dataset-Only Mode

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/scenario-definition.md

Example CLI command to run genai-bench using only dataset mode, without any specific traffic shaping scenarios. This is useful for testing with raw data.

```bash
genai-bench \
  --task text-to-text \
  --dataset-path data.csv \
  --api-backend openai \
  --api-base http://localhost:8000 \
  --api-model-name gpt-test \
  --model-tokenizer gpt2
```

--------------------------------

### Configure OpenAI API Key Authentication for Genai-bench

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-auth-storage.md

This snippet demonstrates how to authenticate with OpenAI's API for model benchmarking using either a direct API key or an environment variable. It requires the API key and specifies the backend and model details.

```bash
genai-bench benchmark \
  --api-backend openai \
  --api-base https://api.openai.com/v1 \
  --api-key sk-... \
  --api-model-name gpt-4 \
  --model-tokenizer gpt2 \
  --task text-to-text \
  --max-requests-per-run 100 \
  --max-time-per-run 10
```

```bash
export MODEL_API_KEY=sk-...
genai-bench benchmark --api-backend openai ...
```

--------------------------------

### Migrate CLI Storage Arguments (Bash)

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-auth-storage.md

Demonstrates the migration of storage-related command-line arguments from the legacy OCI-only CLI to the new GenAI Bench CLI. The primary changes involve renaming `--bucket` to `--storage-bucket` and `--prefix` to `--storage-prefix`, with an optional addition of `--storage-provider oci`.

```bash
## Old command:
```bash
genai-bench benchmark \
  --api-backend oci-cohere \
  --bucket my-bucket \
  --prefix my-prefix \
  ...
```

## New command:
```bash
genai-bench benchmark \
  --api-backend oci-cohere \
  --storage-bucket my-bucket \
  --storage-prefix my-prefix \
  --storage-provider oci \
  ...
```
```

--------------------------------

### Explicitly Use Dataset Mode with Image Task

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/scenario-definition.md

Example CLI command to explicitly run genai-bench in dataset mode for an image-text-to-text task. This demonstrates how to be explicit about using dataset mode.

```bash
genai-bench \
  --task image-text-to-text \
  --dataset-path images.json \
  --traffic-scenario dataset \
  --api-backend openai \
  --api-base http://localhost:8000 \
  --api-model-name gpt-test \
  --model-tokenizer gpt2
```

--------------------------------

### Monitor Benchmark Results with Volume Mount

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/run-benchmark-using-docker.md

Runs the GenAI-Bench container with a volume mount to save benchmark results to a host directory. This allows for monitoring interim results and analysis.

```shell
HOST_OUTPUT_DIR=$HOME/benchmark_results
CONTAINER_OUTPUT_DIR=/genai-bench/benchmark_results
docker run \
    -tid \
    --shm-size 5g \
    --ulimit nofile=65535:65535 \
    --env HF_TOKEN="<your_HF_TOKEN>" \
    --network <your-network> \
    -v <path-to-your-local-model>:/models \
    -v $HOST_OUTPUT_DIR:$CONTAINER_OUTPUT_DIR \
    -v $(pwd)/llava-config.json:/genai-bench/llava-config.json \
    --name llama-3.2-11b-benchmark \
    genai-bench:dev \
    benchmark \
    --api-backend sglang \
    --api-base http://localhost:8080 \
    --api-key your_api_key \
    --api-model-name /models/meta-llama/Llama-4-Scout-17B-16E-Instruct \
    --model-tokenizer /models/meta-llama/Llama-4-Scout-17B-16E-Instruct \
    --task image-text-to-text \
    --max-time-per-run 10 \
    --max-requests-per-run 100 \
    --server-engine "SGLang" \
    --server-gpu-type "H100" \
    --server-version "v0.4.7.post1" \
    --server-gpu-count 4 \
    --traffic-scenario "I(512,512)" \
    --traffic-scenario "I(2048,2048)" \
    --num-concurrency 1 \
    --num-concurrency 2 \
    --num-concurrency 4 \
    --dataset-config /genai-bench/llava-config.json \
    --experiment-base-dir $CONTAINER_OUTPUT_DIR
```

--------------------------------

### Configure OCI Cohere Authentication for Genai-bench (Instance Principal)

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-auth-storage.md

This snippet illustrates how to authenticate with OCI Cohere models using instance principal authentication, suitable for compute instances. It requires specifying the region and other model parameters.

```bash
genai-bench benchmark \
  --api-backend oci-cohere \
  --api-base https://inference.generativeai.us-chicago-1.oci.oraclecloud.com \
  --auth instance_principal \
  --region us-chicago-1 \
  --api-model-name cohere.command-r-plus \
  --model-tokenizer Cohere/command-r-plus \
  --task text-to-text \
  --max-requests-per-run 100 \
  --max-time-per-run 10
```

--------------------------------

### OCI GenAI User Principal Benchmark

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-auth-storage.md

Benchmarks OCI GenAI models using the 'user_principal' authentication method. Requires OCI configuration file and specifies compartment ID and serving type for ON_DEMAND inference.

```bash
genai-bench benchmark \
  --api-backend oci-genai \
  --api-base https://inference.generativeai.us-ashburn-1.oci.oraclecloud.com \
  --auth user_principal \
  --config-file ~/.oci/config \
  --profile DEFAULT \
  --api-model-name xai.grok-3-mini-fast \
  --model-tokenizer gpt2 \
  --task text-to-text \
  --additional-request-params '{"compartmentId": "ocid1.compartment.oc1..example", "servingType": "ON_DEMAND"}' \
  --max-requests-per-run 100 \
  --max-time-per-run 10
```

--------------------------------

### Listing Available Fields for Plotting

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/generate-plot.md

This command lists all available fields from the experiment data that can be used in plot configurations. It helps users understand what metrics are available for visualization. The output can guide the creation of custom plot configurations.

```bash
# List available fields with actual data from your experiment
genai-bench plot --experiments-folder /path/to/experiments \
                 --group-key traffic_scenario \
                 --list-fields
```

--------------------------------

### OCI Object Storage Upload Configuration

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-auth-storage.md

Configures genai-bench to upload benchmark results to OCI Object Storage. Requires specifying the storage provider, bucket, prefix, and authentication details like namespace and auth type.

```bash
genai-bench benchmark \
  ... \
  --upload-results \
  --storage-provider oci \
  --storage-bucket my-benchmark-results \
  --storage-prefix experiments/2024 \
  --storage-auth-type user_principal \
  --namespace my-namespace
```

--------------------------------

### Run Vision Benchmark (Image-Text-to-Text)

Source: https://context7.com/sgl-project/genai-bench/llms.txt

Benchmarks multimodal models capable of processing both images and text. This example uses the SGLang backend and specifies model paths, task type, performance constraints, server details, traffic scenarios, and a dataset configuration file for evaluation.

```bash
genai-bench benchmark \
  --api-backend sglang \
  --api-key "your-api-key" \
  --api-base "http://localhost:8180" \
  --api-model-name "/models/Phi-3-vision-128k-instruct" \
  --model-tokenizer "/models/Phi-3-vision-128k-instruct" \
  --task image-text-to-text \
  --max-time-per-run 15 \
  --max-requests-per-run 300 \
  --server-engine "SGLang" \
  --server-gpu-type "A100-80G" \
  --server-version "v0.4.10" \
  --server-gpu-count 4 \
  --traffic-scenario "I(256,256)" \
  --traffic-scenario "I(1024,1024)" \
  --num-concurrency 1 \
  --num-concurrency 8 \
  --dataset-config ./examples/dataset_configs/config_llava-bench-in-the-wild.json
```

--------------------------------

### Pull GenAI-Bench Docker Image

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/run-benchmark-using-docker.md

Pulls the latest pre-built Docker image for GenAI-Bench from the GitHub Container Registry. This is the simplest way to get started.

```shell
docker pull ghcr.io/moirai-internal/genai-bench:v0.0.3
```

--------------------------------

### Cross-Cloud Benchmarking (OpenAI to AWS S3)

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-auth-storage.md

Benchmark the OpenAI API and store the results in AWS S3. This example specifies OpenAI API details, S3 bucket, profile, and region.

```bash
genai-bench benchmark \
  --api-backend openai \
  --api-base https://api.openai.com/v1 \
  --api-key sk-... \
  --api-model-name gpt-4 \
  --model-tokenizer gpt2 \
  --task text-to-text \
  --max-requests-per-run 100 \
  --max-time-per-run 10 \
  --upload-results \
  --storage-provider aws \
  --storage-bucket my-benchmarks \
  --storage-aws-profile default \
  --storage-aws-region us-east-1
```

--------------------------------

### Configure Datasets for Benchmarking

Source: https://context7.com/sgl-project/genai-bench/llms.txt

Set up custom datasets for benchmarking from local files (e.g., CSV) or HuggingFace repositories. Supports simple CLI arguments or advanced JSON configuration files for detailed dataset specifications.

```bash
# Simple CLI usage with local CSV file
genai-bench benchmark \
  --dataset-path /path/to/data.csv \
  --dataset-prompt-column "prompt" \
  ...

# Simple CLI usage with HuggingFace dataset
genai-bench benchmark \
  --dataset-path "squad" \
  --dataset-prompt-column "question" \
  ...

# Advanced configuration using JSON file
genai-bench benchmark \
  --dataset-config ./examples/dataset_configs/huggingface_simple.json \
  ...
```

--------------------------------

### GenAI Bench CLI Commands Overview

Source: https://context7.com/sgl-project/genai-bench/llms.txt

Provides an overview of the main commands available in the GenAI Bench CLI: `benchmark` for running tests, `excel` for report generation, and `plot` for visualization. Help commands for each are also shown.

```bash
genai-bench --help

genai-bench benchmark --help

genai-bench excel --help

genai-bench plot --help
```

--------------------------------

### Benchmark GCP Vertex AI with Service Account

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-quick-reference.md

This example demonstrates benchmarking Google Cloud's Vertex AI. It uses a service account for authentication, specified via the GOOGLE_APPLICATION_CREDENTIALS environment variable. The command includes the backend, base URL, project ID, location, model, tokenizer, task, and performance metrics.

```bash
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
genai-bench benchmark \
  --api-backend gcp-vertex \
  --api-base https://us-central1-aiplatform.googleapis.com \
  --gcp-project-id my-project-123 \
  --gcp-location us-central1 \
  --api-model-name gemini-1.5-pro \
  --model-tokenizer google/gemini \
  --task text-to-text \
  --max-requests-per-run 100 \
  --max-time-per-run 10
```

--------------------------------

### Docker Usage for GenAI Bench

Source: https://context7.com/sgl-project/genai-bench/llms.txt

Demonstrates how to use Docker to run GenAI Bench, including pulling the latest image, building from source, and executing a benchmark run within a container, mounting local directories and setting environment variables for configuration.

```bash
# Pull the latest image
docker pull ghcr.io/moirai-internal/genai-bench:v0.0.2

# Build from source
docker build . -f Dockerfile -t genai-bench:dev

# Run benchmark in container
docker run --rm \
  -e HF_TOKEN="your-token" \
  -v $(pwd)/experiments:/app/experiments \
  genai-bench:dev benchmark \
  --api-backend openai \
  --api-base "http://host.docker.internal:8080" \
  --api-key "your-api-key" \
  --api-model-name "gpt-4" \
  --model-tokenizer "gpt2" \
  --task text-to-text \
  --max-time-per-run 5 \
  --max-requests-per-run 100
```

--------------------------------

### Create Docker Network for Benchmarking

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/run-benchmark-using-docker.md

Creates a bridge Docker network named 'benchmark-network'. This is recommended to isolate benchmarking traffic and avoid internet disruptions.

```shell
docker network create benchmark-network -d bridge
```

--------------------------------

### Build GenAI-Bench Docker Image from Source

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/run-benchmark-using-docker.md

Builds the GenAI-Bench Docker image locally using the provided Dockerfile. This allows for customization or using the latest development version.

```shell
docker build . -f Dockerfile -t genai-bench:dev
```

--------------------------------

### GitHub Releases Upload

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-auth-storage.md

Upload benchmark results as artifacts to a GitHub release. This requires a personal access token with repository permissions, along with the repository owner and name.

```bash
genai-bench benchmark \
  ... \
  --upload-results \
  --storage-provider github \
  --github-token ghp_xxxxxxxxxxxxxxxxxxxx \
  --github-owner myorg \
  --github-repo benchmark-results
```

--------------------------------

### Benchmark OpenAI API with API Key

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-quick-reference.md

This snippet shows how to benchmark the OpenAI API using a direct API key. It specifies the backend, base URL, API key, model name, tokenizer, task, and performance metrics. The API key can be provided directly or via an environment variable for backward compatibility.

```bash
genai-bench benchmark \
  --api-backend openai \
  --api-base https://api.openai.com/v1 \
  --api-key sk-... \
  --api-model-name gpt-4 \
  --model-tokenizer gpt2 \
  --task text-to-text \
  --max-requests-per-run 100 \
  --max-time-per-run 10
```

```bash
export MODEL_API_KEY=sk-...
genai-bench benchmark \
  --api-backend openai \
  --api-base https://api.openai.com/v1 \
  --api-model-name gpt-4 \
  --model-tokenizer gpt2 \
  --task text-to-text \
  --max-requests-per-run 100 \
  --max-time-per-run 10
```

--------------------------------

### Basic Text Benchmark: SGLang vs. GenAI Bench

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/getting-started/migrating-from-sglang-vllm.md

This snippet shows the equivalent commands for performing a basic text benchmark. SGLang uses specific parameters for input/output lengths, while GenAI Bench uses a traffic scenario string to define token distributions.

```bash
python -m sglang.bench_serving \
    --backend sglang-oai-chat \
    --num-prompts 500 \
    --random-input-len 512 \
    --random-output-len 512 \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --base-url http://127.0.0.1:8080
```

```bash
genai-bench benchmark \
    --api-backend sglang \
    --api-base http://127.0.0.1:8080 \
    --api-model-name sglang-model \
    --model-tokenizer meta-llama/Llama-3.1-8B-Instruct \
    --task text-to-text \
    --max-requests-per-run 500 \
    --max-time-per-run 30 \
    --traffic-scenario "U(1,512)/(0,512)" \
    --num-concurrency 1 \
    --server-engine "SGLang"
```

--------------------------------

### AWS Bedrock AWS Profile Benchmark

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-auth-storage.md

Benchmarks AWS Bedrock models using a named AWS profile from the credentials file. Requires specifying the AWS profile name and region.

```bash
genai-bench benchmark \
  --api-backend aws-bedrock \
  --api-base https://bedrock-runtime.us-west-2.amazonaws.com \
  --aws-profile production \
  --aws-region us-west-2 \
  --api-model-name amazon.titan-text-express-v1 \
  --model-tokenizer amazon/titan \
  --task text-to-text \
  --max-requests-per-run 100 \
  --max-time-per-run 10
```

--------------------------------

### Upload Benchmark Results to Cloud Storage

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-quick-reference.md

These snippets illustrate how to upload benchmark results to various cloud storage providers: OCI Object Storage, AWS S3, Azure Blob Storage, and GCP Cloud Storage. Each example uses the `--upload-results` flag along with provider-specific parameters for bucket, prefix, and authentication.

```bash
genai-bench benchmark \
  ... \
  --upload-results \
  --storage-provider oci \
  --storage-bucket my-benchmarks \
  --storage-prefix experiments/2024 \
  --namespace my-namespace
```

```bash
genai-bench benchmark \
  ... \
  --upload-results \
  --storage-provider aws \
  --storage-bucket my-benchmarks \
  --storage-prefix experiments/2024 \
  --storage-aws-profile default
```

```bash
genai-bench benchmark \
  ... \
  --upload-results \
  --storage-provider azure \
  --storage-bucket my-container \
  --storage-azure-account-name myaccount \
  --storage-azure-account-key YOUR_ACCOUNT_KEY
```

```bash
genai-bench benchmark \
  ... \
  --upload-results \
  --storage-provider gcp \
  --storage-bucket my-benchmarks \
  --storage-gcp-project-id my-project \
  --storage-gcp-credentials-path /path/to/service-account.json
```

--------------------------------

### Environment Variables for Storage Authentication

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-quick-reference.md

This section details environment variables for authenticating with different storage providers, including Azure Storage and GitHub. These are necessary for uploading benchmark results to remote storage.

```bash
# Azure Storage
export AZURE_STORAGE_ACCOUNT_NAME=myaccount
export AZURE_STORAGE_ACCOUNT_KEY=...
export AZURE_STORAGE_CONNECTION_STRING=...

# GitHub
export GITHUB_TOKEN=ghp_...
export GITHUB_OWNER=myorg
export GITHUB_REPO=benchmarks
```

--------------------------------

### OCI GenAI Dedicated Endpoint Benchmark

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-auth-storage.md

Benchmarks OCI GenAI models using a dedicated endpoint. Requires 'user_principal' authentication and specifies 'DEDICATED' serving type along with the endpoint ID.

```bash
genai-bench benchmark \
  --api-backend oci-genai \
  --api-base https://inference.generativeai.us-ashburn-1.oci.oraclecloud.com \
  --auth user_principal \
  --api-model-name xai.grok-3-mini-fast \
  --model-tokenizer gpt2 \
  --task text-to-text \
  --additional-request-params '{"compartmentId": "ocid1.compartment.oc1..example", "servingType": "DEDICATED", "endpointId": "ocid1.endpoint.oc1..example"}' \
  --max-requests-per-run 100 \
  --max-time-per-run 10
```

--------------------------------

### Benchmark Azure OpenAI with API Key

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-quick-reference.md

This snippet shows how to benchmark Azure OpenAI services. It requires specifying the backend, base URL, Azure endpoint, Azure deployment name, model API key, model name, tokenizer, task, and performance metrics.

```bash
genai-bench benchmark \
  --api-backend azure-openai \
  --api-base https://myresource.openai.azure.com \
  --azure-endpoint https://myresource.openai.azure.com \
  --azure-deployment my-gpt-4-deployment \
  --model-api-key YOUR_API_KEY \
  --api-model-name gpt-4 \
  --model-tokenizer gpt2 \
  --task text-to-text \
  --max-requests-per-run 100 \
  --max-time-per-run 10
```

--------------------------------

### Azure Blob Storage Upload (Connection String)

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-auth-storage.md

Upload benchmark results to Azure Blob Storage using a connection string for authentication. This is an alternative to using the account key and provides a more comprehensive set of credentials.

```bash
genai-bench benchmark \
  ... \
  --upload-results \
  --storage-provider azure \
  --storage-bucket my-container \
  --storage-azure-connection-string "DefaultEndpointsProtocol=https;AccountName=..."
```

--------------------------------

### Azure Blob Storage Upload (Account Key)

Source: https://github.com/sgl-project/genai-bench/blob/main/docs/user-guide/multi-cloud-auth-storage.md

Upload benchmark results to Azure Blob Storage using an account key for authentication. This method requires specifying the storage account name and the account key.

```bash
genai-bench benchmark \
  ... \
  --upload-results \
  --storage-provider azure \
  --storage-bucket my-container \
  --storage-prefix experiments/2024 \
  --storage-azure-account-name mystorageaccount \
  --storage-azure-account-key YOUR_ACCOUNT_KEY
```