infermark

https://github.com/stef41/infermark
Admin
Infermark is a command-line benchmarking tool for OpenAI-compatible LLM endpoints that measures...
Tokens:8,876
Snippets:64
Trust Score:7.5
License:Apache-2.0
Update:5 months ago
Show doc for...
Tokens:
Raw
### Install infermark

Source: https://github.com/stef41/infermark/blob/main/README.md

Install the infermark package using pip. For enhanced CLI features like rich tables and progress indicators, install with the 'cli' extra.

```bash
pip install infermark
```

```bash
pip install infermark[cli]
```

--------------------------------

### Backend API for Inference Requests

Source: https://context7.com/stef41/infermark/llms.txt

Shows how to use `OpenAIBackend` for OpenAI-compatible endpoints and `VLLMBackend` for vLLM's native API to send inference requests. Includes example of `detect_backend`.

```python
from infermark import OpenAIBackend, VLLMBackend, TGIBackend, detect_backend

# OpenAI-compatible backend (vLLM, Ollama, llama.cpp, etc.)
openai_b = OpenAIBackend(
    url="http://localhost:8000",
    model="meta-llama/Llama-3-8B-Instruct",
    api_key="",  # omit for local servers
)
result = openai_b.send_request("Say hello.", {"max_tokens": 32, "temperature": 0.7})
print(f"Success: {result.success}  Latency: {result.latency:.3f}s  "
      f"Tokens: {result.output_tokens}  TPS: {result.tokens_per_second:.1f}")

# vLLM native API (/v1/completions with vLLM-specific params)
vllm_b = VLLMBackend(url="http://localhost:8000", model="llama-3-8b")
result = vllm_b.send_request("What is 2+2?", {
    "max_tokens": 16, "top_k": 50, "use_beam_search": False
})
print(f"vLLM  ok={result.success}  lat={result.latency:.3f}s")
```

--------------------------------

### CLI Comparison Expected Output

Source: https://context7.com/stef41/infermark/llms.txt

Example of the terminal output when using `infermark compare`, showing a formatted table comparing endpoint performance metrics across different concurrency levels.

```bash
# Expected output:
# === Endpoint Comparison ===
#
# Concurrency: 1
# Endpoint                                            Tok/s    P50ms    P95ms     TTFT   Err%
# ------------------------------------------------------------------------------------------
# http://gpu1:8000/v1 (llama-3)                       54.3    470.1    520.4     80.2    0.0%
# http://gpu2:8080/v1 (llama-3)                       48.1    510.0    570.3     95.0    0.0%
# http://gpu3:11434/v1 (llama-3)                      31.2    700.5    810.2    140.0    0.0%
#
# Highest throughput: http://gpu1:8000/v1 (llama-3) at 487.3 tok/s
```

--------------------------------

### CLI Benchmark Expected Terminal Output

Source: https://context7.com/stef41/infermark/llms.txt

Example of the terminal output when running an `infermark run` command, showing benchmarking progress and performance metrics for different concurrency levels.

```bash
# Expected terminal output:
# Benchmarking http://localhost:8000/v1
#   model=meta-llama/...  requests=50  levels=[1, 4, 8, 16, 32]
#
#   concurrency=  1  tok/s=    54.3  p50=  470.1ms  ok=50/50
#   concurrency=  4  tok/s=   201.4  p50=  510.8ms  ok=50/50
#   concurrency= 32  tok/s=   487.3  p50=  870.5ms  ok=49/50
```

--------------------------------

### Programmatic Report Comparison and Formatting

Source: https://context7.com/stef41/infermark/llms.txt

Use `compare_reports` to programmatically compare multiple `BenchmarkReport` objects and get structured results. Use `format_comparison_text` to render these results as a plain-text table.

```python
from infermark import (
    BenchmarkConfig, run_benchmark,
    compare_reports, format_comparison_text,
    load_json,
)
from infermark._types import BenchmarkReport, ConcurrencyResult, LatencyStats
```

--------------------------------

### Define and Run Load Test Plan

Source: https://context7.com/stef41/infermark/llms.txt

Sets up a load test profile with ramp-up, hold, and cooldown phases, then runs it against a callable. Requires httpx for making requests.

```python
import httpx
from infermark import (
    LoadProfile, LoadTestPlan, LoadTestRunner, format_load_report
)

# Define a ramp: 1 → 20 RPS over 10 s, hold 20 s, cool down 5 s
profile = LoadProfile(
    initial_rps=1.0,
    target_rps=20.0,
    ramp_duration_s=10.0,
    hold_duration_s=20.0,
    cooldown_s=5.0,
)
plan = LoadTestPlan(profile)
print(f"Total test duration: {plan.total_duration:.0f}s")  # 35s
print(f"RPS at t=5s  : {plan.rps_at(5):.1f}")   # 10.5
print(f"RPS at t=15s : {plan.rps_at(15):.1f}")  # 20.0
print(f"RPS at t=32s : {plan.rps_at(32):.1f}")  # 12.0

# The callable to benchmark — any zero-arg function that raises on failure
def call_llm() -> None:
    resp = httpx.post(
        "http://localhost:8000/v1/chat/completions",
        json={"model": "llama-3", "messages": [{"role": "user", "content": "Hi"}],
              "max_tokens": 16},
        timeout=30.0,
    )
    resp.raise_for_status()

runner = LoadTestRunner(plan, call_llm)
results = runner.run_sync()

summary = runner.summary()
print(format_load_report(summary))
```

--------------------------------

### Benchmark Local vLLM Server (CLI)

Source: https://github.com/stef41/infermark/blob/main/README.md

Run a benchmark against a local vLLM server. Specify the API endpoint, model, and the number of requests to perform.

```bash
# Benchmark a local vLLM server
infermark run http://localhost:8000/v1 --model meta-llama/Llama-3-70B -n 50
```

--------------------------------

### Advanced CLI Benchmark with Options

Source: https://context7.com/stef41/infermark/llms.txt

Run a benchmark with concurrency sweep, saving results to JSON, and exporting to Markdown. Configure model, concurrency levels, max tokens, mode, warmup, and timeout.

```bash
# Sweep concurrency levels and save JSON
infermark run http://localhost:8000/v1 \
  --model meta-llama/Llama-3-8B-Instruct \
  -c 1,4,8,16,32,64 \
  -n 100 \
  --max-tokens 256 \
  --mode streaming \
  --warmup 3 \
  --timeout 120 \
  -o results.json \
  --markdown report.md
```

--------------------------------

### Detailed Benchmark Configuration (Python)

Source: https://github.com/stef41/infermark/blob/main/README.md

Illustrates a comprehensive set of configuration options for the BenchmarkConfig class, including API URL, model, prompt, token limits, concurrency, request counts, timeouts, and measurement modes.

```python
BenchmarkConfig(
    url="http://localhost:8000/v1",     # Any OpenAI-compatible endpoint
    model="meta-llama/Llama-3-70B",     # Model name
    prompt="Explain relativity.",        # Prompt to send
    max_tokens=256,                      # Max output tokens per request
    concurrency_levels=[1, 4, 8, 16],   # Test these concurrency levels
    n_requests=100,                      # Requests per level
    timeout=120.0,                       # Per-request timeout (seconds)
    mode=BenchmarkMode.STREAMING,        # STREAMING or NON_STREAMING
    warmup=3,                            # Warmup requests before measurement
    api_key="sk-…",                    # Optional API key
)
```

--------------------------------

### Basic CLI Benchmark Run

Source: https://context7.com/stef41/infermark/llms.txt

Perform a basic benchmark against a local vLLM server using the `infermark run` command. Specify the endpoint, model, and number of requests.

```bash
# Basic run against a local vLLM server
infermark run http://localhost:8000/v1 --model meta-llama/Llama-3-70B -n 50
```

--------------------------------

### Configure Benchmark with BenchmarkConfig

Source: https://context7.com/stef41/infermark/llms.txt

Define benchmark parameters including endpoint URL, model, prompt, concurrency levels, and request settings. Validation errors are raised immediately upon construction.

```python
from infermark import BenchmarkConfig, BenchmarkMode

config = BenchmarkConfig(
    url="http://localhost:8000/v1",          # Any OpenAI-compatible base URL
    model="meta-llama/Llama-3-70B-Instruct",
    prompt="Explain the theory of relativity in simple terms.",
    max_tokens=256,
    concurrency_levels=[1, 4, 8, 16, 32],   # Sweep these levels in order
    n_requests=100,                           # Requests per concurrency level
    timeout=120.0,                            # Per-request timeout (seconds)
    mode=BenchmarkMode.STREAMING,             # STREAMING or NON_STREAMING
    warmup=3,                                 # Warmup requests before measurement
    api_key="sk-...",                         # Optional; omit for local servers
    extra_body={"top_p": 0.9},               # Backend-specific extra fields
)

# Validation errors raised immediately
try:
    bad = BenchmarkConfig(url="", model="llama")
except ValueError as e:
    print(e)  # url must not be empty
```

--------------------------------

### Run Benchmark with Python Configuration

Source: https://github.com/stef41/infermark/blob/main/README.md

Configure and run a benchmark programmatically using Python. Define parameters such as the API URL, model, concurrency levels, and number of requests.

```python
from infermark import BenchmarkConfig, run_benchmark

config = BenchmarkConfig(
    url="http://localhost:8000/v1",
    model="meta-llama/Llama-3-70B-Instruct",
    concurrency_levels=[1, 4, 8, 16, 32],
    n_requests=100,
    max_tokens=256,
)

report = run_benchmark(config)

# Best throughput
best = report.best_throughput()
print(f"Peak: {best.tokens_per_second:.1f} tok/s at concurrency {best.concurrency}")

# Lowest latency
low = report.lowest_latency()
print(f"Lowest P50: {low.latency.p50 * 1000:.1f} ms at concurrency {low.concurrency}")
```

--------------------------------

### Benchmark and Compare Endpoints (CLI)

Source: https://github.com/stef41/infermark/blob/main/README.md

Benchmark multiple LLM endpoints individually and then compare their performance side-by-side using the 'infermark compare' command.

```bash
# Benchmark each endpoint separately
infermark run http://gpu1:8000/v1 --model llama-3 -o vllm.json
infermark run http://gpu2:8080/v1 --model llama-3 -o tgi.json
infermark run http://gpu3:11434/v1 --model llama-3 -o ollama.json

# Side-by-side comparison
infermark compare vllm.json tgi.json ollama.json
```

--------------------------------

### Compare Multiple Endpoints (CLI)

Source: https://github.com/stef41/infermark/blob/main/README.md

Compare the performance of multiple LLM endpoints by providing their respective result files generated from previous benchmarks.

```bash
# Compare multiple endpoints
infermark compare vllm.json tgi.json ollama.json
```

--------------------------------

### Run Benchmark and Analyze Results with Python

Source: https://context7.com/stef41/infermark/llms.txt

Use the `run_benchmark` function to execute benchmarks and iterate through the results. Access metrics like tokens per second, latency, and error rates for each concurrency level.

```python
from infermark import BenchmarkConfig, run_benchmark

report = run_benchmark(BenchmarkConfig(
    url="http://localhost:8000/v1",
    model="llama-3",
    concurrency_levels=[1, 4, 8, 16],
    n_requests=50,
))

# Iterate all concurrency results
for r in report.results:
    ttft_p50 = f"{r.ttft.p50 * 1000:.1f}ms" if r.ttft else "N/A"
    itl_p50  = f"{r.itl.p50 * 1000:.1f}ms"  if r.itl  else "N/A"
    print(
        f"conc={r.concurrency:>3}  ok={r.n_success}/{r.n_requests}  "
        f"tok/s={r.tokens_per_second:.1f}  "
        f"p99={r.latency.p99 * 1000:.1f}ms  "
        f"ttft={ttft_p50}  itl={itl_p50}  "
        f"err_rate={r.n_error/r.n_requests:.1%}"
    )

best = report.best_throughput()
print(f"\nBest: {best.tokens_per_second:.1f} tok/s @ concurrency {best.concurrency}")
```

--------------------------------

### Export Benchmark Results (CLI)

Source: https://github.com/stef41/infermark/blob/main/README.md

Export benchmark results in different formats. Use JSON for programmatic analysis or Markdown for easy integration into documentation.

```bash
# JSON (for programmatic analysis)
infermark run http://localhost:8000/v1 -o report.json

# Markdown (paste into docs/PRs)
infermark run http://localhost:8000/v1 --markdown report.md
```

--------------------------------

### Interact with TGI Backend

Source: https://context7.com/stef41/infermark/llms.txt

Demonstrates sending a request to a TGI backend and printing the result. Requires a running TGI server at the specified URL.

```python
from infermark import TGIBackend, detect_backend

tgi_b = TGIBackend(url="http://localhost:8080")
result = tgi_b.send_request("Explain gravity.", {
    "max_tokens": 64, "top_p": 0.9, "repetition_penalty": 1.1
})
print(f"TGI   ok={result.success}  tokens={result.output_tokens}")

backend_type = detect_backend("http://localhost:8000")
print(f"Detected backend: {backend_type}")  # "vllm", "tgi", "openai", or "unknown"
```

--------------------------------

### Report Serialization and Export

Source: https://context7.com/stef41/infermark/llms.txt

Demonstrates saving and loading benchmark reports using JSON, and formatting/saving them as CSV or Markdown. Requires `BenchmarkConfig`, `run_benchmark`, and serialization functions.

```python
from infermark import (
    BenchmarkConfig, run_benchmark,
    save_json, load_json,
    format_csv, save_csv,
    format_markdown,
)
from pathlib import Path

config = BenchmarkConfig(
    url="http://localhost:8000/v1",
    model="test-model",
    concurrency_levels=[1, 4],
    n_requests=20,
    warmup=0,
)
report = run_benchmark(config)

# JSON round-trip
save_json(report, "report.json")
data = load_json("report.json")
print(f"Loaded {len(data['results'])} concurrency levels from JSON")
for r in data["results"]:
    print(f"  conc={r['concurrency']}  tok/s={r['tokens_per_second']:.1f}  "
          f"p50={r['latency']['p50']*1000:.1f}ms")

# CSV export (one row per concurrency level)
save_csv(report, "report.csv")
csv_str = format_csv(report)
# url,model,timestamp,concurrency,n_requests,n_success,n_error,...,latency_p50_ms,...

# Markdown export (for docs / PRs)
md = format_markdown(report)
Path("report.md").write_text(md)
print(md[:300])
# # Benchmark Report
# - **URL:** http://localhost:8000/v1
# - **Model:** test-model
# | Conc | Reqs | OK | Err | RPS | Tok/s | P50 (ms) | P95 (ms) | P99 (ms) | ...
```

--------------------------------

### Compare Benchmark Reports Side-by-Side

Source: https://context7.com/stef41/infermark/llms.txt

Use the `infermark compare` command to load multiple JSON reports generated by `infermark run -o` and display a side-by-side comparison table, sorted by throughput per concurrency level.

```bash
# Side-by-side comparison
infermark compare vllm.json tgi.json ollama.json
```

--------------------------------

### Save Results as JSON (CLI)

Source: https://github.com/stef41/infermark/blob/main/README.md

Execute a benchmark and save the detailed results to a JSON file for programmatic analysis.

```bash
# Save results as JSON
infermark run http://localhost:8000/v1 -o results.json
```

--------------------------------

### Benchmark Multiple Backends Separately for Comparison

Source: https://context7.com/stef41/infermark/llms.txt

Use `infermark run -o` to benchmark different model backends individually and save their results to separate JSON files. This prepares data for the `infermark compare` command.

```bash
# Benchmark three backends separately
infermark run http://gpu1:8000/v1  --model llama-3 -n 100 -o vllm.json
infermark run http://gpu2:8080/v1  --model llama-3 -n 100 -o tgi.json
infermark run http://gpu3:11434/v1 --model llama-3 -n 100 -o ollama.json
```

--------------------------------

### run_benchmark

Source: https://context7.com/stef41/infermark/llms.txt

The synchronous entry point for running benchmarks. It executes a full sweep across configured concurrency levels, performs warmup requests, and returns a BenchmarkReport.

```APIDOC
## run_benchmark

`run_benchmark` is the synchronous entry point. It runs a full sweep across all concurrency levels, fires warmup requests, then returns a `BenchmarkReport`.

```python
from infermark import BenchmarkConfig, run_benchmark, format_report_text

config = BenchmarkConfig(
    url="http://localhost:8000/v1",
    model="meta-llama/Llama-3-8B-Instruct",
    prompt="Explain the theory of relativity in simple terms.",
    max_tokens=128,
    concurrency_levels=[1, 4, 8, 16],
    n_requests=50,
    warmup=2,
    timeout=60.0,
)

def on_progress(concurrency: int, result) -> None:
    print(
        f"  concurrency={concurrency:>3}  "
        f"rps={result.requests_per_second:.1f}  "
        f"tok/s={result.tokens_per_second:.1f}  "
        f"p50={result.latency.p50 * 1000:.0f}ms  "
        f"errors={result.n_error}"
    )

report = run_benchmark(config, on_progress=on_progress)

# Example output:
#   concurrency=  1  rps=2.1  tok/s=54.3  p50=470ms  errors=0
#   concurrency=  4  rps=7.8  tok/s=201.4  p50=510ms  errors=0
#   concurrency= 16  rps=18.2  tok/s=463.0  p50=870ms  errors=0

print(format_report_text(report))
# Benchmark: http://localhost:8000/v1
# Model: meta-llama/Llama-3-8B-Instruct
#  Conc  Reqs    OK  Err     RPS    Tok/s    P50ms    P95ms    P99ms  TTFT-P50  ITL-P50
# -----------------------------------------------------------------------
#     1    50    50     0     2.1     54.3    470.1    520.4    541.0      80.2     18.0
#    16    50    50     0    18.2    463.0    870.5   1100.2   1200.0      90.1     22.5
```
```

--------------------------------

### Load and Compare Benchmark Reports

Source: https://context7.com/stef41/infermark/llms.txt

Loads two JSON benchmark reports, compares them, and prints a formatted summary. Requires `load_json` and `compare_reports` functions.

```python
from infermark import BenchmarkReport, ConcurrencyResult, LatencyStats, load_json, compare_reports, format_comparison_text

def load_report(path: str) -> BenchmarkReport:
    data = load_json(path)
    results = []
    for r in data["results"]:
        results.append(ConcurrencyResult(
            concurrency=r["concurrency"],
            n_requests=r["n_requests"],
            n_success=r["n_success"],
            n_error=r["n_error"],
            total_duration=r["total_duration"],
            requests_per_second=r["requests_per_second"],
            tokens_per_second=r["tokens_per_second"],
            latency=LatencyStats(**r["latency"]),
            ttft=LatencyStats(**r["ttft"]) if r.get("ttft") else None,
            itl=LatencyStats(**r["itl"]) if r.get("itl") else None,
            errors=r.get("errors", {})
        ))
    return BenchmarkReport(url=data["url"], model=data["model"],
                           timestamp=data["timestamp"], results=results)

reports = [load_report("vllm.json"), load_report("tgi.json")]

comp = compare_reports(reports)
for level, entries in sorted(comp.items(), key=lambda x: int(x[0])):
    print(f"Concurrency {level}:")
    for e in sorted(entries, key=lambda x: -x["tokens_per_second"]):
        print(f"  {e['url']}  tok/s={e['tokens_per_second']:.1f}  "
              f"p50={e['latency_p50']*1000:.1f}ms  err={e['error_rate']:.1%}")

print(format_comparison_text(reports))
```

--------------------------------

### Run Synchronous Benchmark with run_benchmark

Source: https://context7.com/stef41/infermark/llms.txt

Execute a synchronous benchmark sweep across configured concurrency levels. Includes warmup requests and provides a `BenchmarkReport`. An optional `on_progress` callback can display real-time results.

```python
from infermark import BenchmarkConfig, run_benchmark, format_report_text

config = BenchmarkConfig(
    url="http://localhost:8000/v1",
    model="meta-llama/Llama-3-8B-Instruct",
    prompt="Explain the theory of relativity in simple terms.",
    max_tokens=128,
    concurrency_levels=[1, 4, 8, 16],
    n_requests=50,
    warmup=2,
    timeout=60.0,
)

def on_progress(concurrency: int, result) -> None:
    print(
        f"  concurrency={concurrency:>3}  "
        f"rps={result.requests_per_second:.1f}  "
        f"tok/s={result.tokens_per_second:.1f}  "
        f"p50={result.latency.p50 * 1000:.0f}ms  "
        f"errors={result.n_error}"
    )

report = run_benchmark(config, on_progress=on_progress)

# Example output:
#   concurrency=  1  rps=2.1  tok/s=54.3  p50=470ms  errors=0
#   concurrency=  4  rps=7.8  tok/s=201.4  p50=510ms  errors=0
#   concurrency= 16  rps=18.2  tok/s=463.0  p50=870ms  errors=0

print(format_report_text(report))
```

--------------------------------

### CLI Benchmark with API Key and Non-Streaming Mode

Source: https://context7.com/stef41/infermark/llms.txt

Benchmark a non-streaming endpoint like OpenAI's API, providing an API key and specifying non-streaming mode. Adjust concurrency and request counts as needed.

```bash
# Non-streaming mode with API key
infermark run https://api.openai.com/v1 \
  --model gpt-4o \
  --api-key sk-... \
  --mode non_streaming \
  -n 20 -c 1,2,4
```

--------------------------------

### Run Benchmark Asynchronously with Python

Source: https://github.com/stef41/infermark/blob/main/README.md

Execute a benchmark asynchronously using Python's asyncio. This is useful for integrating benchmarking into larger asynchronous applications.

```python
import asyncio
from infermark import BenchmarkConfig, run_benchmark_async

async def main():
    config = BenchmarkConfig(url="http://localhost:8000/v1", model="llama-3")
    report = await run_benchmark_async(config)
    print(f"Peak throughput: {report.best_throughput().tokens_per_second:.1f} tok/s")

asyncio.run(main())
```

--------------------------------

### BenchmarkConfig

Source: https://context7.com/stef41/infermark/llms.txt

The central configuration dataclass for setting up benchmark parameters such as endpoint URL, model, prompts, concurrency levels, and request settings.

```APIDOC
## BenchmarkConfig

`BenchmarkConfig` is the central configuration dataclass. It specifies the endpoint URL, model, prompt, concurrency sweep, and request parameters. Validation runs at construction time.

```python
from infermark import BenchmarkConfig, BenchmarkMode

config = BenchmarkConfig(
    url="http://localhost:8000/v1",          # Any OpenAI-compatible base URL
    model="meta-llama/Llama-3-70B-Instruct",
    prompt="Explain the theory of relativity in simple terms.",
    max_tokens=256,
    concurrency_levels=[1, 4, 8, 16, 32],   # Sweep these levels in order
    n_requests=100,                           # Requests per concurrency level
    timeout=120.0,                            # Per-request timeout (seconds)
    mode=BenchmarkMode.STREAMING,             # STREAMING or NON_STREAMING
    warmup=3,                                 # Warmup requests before measurement
    api_key="sk-...",                         # Optional; omit for local servers
    extra_body={"top_p": 0.9},               # Backend-specific extra fields
)

# Validation errors raised immediately
try:
    bad = BenchmarkConfig(url="", model="llama")
except ValueError as e:
    print(e)  # url must not be empty
```
```

--------------------------------

### Run Asynchronous Benchmark with run_benchmark_async

Source: https://context7.com/stef41/infermark/llms.txt

Utilize the native asynchronous version of the benchmark runner within an asyncio event loop. Supports an `on_progress` callback for real-time updates and returns a `BenchmarkReport`.

```python
import asyncio
from infermark import BenchmarkConfig, run_benchmark_async

async def main():
    config = BenchmarkConfig(
        url="http://localhost:8000/v1",
        model="meta-llama/Llama-3-8B-Instruct",
        concurrency_levels=[1, 4, 8, 16, 32],
        n_requests=100,
    )

    def on_progress(concurrency: int, result) -> None:
        print(f"Done concurrency={concurrency}: {result.tokens_per_second:.1f} tok/s")

    report = await run_benchmark_async(config, on_progress=on_progress)

    best = report.best_throughput()
    low = report.lowest_latency()
    print(f"Peak throughput : {best.tokens_per_second:.1f} tok/s at concurrency {best.concurrency}")
    print(f"Lowest P50      : {low.latency.p50 * 1000:.1f} ms at concurrency {low.concurrency}")
    # Peak throughput : 487.3 tok/s at concurrency 32
    # Lowest P50      : 462.1 ms at concurrency 1

asyncio.run(main())
```

--------------------------------

### Sweep Concurrency Levels (CLI)

Source: https://github.com/stef41/infermark/blob/main/README.md

Benchmark an endpoint while sweeping through various concurrency levels to understand performance under different loads. Specify a comma-separated list of concurrency levels.

```bash
# Sweep concurrency levels
infermark run http://localhost:8000/v1 -c 1,4,8,16,32,64 -n 100
```

--------------------------------

### Analyze Latency with LatencyHistogram

Source: https://context7.com/stef41/infermark/llms.txt

Computes and renders ASCII histograms and CDF plots from latency data, and calculates exact percentiles. Requires latency samples in milliseconds.

```python
from infermark import (
    LatencyHistogram, HistogramConfig, compute_bins, format_histogram_report
)

# Latency samples in milliseconds
latencies_ms = [120, 135, 142, 118, 160, 210, 125, 130, 145, 155,
                170, 115, 190, 138, 127, 148, 200, 122, 133, 250]

cfg = HistogramConfig(bins=10, width=50, show_percentiles=True, unit="ms")
hist = LatencyHistogram(cfg)

# Compute and render ASCII histogram
data = hist.compute(latencies_ms)
print(hist.render_ascii(data))
# [  115.00,   150.50) | ████████████████████████████████████████████ 13
# [  150.50,   186.00) | ██████████████████████ 4
# ...
#   total=20  min=115.00ms  max=250.00ms  mean=151.65ms  std=35.21ms

# CDF plot
print(hist.render_cdf(latencies_ms))

# Exact percentiles (linear interpolation)
pcts = hist.percentiles(latencies_ms, pcts=[50, 90, 95, 99])
for p, v in pcts.items():
    print(f"  p{p}: {v:.1f} ms")  # p50: 138.5 ms  p95: 207.5 ms  p99: 247.5 ms

# Full summary dict
summary = hist.summary(latencies_ms)
print(f"count={summary['count']}  mean={summary['mean']:.1f}ms  "
      f"median={summary['median']:.1f}ms")

# compute_bins standalone
edges, counts = compute_bins(latencies_ms, n_bins=5)
for i, c in enumerate(counts):
    print(f"  [{edges[i]:.0f}, {edges[i+1]:.0f}) → {c} samples")
```

--------------------------------

### run_benchmark_async

Source: https://context7.com/stef41/infermark/llms.txt

The native asynchronous version of the benchmark runner. It can be used within existing asyncio event loops and accepts an optional `on_progress` callback.

```APIDOC
## run_benchmark_async

`run_benchmark_async` is the native async version. Use this inside existing asyncio event loops with an optional `on_progress` callback.

```python
import asyncio
from infermark import BenchmarkConfig, run_benchmark_async

async def main():
    config = BenchmarkConfig(
        url="http://localhost:8000/v1",
        model="meta-llama/Llama-3-8B-Instruct",
        concurrency_levels=[1, 4, 8, 16, 32],
        n_requests=100,
    )

    def on_progress(concurrency: int, result) -> None:
        print(f"Done concurrency={concurrency}: {result.tokens_per_second:.1f} tok/s")

    report = await run_benchmark_async(config, on_progress=on_progress)

    best = report.best_throughput()
    low = report.lowest_latency()
    print(f"Peak throughput : {best.tokens_per_second:.1f} tok/s at concurrency {best.concurrency}")
    print(f"Lowest P50      : {low.latency.p50 * 1000:.1f} ms at concurrency {low.concurrency}")
    # Peak throughput : 487.3 tok/s at concurrency 32
    # Lowest P50      : 462.1 ms at concurrency 1

asyncio.run(main())
```
```

--------------------------------

### Compute Latency Statistics

Source: https://context7.com/stef41/infermark/llms.txt

Calculates various latency statistics (P50-P99, mean, std) from raw latency samples using `compute_stats`. Also demonstrates computing arbitrary percentiles with `compute_percentiles` and a single percentile with `percentile`.

```python
from infermark import compute_stats, compute_percentiles, percentile

# Raw latency samples (seconds)
latencies = [0.45, 0.51, 0.48, 0.62, 0.47, 0.55, 0.90, 0.49, 0.50, 0.53]

stats = compute_stats(latencies)
print(f"P50  : {stats.p50 * 1000:.1f} ms")   # P50  : 505.0 ms
print(f"P95  : {stats.p95 * 1000:.1f} ms")   # P95  : 855.0 ms
print(f"P99  : {stats.p99 * 1000:.1f} ms")   # P99  : 891.0 ms
print(f"Mean : {stats.mean * 1000:.1f} ms")  # Mean : 550.0 ms
print(f"Std  : {stats.std * 1000:.1f} ms")   # Std  : 128.0 ms

# Arbitrary percentiles
pcts = compute_percentiles(latencies, percentiles=[50, 75, 90, 95, 99])
for p, v in pcts.items():
    print(f"  p{int(p)}: {v * 1000:.2f} ms")

# Single percentile
p90 = percentile(latencies, 90)
print(f"P90 (single): {p90 * 1000:.1f} ms")
```
Tokens:
Raw