### Quick Start Deployment

Source: https://github.com/aliez-ren/vllm-qwen3.5-nvfp4-sm120/blob/main/README.md

Commands to clone the repository, configure the environment, and launch the vLLM server using Docker Compose.

```bash
# Clone this repo
git clone https://github.com/Li-Lee/vllm-qwen3.5-nvfp4-5090.git
cd vllm-qwen3.5-nvfp4-5090

# Create your .env from the template
cp .env.example .env
# Edit .env with your HF token and cache path
vim .env

# Start the server
docker compose up -d

# Check logs (model loading takes ~5-10 min on first run)
docker compose logs -f
```

--------------------------------

### Clone Repository and Start Server

Source: https://context7.com/aliez-ren/vllm-qwen3.5-nvfp4-sm120/llms.txt

Clone the repository, set up environment variables in .env, and start the vLLM server using Docker Compose. Model loading may take 5-10 minutes on the first run.

```bash
# Clone the repository
git clone https://github.com/Li-Lee/vllm-qwen3.5-nvfp4-5090.git
cd vllm-qwen3.5-nvfp4-5090

# Create environment configuration
cp .env.example .env

# Edit .env with your settings
cat > .env << 'EOF'
HF_TOKEN=hf_your_huggingface_token_here
HF_CACHE=/home/user/.cache/huggingface
NVFP4_BACKEND=marlin
EOF

# Start the server (model loading takes ~5-10 min on first run)
docker compose up -d

# Monitor logs
docker compose logs -f

# Check server health
curl http://localhost:8000/health
# Expected output: {"status":"healthy"}
```

--------------------------------

### Legacy vLLM v0.16 Docker Compose Start

Source: https://context7.com/aliez-ren/vllm-qwen3.5-nvfp4-sm120/llms.txt

Command to start vLLM with a legacy v0.16 Docker Compose configuration. This setup includes specific patches for NVFP4 quantization compatibility.

```bash
# Start with vLLM v0.16 legacy configuration
docker compose -f docker-compose.v16.yml up -d
```

--------------------------------

### Performance Benchmarking with llama-benchy

Source: https://context7.com/aliez-ren/vllm-qwen3.5-nvfp4-sm120/llms.txt

Install and run llama-benchy to benchmark the vLLM deployment. Specify the model, context depths, and concurrency level for throughput measurements.

```bash
# Install llama-benchy
uvx llama-benchy --base-url http://localhost:8000/v1 \
  --model Kbenkhaled/Qwen3.5-27B-NVFP4 \
  --depth 2048 4096 8192 131072 \
  --concurrency 1
```

--------------------------------

### Python Client Streaming Example with vLLM

Source: https://context7.com/aliez-ren/vllm-qwen3.5-nvfp4-sm120/llms.txt

Shows how to receive chat completion responses in a streaming fashion using the OpenAI Python SDK. This is useful for real-time applications.

```python
# Streaming example
stream = client.chat.completions.create(
    model="Kbenkhaled/Qwen3.5-27B-NVFP4",
    messages=[{"role": "user", "content": "Explain recursion."}],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
```

--------------------------------

### Tool Calling Request with vLLM

Source: https://context7.com/aliez-ren/vllm-qwen3.5-nvfp4-sm120/llms.txt

Example cURL request to initiate a chat completion with tool calling enabled. Ensure the model and tool definitions match your requirements.

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Kbenkhaled/Qwen3.5-27B-NVFP4",
    "messages": [
      {"role": "user", "content": "What is the weather in San Francisco?"}
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get the current weather in a location",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, CA"
              },
              "unit": {
                "type": "string",
                "enum": ["celsius", "fahrenheit"]
              }
            },
            "required": ["location"]
          }
        }
      }
    ],
    "tool_choice": "auto"
  }'
```

--------------------------------

### vLLM Docker Compose Configuration

Source: https://context7.com/aliez-ren/vllm-qwen3.5-nvfp4-sm120/llms.txt

Key parameters for optimizing vLLM performance on RTX 5090 within a Docker Compose setup. Adjust `gpu-memory-utilization` and `max-num-seqs` based on your hardware and workload.

```yaml
# docker-compose.yml key parameters explained
services:
  vllm:
    image: vllm/vllm-openai:cu130-nightly
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - VLLM_NVFP4_GEMM_BACKEND=${NVFP4_BACKEND:-marlin}
    command:
      - >-
        vllm serve Kbenkhaled/Qwen3.5-27B-NVFP4
        --host=0.0.0.0
        --port=8000
        --max-model-len=234567        # 229K context window
        --gpu-memory-utilization=0.89 # Use ~28.5 GB of 32 GB VRAM
        --max-num-seqs=4              # Max concurrent sequences
        --max-num-batched-tokens=4096 # Per-batch token budget
        --kv-cache-dtype=fp8          # FP8 KV cache for memory efficiency
        --reasoning-parser=qwen3      # Qwen3 reasoning format support
        --enable-auto-tool-choice     # Automatic tool calling
        --enable-prefix-caching       # Cache common prefixes
        --tool-call-parser=qwen3_coder # Tool parsing for Qwen3
        --language-model-only         # Disable vision capabilities
```

--------------------------------

### Environment Configuration (.env file)

Source: https://context7.com/aliez-ren/vllm-qwen3.5-nvfp4-sm120/llms.txt

Configure the vLLM deployment by setting environment variables in the .env file, including Hugging Face token, cache path, and NVFP4 GEMM backend.

```bash
# .env file configuration

# Required: Hugging Face API token for accessing gated models
# Get your token from: https://huggingface.co/settings/tokens
HF_TOKEN=hf_your_token_here

# Required: Path to local Hugging Face cache directory
# This persists downloaded model weights between container restarts
HF_CACHE=/home/your_username/.cache/huggingface

# Optional: NVFP4 GEMM backend selection
# "marlin" (default) - Better text generation/decode throughput (~80 t/s)
# "flashinfer-cutlass" - Better prompt processing/prefill throughput
NVFP4_BACKEND=marlin
```

--------------------------------

### Run Benchmark with Concurrency 4

Source: https://github.com/aliez-ren/vllm-qwen3.5-nvfp4-sm120/blob/main/README.md

Execute the llama-benchy tool to benchmark the Qwen3.5-27B-NVFP4 model with a concurrency of 4. This command specifies the base URL for the vLLM server, the model to use, and a range of sequence depths to test.

```bash
uvx llama-benchy --base-url http://localhost:8000/v1 --model Kbenkhaled/Qwen3.5-27B-NVFP4 --depth 2048 4096 8192 131072 --concurrency 4
```

--------------------------------

### Benchmark Results with Flashinfer-cutlass Backend

Source: https://github.com/aliez-ren/vllm-qwen3.5-nvfp4-sm120/blob/main/README.md

Performance metrics for the Qwen3.5-27B-NVFP4 model using vLLM with the Flashinfer-cutlass GEMM backend and a concurrency of 4. This table presents throughput and time-to-first-token metrics for various sequence depths, highlighting significant performance gains over the Marlin backend.

```markdown
| model                        |                  test |       t/s (total) |         t/s (req) |      peak t/s |   peak t/s (req) |           ttfr (ms) |        est_ppt (ms) |       e2e_ttft (ms) |
|:-----------------------------|----------------------:|------------------:|------------------:|--------------:|-----------------:|--------------------:|--------------------:|--------------------:|
| Kbenkhaled/Qwen3.5-27B-NVFP4 |   pp2048 @ d2048 (c4) | 14471.27 ± 201.50 | 5104.29 ± 1713.70 |               |                  |     884.55 ± 238.76 |     881.05 ± 238.76 |     884.66 ± 238.72 |
| Kbenkhaled/Qwen3.5-27B-NVFP4 |     tg32 @ d2048 (c4) |     104.41 ± 4.71 |     41.84 ± 10.12 | 126.33 ± 0.47 |     43.71 ± 9.76 |                     |                     |                     |
| Kbenkhaled/Qwen3.5-27B-NVFP4 |   pp2048 @ d4096 (c4) |  14276.57 ± 41.90 | 5154.74 ± 1739.96 |               |                  |    1318.28 ± 371.24 |    1314.77 ± 371.24 |    1318.36 ± 371.22 |
| Kbenkhaled/Qwen3.5-27B-NVFP4 |     tg32 @ d4096 (c4) |      80.77 ± 0.15 |     37.34 ± 12.32 | 124.00 ± 0.00 |    40.34 ± 10.70 |                     |                     |                     |
| Kbenkhaled/Qwen3.5-27B-NVFP4 |   pp2048 @ d8192 (c4) |  13229.26 ± 44.49 | 5414.06 ± 2213.46 |               |                  |    2188.02 ± 742.92 |    2184.51 ± 742.92 |    2188.14 ± 742.89 |
| Kbenkhaled/Qwen3.5-27B-NVFP4 |     tg32 @ d8192 (c4) |      48.66 ± 0.14 |     28.59 ± 15.12 | 116.00 ± 0.00 |    34.99 ± 11.10 |                     |                     |                     |
| Kbenkhaled/Qwen3.5-27B-NVFP4 | pp2048 @ d131072 (c4) |   4445.64 ± 24.46 | 2326.61 ± 1307.39 |               |                  | 74729.35 ± 33575.52 | 74725.84 ± 33575.52 | 74729.50 ± 33575.52 |
| Kbenkhaled/Qwen3.5-27B-NVFP4 |   tg32 @ d131072 (c4) |       1.32 ± 0.01 |      53.51 ± 0.19 |  31.00 ± 0.00 |     55.32 ± 0.20 |                     |                     |                     |
```

--------------------------------

### Legacy vLLM v0.16 Deployment

Source: https://github.com/aliez-ren/vllm-qwen3.5-nvfp4-sm120/blob/main/README.md

Command to launch the server using the legacy configuration file for vLLM v0.16.

```bash
docker compose -f docker-compose.v16.yml up -d
```

--------------------------------

### Python Client Chat Completion with vLLM

Source: https://context7.com/aliez-ren/vllm-qwen3.5-nvfp4-sm120/llms.txt

Demonstrates how to use the OpenAI Python SDK to interact with a local vLLM server for chat completions. Initialize the client with the correct base URL and API key.

```python
from openai import OpenAI

# Initialize client pointing to local vLLM server
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed"  # vLLM doesn't require auth by default
)

# Chat completion
response = client.chat.completions.create(
    model="Kbenkhaled/Qwen3.5-27B-NVFP4",
    messages=[
        {"role": "system", "content": "You are a coding assistant."},
        {"role": "user", "content": "Write a Python function to calculate fibonacci numbers."}
    ],
    temperature=0.7,
    max_tokens=500
)

print(response.choices[0].message.content)
```

--------------------------------

### Benchmark Results with Marlin Backend

Source: https://github.com/aliez-ren/vllm-qwen3.5-nvfp4-sm120/blob/main/README.md

Performance metrics for the Qwen3.5-27B-NVFP4 model using vLLM with the Marlin GEMM backend and a concurrency of 4. The table shows throughput (t/s) for total and per-request, peak throughput, and time-to-first-token (ttfr) for various sequence depths.

```markdown
| model                        |                  test |     t/s (total) |        t/s (req) |      peak t/s |   peak t/s (req) |            ttfr (ms) |         est_ppt (ms) |        e2e_ttft (ms) |
|:-----------------------------|----------------------:|----------------:|-----------------:|--------------:|-----------------:|---------------------:|---------------------:|---------------------:|
| Kbenkhaled/Qwen3.5-27B-NVFP4 |   pp2048 @ d2048 (c4) | 4046.26 ± 16.44 | 1394.19 ± 446.44 |               |                  |     3196.22 ± 802.64 |     3191.98 ± 802.64 |     3196.32 ± 802.60 |
| Kbenkhaled/Qwen3.5-27B-NVFP4 |     tg32 @ d2048 (c4) |    47.73 ± 0.08 |    35.14 ± 20.57 | 124.00 ± 0.00 |    41.90 ± 16.37 |                      |                      |                      |
| Kbenkhaled/Qwen3.5-27B-NVFP4 |   pp2048 @ d4096 (c4) | 3979.38 ± 15.12 | 1412.82 ± 457.28 |               |                  |    4759.99 ± 1281.47 |    4755.75 ± 1281.47 |    4760.10 ± 1281.42 |
| Kbenkhaled/Qwen3.5-27B-NVFP4 |     tg32 @ d4096 (c4) |    32.72 ± 0.09 |    30.29 ± 22.89 | 121.00 ± 0.00 |    39.99 ± 16.97 |                      |                      |                      |
| Kbenkhaled/Qwen3.5-27B-NVFP4 |   pp2048 @ d8192 (c4) |  3876.30 ± 3.28 | 1552.86 ± 630.52 |               |                  |    7593.40 ± 2530.48 |    7589.16 ± 2530.48 |    7593.49 ± 2530.42 |
| Kbenkhaled/Qwen3.5-27B-NVFP4 |     tg32 @ d8192 (c4) |    17.57 ± 0.03 |    23.94 ± 25.09 | 116.00 ± 0.00 |    38.19 ± 17.77 |                      |                      |                      |
| Kbenkhaled/Qwen3.5-27B-NVFP4 | pp2048 @ d131072 (c4) |  2477.61 ± 2.08 | 1284.53 ± 714.02 |               |                  | 134608.95 ± 59943.20 | 134604.71 ± 59943.20 | 134609.12 ± 59943.22 |
| Kbenkhaled/Qwen3.5-27B-NVFP4 |   tg32 @ d131072 (c4) |     0.72 ± 0.00 |     69.66 ± 0.45 |  30.00 ± 0.00 |     72.11 ± 0.45 |                      |                      |                      |
```

--------------------------------

### List Available Models with vLLM

Source: https://context7.com/aliez-ren/vllm-qwen3.5-nvfp4-sm120/llms.txt

Use cURL to query the vLLM server for a list of available models. This helps confirm that your model has been loaded correctly.

```bash
curl http://localhost:8000/v1/models
```

--------------------------------

### Verify Legacy Patch Application

Source: https://context7.com/aliez-ren/vllm-qwen3.5-nvfp4-sm120/llms.txt

Check Docker Compose logs to confirm that the necessary patches for vLLM v0.16 compatibility, particularly for NVFP4 quantization, have been applied successfully.

```bash
# Check if patch was applied successfully in logs
docker compose -f docker-compose.v16.yml logs | grep -i patch
```

--------------------------------

### Benchmark Model Concurrency

Source: https://context7.com/aliez-ren/vllm-qwen3.5-nvfp4-sm120/llms.txt

Execute a concurrency test against the local model server using the llama-benchy tool.

```bash
uvx llama-benchy --base-url http://localhost:8000/v1 \
  --model Kbenkhaled/Qwen3.5-27B-NVFP4 \
  --depth 4096 \
  --concurrency 4
```

--------------------------------

### Text Completion Request

Source: https://context7.com/aliez-ren/vllm-qwen3.5-nvfp4-sm120/llms.txt

Send a text completion request to the legacy completions endpoint for raw text generation. Adjust `max_tokens` and `temperature` as needed.

```bash
# Text completion request
curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Kbenkhaled/Qwen3.5-27B-NVFP4",
    "prompt": "The benefits of open source software include:",
    "max_tokens": 256,
    "temperature": 0.8,
    "top_p": 0.95
  }'

# Expected response:
# {
#   "id": "cmpl-...",
#   "object": "text_completion",
#   "choices": [{
#     "text": " transparency, community collaboration, cost savings...",
#     "index": 0,
#     "finish_reason": "stop"
#   }],
#   "usage": {"prompt_tokens": 10, "completion_tokens": 100, "total_tokens": 110}
# }
```

--------------------------------

### Basic Chat Completion Request

Source: https://context7.com/aliez-ren/vllm-qwen3.5-nvfp4-sm120/llms.txt

Send a basic chat completion request to the vLLM server using the OpenAI API format. Ensure the model name and messages are correctly formatted.

```bash
# Basic chat completion request
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Kbenkhaled/Qwen3.5-27B-NVFP4",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Explain quantum computing in simple terms."}
    ],
    "temperature": 0.7,
    "max_tokens": 512
  }'

# Expected response:
# {
#   "id": "chatcmpl-...",
#   "object": "chat.completion",
#   "created": 1234567890,
#   "model": "Kbenkhaled/Qwen3.5-27B-NVFP4",
#   "choices": [{
#     "index": 0,
#     "message": {
#       "role": "assistant",
#       "content": "Quantum computing is a type of computation that..."
#     },
#     "finish_reason": "stop"
#   }],
#   "usage": {"prompt_tokens": 25, "completion_tokens": 150, "total_tokens": 175}
# }
```

--------------------------------

### Streaming Chat Completion Request

Source: https://context7.com/aliez-ren/vllm-qwen3.5-nvfp4-sm120/llms.txt

Enable streaming for real-time token generation by setting `"stream": true` in the request payload. The response will be in Server-Sent Events format.

```bash
# Streaming chat completion
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Kbenkhaled/Qwen3.5-27B-NVFP4",
    "messages": [
      {"role": "user", "content": "Write a haiku about programming."}
    ],
    "stream": true,
    "max_tokens": 100
  }'

# Streamed response (Server-Sent Events):
# data: {"id":"chatcmpl-...","choices":[{"delta":{"role":"assistant"},"index":0}]}
# data: {"id":"chatcmpl-...","choices":[{"delta":{"content":"Code"},"index":0}]}
# data: {"id":"chatcmpl-...","choices":[{"delta":{"content":" flows"},"index":0}]}
# ...
# data: [DONE]
```

--------------------------------

### vLLM Server Health Check

Source: https://context7.com/aliez-ren/vllm-qwen3.5-nvfp4-sm120/llms.txt

A simple cURL command to check the health status of the vLLM server. It should return a JSON object with a 'healthy' status.

```bash
curl http://localhost:8000/health
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.