### Install and Serve Model with vllm-mlx

Source: https://github.com/waybarrios/vllm-mlx/blob/main/README.md

Install the vllm-mlx package and start a serving instance with continuous batching enabled. This is the quickest way to get started.

```bash
pip install vllm-mlx
vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching
```

--------------------------------

### Start vLLM-MLX Server and Configure Claude Code

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/server.md

This example shows how to start the vLLM-MLX server with specific configurations like continuous batching and auto tool choice, and then how to set environment variables to point Claude Code to the running server.

```bash
# Start the server
vllm-mlx serve mlx-community/Qwen3-Coder-Next-235B-A22B-4bit \
  --continuous-batching \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

# In another terminal, configure Claude Code
export ANTHROPIC_BASE_URL=http://localhost:8000
export ANTHROPIC_API_KEY=not-needed
claude
```

--------------------------------

### Clone Repository and Install with uv

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/getting-started/installation.md

Clone the vllm-mlx repository and install it using 'uv' for a recommended setup.

```bash
git clone https://github.com/waybarrios/vllm-mlx.git
cd vllm-mlx

uv pip install -e .
```

--------------------------------

### Initialize and Start SimpleEngine

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/python-api.md

Initialize and start the SimpleEngine for advanced use cases. Requires an await call for starting.

```python
from vllm_mlx.engine import SimpleEngine

engine = SimpleEngine("mlx-community/Llama-3.2-3B-Instruct-4bit")
await engine.start()

output = await engine.generate(
    prompt="Hello world",
    max_tokens=100
)
print(output.text)

await engine.stop()
```

--------------------------------

### Initialize and Start BatchedEngine

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/python-api.md

Initialize and start the BatchedEngine for handling multiple concurrent requests. Requires await calls for starting and stopping.

```python
from vllm_mlx.engine import BatchedEngine

engine = BatchedEngine("mlx-community/Llama-3.2-3B-Instruct-4bit")
await engine.start()

# Multiple concurrent requests
output = await engine.generate(
    prompt="Hello world",
    max_tokens=100
)

await engine.stop()
```

--------------------------------

### Test Installation with a Small Model

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/getting-started/installation.md

Run a benchmark test with a small model to confirm the installation is functional.

```bash
# Test with a small model
vllm-mlx-bench --model mlx-community/Llama-3.2-1B-Instruct-4bit --prompts 1
```

--------------------------------

### MCP Server Configuration: GitHub

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/mcp-tools.md

Example JSON configuration for starting an MCP GitHub server. Requires a GITHUB_TOKEN environment variable for authentication.

```json
{
  "mcpServers": {
    "github": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-github"],
      "env": {
        "GITHUB_TOKEN": "your-token"
      }
    }
  }
}
```

--------------------------------

### Install Optional Audio Support

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/getting-started/installation.md

Install the 'mlx-audio' package for speech-to-text and text-to-speech capabilities.

```bash
pip install mlx-audio
```

--------------------------------

### MCP Server Configuration: PostgreSQL

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/mcp-tools.md

Example JSON configuration for starting an MCP PostgreSQL server. Requires a DATABASE_URL environment variable to connect to the database.

```json
{
  "mcpServers": {
    "postgres": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-postgres"],
      "env": {
        "DATABASE_URL": "postgresql://user:pass@localhost/db"
      }
    }
  }
}
```

--------------------------------

### Install vllm-mlx CLI with uv

Source: https://github.com/waybarrios/vllm-mlx/blob/main/README.md

Installs the vllm-mlx command-line interface system-wide using uv.

```bash
uv tool install vllm-mlx
```

--------------------------------

### Enable and Start systemd Service

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/server.md

Commands to enable the vLLM-MLX systemd service to start on boot and to manually start the service.

```bash
sudo systemctl enable vllm-mlx
sudo systemctl start vllm-mlx
```

--------------------------------

### Install All Audio Dependencies at Once

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/audio.md

A convenient command to install all audio-related dependencies for vllm-mlx, including TTS requirements and language models. Ensure espeak-ng is installed for non-English TTS.

```bash
pip install vllm-mlx[audio]
python -m spacy download en_core_web_sm
brew install espeak-ng  # macOS, for non-English languages
```

--------------------------------

### Start OpenAI-Compatible Server

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/reference/cli.md

Use this command to start the API server. Specify the model path or a configuration file for multi-model serving. Options control server behavior, resource allocation, and authentication.

```bash
vllm-mlx serve <model> [options]
vllm-mlx serve --models-config <yaml> [options]
```

```bash
# Simple mode (single user, max throughput)
# Model path is used as the model name in the OpenAI API (e.g. model="mlx-community/Llama-3.2-3B-Instruct-4bit")
vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit
```

```bash
# With a custom API model name (model is accessed as "my-model" via the OpenAI API)
# --served-model-name sets the name clients must use when calling the API (e.g. model="my-model")
vllm-mlx serve --served-model-name my-model mlx-community/Llama-3.2-3B-Instruct-4bit
```

```bash
# Continuous batching (multiple users)
vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --continuous-batching
```

```bash
# With memory limit for large models
vllm-mlx serve mlx-community/GLM-4.7-Flash-4bit \
  --continuous-batching \
  --cache-memory-mb 2048
```

```bash
# Production with paged cache
vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \
  --continuous-batching \
  --use-paged-cache \
  --port 8000
```

```bash
# With MCP tools
vllm-mlx serve mlx-community/Qwen3-4B-4bit --mcp-config mcp.json
```

```bash
# Multimodal model
vllm-mlx serve mlx-community/Qwen3-VL-4B-Instruct-3bit
```

```bash
# Reasoning model (separates thinking from answer)
vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3
```

```bash
# Disable server-wide thinking by default (request-level chat_template_kwargs still override)
vllm-mlx serve mlx-community/Qwen3-8B-4bit \
  --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"enable_thinking": false}'
```

```bash
# DeepSeek reasoning model
vllm-mlx serve mlx-community/DeepSeek-R1-Distill-Qwen-7B-4bit --reasoning-parser deepseek_r1
```

```bash
# Tool calling with Mistral/Devstral
vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \
  --enable-auto-tool-choice --tool-call-parser mistral
```

```bash
# Tool calling with Granite
vllm-mlx serve mlx-community/granite-4.0-tiny-preview-4bit \
  --enable-auto-tool-choice --tool-call-parser granite
```

```bash
# With API key authentication
vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --api-key your-secret-key
```

--------------------------------

### Start Server in Simple Mode

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/server.md

Launches the server with default settings for maximum throughput for a single user.

```bash
vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000
```

--------------------------------

### Tool Calling Example (Python)

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/mcp-tools.md

A Python example demonstrating the full tool calling workflow using httpx.

```APIDOC
## Tool Calling Workflow Example

This example demonstrates how to use the MCP endpoints to enable tool calling with an LLM.

### Steps:
1. Get available tools from the MCP server.
2. Send a user request along with the available tools to the chat completions endpoint.
3. If the LLM generates tool calls, extract them.
4. Execute the tool call via the MCP execute endpoint.
5. Send the tool's result back to the LLM to generate a final response.

```python
import json
import httpx

BASE_URL = "http://localhost:8000"

# 1. Get available tools
tools_response = httpx.get(f"{BASE_URL}/v1/mcp/tools")
tools = tools_response.json()["tools"]

# 2. Send request with tools
response = httpx.post(
    f"{BASE_URL}/v1/chat/completions",
    json={
        "model": "default",
        "messages": [{"role": "user", "content": "List files in /tmp"}],
        "tools": tools,
        "max_tokens": 1024
    }
)

result = response.json()
message = result["choices"][0]["message"]

# 3. Check for tool calls
if message.get("tool_calls"):
    tool_call = message["tool_calls"][0]

    # 4. Execute tool via MCP
    exec_response = httpx.post(
        f"{BASE_URL}/v1/mcp/execute",
        json={
            "server": "filesystem",
            "tool": tool_call["function"]["name"],
            "arguments": json.loads(tool_call["function"]["arguments"])
        }
    )
    tool_result = exec_response.json()

    # 5. Send result back to LLM
    messages = [
        {"role": "user", "content": "List files in /tmp"},
        message,
        {
            "role": "tool",
            "tool_call_id": tool_call["id"],
            "content": json.dumps(tool_result["result"])
        }
    ]

    final_response = httpx.post(
        f"{BASE_URL}/v1/chat/completions",
        json={"model": "default", "messages": messages}
    )
    print(final_response.json()["choices"][0]["message"]["content"])
```
```

--------------------------------

### Production Server Setup with Security

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/reference/cli.md

Configure a production server with API key authentication, rate limiting, and timeouts.

```bash
vllm-mlx serve mlx-community/Qwen3-4B-4bit \
  --api-key your-secret-key \
  --rate-limit 60 \
  --timeout 120 \
  --continuous-batching
```

--------------------------------

### Clone Repository and Install with pip

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/getting-started/installation.md

Clone the vllm-mlx repository and install it using 'pip'.

```bash
git clone https://github.com/waybarrios/vllm-mlx.git
cd vllm-mlx

pip install -e .
```

--------------------------------

### Install vllm-mlx with audio extras

Source: https://github.com/waybarrios/vllm-mlx/blob/main/README.md

Installs vllm-mlx with additional dependencies for audio processing.

```bash
pip install vllm-mlx[audio]
```

--------------------------------

### Clone Repository and Install Dependencies

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/development/contributing.md

Clone the vllm-mlx repository and install it with development dependencies.

```bash
git clone https://github.com/waybarrios/vllm-mlx.git
cd vllm-mlx

pip install -e ".[dev"]
```

--------------------------------

### Install vllm-mlx with audio support and espeak-ng

Source: https://github.com/waybarrios/vllm-mlx/blob/main/README.md

Install the vllm-mlx library with optional audio support. Also, install espeak-ng on macOS, which is required for non-English TTS.

```bash
pip install vllm-mlx[audio]
brew install espeak-ng        # macOS, needed for non-English TTS
```

--------------------------------

### Basic Text-to-Speech CLI Examples

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/audio.md

Generate speech from text using the TTS CLI. Examples cover playing audio directly, using different voices, saving output to a file, and listing available voices.

```bash
# Simple TTS example
python examples/tts_example.py "Hello, how are you?" --play

# With different voice
python examples/tts_example.py "Hello!" --voice am_michael --play

# Save to file
python examples/tts_example.py "Welcome to the demo" -o greeting.wav

# List available voices
python examples/tts_example.py --list-voices
```

--------------------------------

### Install Core Audio Support and TTS Dependencies

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/audio.md

Install the core mlx-audio library and its dependencies for Text-to-Speech functionality. This includes downloading necessary language models for spaCy.

```bash
# Core audio support
pip install mlx-audio>=0.2.9

# Required dependencies for TTS
pip install sounddevice soundfile scipy numba tiktoken misaki spacy num2words loguru phonemizer

# Download spacy English model
python -m spacy download en_core_web_sm

# For non-English TTS (Spanish, French, etc.), install espeak-ng:
# macOS
brew install espeak-ng

# Ubuntu/Debian
# sudo apt-get install espeak-ng
```

--------------------------------

### Install Optional Vision Support

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/getting-started/installation.md

Install the necessary dependencies for video processing with transformers.

```bash
pip install -e ".[vision]"
```

--------------------------------

### Serve Qwen Model with Tool Calling

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/tool-calling.md

Start the vllm-mlx server with a Qwen model, enabling auto tool choice and specifying the 'qwen' parser for its output format.

```bash
# Qwen3
vllm-mlx serve mlx-community/Qwen3-4B-4bit \
  --enable-auto-tool-choice --tool-call-parser qwen
```

--------------------------------

### MCP Configuration File Example

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/reference/configuration.md

Define MCP server configurations in a JSON file. Specify the command, arguments, and environment variables for each MCP server.

```json
{
  "mcpServers": {
    "server-name": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-name", "arg1"],
      "env": {
        "ENV_VAR": "value"
      }
    }
  }
}
```

--------------------------------

### Pre-loading at startup

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/embeddings.md

Demonstrates how to pre-load a specific embedding model when starting the vLLM-MLX server using the `--embedding-model` flag.

```APIDOC
## Pre-loading at startup

Use `--embedding-model` to load a model at startup. When this flag is set, only that specific model can be used for embeddings:

```bash
vllm-mlx serve my-llm-model --embedding-model mlx-community/all-MiniLM-L6-v2-4bit
```

Requesting a different model will return a 400 error.
```

--------------------------------

### Start Server with Multi-Model Serving

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/server.md

Configures the server to serve a registry of models from a YAML file, allowing clients to route requests by specifying a model name.

```bash
vllm-mlx serve --models-config /etc/vllm-mlx/models.yaml --port 8000
```

--------------------------------

### Install vllm-mlx in a project with uv

Source: https://github.com/waybarrios/vllm-mlx/blob/main/README.md

Installs vllm-mlx within a specific project environment using uv.

```bash
uv pip install vllm-mlx
```

--------------------------------

### Install Optional Embeddings Support

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/getting-started/installation.md

Install the 'mlx-embeddings' package for text embedding functionalities.

```bash
pip install mlx-embeddings
```

--------------------------------

### Start Server with Chat Template Defaults

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/server.md

Sets default chat template keyword arguments for the server. Request-level values can override these defaults.

```bash
vllm-mlx serve mlx-community/Qwen3-8B-4bit \
  --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"enable_thinking": false}'
```

--------------------------------

### Start OpenAI-Compatible Server

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/getting-started/quickstart.md

Run the vllm-mlx server in simple mode for maximum throughput or with continuous batching for multiple concurrent users. Specify the model and port.

```bash
vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000
```

```bash
vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching
```

--------------------------------

### Text-to-Speech example

Source: https://github.com/waybarrios/vllm-mlx/blob/main/README.md

Generate and play speech from a given text. This example demonstrates basic TTS functionality.

```python
python examples/tts_example.py "Hello, how are you?" --play
```

--------------------------------

### Prompting for Step-by-Step Thinking

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/reasoning.md

Example of how to prompt a reasoning model to encourage step-by-step thinking by including an instruction in the system message.

```python
messages = [
    {"role": "system", "content": "Think through problems step by step before answering."},
    {"role": "user", "content": "What is 17 × 23?"}
]
```

--------------------------------

### Install mlx-audio Package

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/audio.md

Install or upgrade the mlx-audio package to version 0.2.9 or later. This is a prerequisite for using the audio functionalities.

```bash
pip install mlx-audio>=0.2.9
```

--------------------------------

### Start vllm-mlx Server with Tool Calling Enabled

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/tool-calling.md

Enable tool calling functionality by including the `--enable-auto-tool-choice` flag when starting the vllm-mlx server. Specify the `--tool-call-parser` for your model family.

```bash
vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \
  --enable-auto-tool-choice \
  --tool-call-parser mistral
```

--------------------------------

### Integrate vLLM-MLX with Open WebUI

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/server.md

Steps to start a vLLM-MLX server and connect it to Open WebUI for a graphical interface. Ensure `host.docker.internal` is accessible from the Docker container.

```bash
# 1. Start vllm-mlx server
vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000

# 2. Start Open WebUI
docker run -d -p 3000:8080 \
  -e OPENAI_API_BASE_URL=http://host.docker.internal:8000/v1 \
  -e OPENAI_API_KEY=not-needed \
  --name open-webui \
  ghcr.io/open-webui/open-webui:main

# 3. Open http://localhost:3000
```

--------------------------------

### Start vLLM-MLX server with an embedding model

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/embeddings.md

Start the vLLM-MLX server and pre-load a specific embedding model at startup using the --embedding-model flag. This model will be used for all subsequent embedding requests.

```bash
# Pre-load a specific embedding model at startup
vllm-mlx serve my-llm-model --embedding-model mlx-community/all-MiniLM-L6-v2-4bit
```

--------------------------------

### Multilingual Text-to-Speech CLI Examples

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/audio.md

Synthesize speech in various languages using the multilingual TTS script. Examples demonstrate usage for English, Spanish, French, Japanese, and Chinese, including specifying models and listing available models and languages.

```bash
# English (auto-selects best model)
python examples/tts_multilingual.py "Hello world" --play

# Spanish
python examples/tts_multilingual.py "Hola mundo" --lang es --play

# French
python examples/tts_multilingual.py "Bonjour le monde" --lang fr --play

# Japanese
python examples/tts_multilingual.py "こんにちは" --lang ja --play

# Chinese
python examples/tts_multilingual.py "你好世界" --lang zh --play

# Use specific model
python examples/tts_multilingual.py "Hello" --model chatterbox --play

# List all models
python examples/tts_multilingual.py --list-models

# List all languages
python examples/tts_multilingual.py --list-languages
```

--------------------------------

### Completions API

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/server.md

Example of making a completion request using the OpenAI-compatible API.

```python
response = client.completions.create(
    model="default",
    prompt="The capital of France is",
    max_tokens=50
)
```

--------------------------------

### Real-time Transcription CLI Examples

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/audio.md

These Python scripts enable real-time audio transcription from a microphone. Examples include closed captions, faster models for lower latency, basic recording and transcription, chunked transcription, and live transcription with voice activity detection.

```bash
# Closed captions with whisper-large-v3 (best quality)
python examples/closed_captions.py --language es --chunk 5

# Faster model for lower latency
python examples/closed_captions.py --language en --model whisper-turbo --chunk 3

# Basic mic transcription (record then transcribe)
python examples/mic_transcribe.py --language es

# Real-time chunked transcription
python examples/mic_realtime.py --language es --chunk 3

# Live transcription with voice activity detection
python examples/mic_live.py --language es
```

--------------------------------

### Create MCP Configuration File

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/mcp-tools.md

Defines the MCP server configuration, specifying the command to run and its arguments. This example configures a filesystem server.

```json
{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"]
    }
  }
}
```

--------------------------------

### Production Setup for Continuous Batching

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/continuous-batching.md

Serve a model with continuous batching and paged attention enabled for production environments. Adjust cache memory for large models.

```bash
vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \
  --continuous-batching \
  --use-paged-cache \
  --port 8000
```

--------------------------------

### Start vllm-mlx Server with MCP Config

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/mcp-tools.md

Launches the vllm-mlx server, enabling MCP by providing a configuration file. Supports both simple mode and continuous batching.

```bash
# Simple mode
vllm-mlx serve mlx-community/Qwen3-4B-4bit --mcp-config mcp.json

# Continuous batching
vllm-mlx serve mlx-community/Qwen3-4B-4bit --mcp-config mcp.json --continuous-batching
```

--------------------------------

### Tool Calling

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/server.md

Example of how to implement tool calling, where the model can invoke defined tools and receive results back.

```python
# Step 1: Send request with tools
response = client.messages.create(
    model="default",
    max_tokens=1024,
    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
    tools=[
        {
            "name": "get_weather",
            "description": "Get weather for a city",
            "input_schema": {
                "type": "object",
                "properties": {"city": {"type": "string"}},
                "required": ["city"]
            }
        }
    ]
)

# Step 2: Check if model wants to use tools
for block in response.content:
    if block.type == "tool_use":
        print(f"Tool: {block.name}, Input: {block.input}, ID: {block.id}")
        # response.stop_reason will be "tool_use"

# Step 3: Send tool result back
response = client.messages.create(
    model="default",
    max_tokens=1024,
    messages=[
        {"role": "user", "content": "What's the weather in Paris?"},
        {"role": "assistant", "content": response.content},
        {"role": "user", "content": [
            {
                "type": "tool_result",
                "tool_use_id": block.id,
                "content": "Sunny, 22C"
            }
        ]}
    ],
    tools=[
        {
            "name": "get_weather",
            "description": "Get weather for a city",
            "input_schema": {
                "type": "object",
                "properties": {"city": {"type": "string"}},
                "required": ["city"]
            }
        }
    ]
)
print(response.content[0].text)  # "The weather in Paris is sunny, 22C."
```

--------------------------------

### List Available Voices API Example

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/audio.md

Retrieve a list of available voices for a specific speech synthesis model by querying the /v1/audio/voices endpoint.

```bash
curl http://localhost:8000/v1/audio/voices?model=kokoro
```

--------------------------------

### Serve Llama Model with Tool Calling

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/tool-calling.md

Start the vllm-mlx server with a Llama model, enabling auto tool choice and specifying the 'llama' parser for its output format.

```bash
# Llama 3.2
vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit \
  --enable-auto-tool-choice --tool-call-parser llama
```

--------------------------------

### Direct usage without server

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/embeddings.md

Example of using the EmbeddingEngine directly within a Python script without starting a separate server.

```APIDOC
## Direct usage without server

```python
from vllm_mlx.embedding import EmbeddingEngine

engine = EmbeddingEngine("mlx-community/all-MiniLM-L6-v2-4bit")
engine.load()

vectors = engine.embed(["Hello world", "How are you?"])
print(f"Dimensions: {len(vectors[0])}")

tokens = engine.count_tokens(["Hello world"])
print(f"Token count: {tokens}")
```
```

--------------------------------

### Serve DeepSeek Model with Tool Calling

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/tool-calling.md

Start the vllm-mlx server with a DeepSeek model, enabling auto tool choice and specifying the 'deepseek' parser for its output format.

```bash
# DeepSeek V3
vllm-mlx serve mlx-community/DeepSeek-V3-0324-4bit \
  --enable-auto-tool-choice --tool-call-parser deepseek
```

--------------------------------

### Serve Models with Configuration

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/reference/cli.md

Use this command to serve models using a configuration file for multi-model serving.

```bash
vllm-mlx serve --models-config /etc/vllm-mlx/models.yaml --continuous-batching
```

--------------------------------

### Run LLM Benchmarks

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/benchmarks/llm.md

Execute LLM benchmarks with a specified model, number of prompts, and maximum tokens. Ensure the vllm-mlx tool is installed and accessible in your PATH.

```bash
vllm-mlx-bench --model mlx-community/Llama-3.2-1B-Instruct-4bit --prompts 5 --max-tokens 256
```

--------------------------------

### Solve Math Problem with Reasoning in Python

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/reasoning.md

This example demonstrates how to use the OpenAI client to solve a math problem, capturing both the reasoning process and the final answer.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

def solve_math(problem: str) -> dict:
    """Solve a math problem and return reasoning + answer."""
    response = client.chat.completions.create(
        model="default",
        messages=[
            {"role": "system", "content": "You are a math tutor. Show your work."},
            {"role": "user", "content": problem}
        ],
        temperature=0.2
    )

    message = response.choices[0].message
    return {
        "problem": problem,
        "work": message.reasoning,
        "answer": message.content
    }

result = solve_math("If a train travels 120 km in 2 hours, what is its average speed?")
print(f"Problem: {result['problem']}")
print(f"\nWork shown:\n{result['work']}")
print(f"\nFinal answer: {result['answer']}")
```

--------------------------------

### Install vllm-mlx with pip

Source: https://github.com/waybarrios/vllm-mlx/blob/main/README.md

Installs the vllm-mlx package using pip.

```bash
pip install vllm-mlx
```

--------------------------------

### Run Full and Quick Image Benchmarks

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/benchmarks/image.md

Execute the full benchmark suite which tests 10 resolutions, or use the quick benchmark for a faster test across 4 resolutions. Both commands require specifying the model to benchmark.

```bash
# Full benchmark (10 resolutions)
vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit

# Quick benchmark (4 resolutions)
vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --quick
```

--------------------------------

### Serve Kimi K2 Model with Tool Calling

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/tool-calling.md

Start the vllm-mlx server with a Kimi K2 model, enabling auto tool choice and specifying the 'kimi' parser for its output format.

```bash
# Kimi K2
vllm-mlx serve mlx-community/Kimi-K2-Instruct-4bit \
  --enable-auto-tool-choice --tool-call-parser kimi
```

--------------------------------

### Serve Model with MCP Tools

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/es/reference/configuration.md

Serve a model using a specific MCP configuration file, enabling auto tool choice and Qwen as the tool call parser. Continuous batching is active.

```bash
vllm-mlx serve mlx-community/Qwen3-4B-4bit \
  --mcp-config mcp.json \
  --enable-auto-tool-choice \
  --tool-call-parser qwen \
  --continuous-batching
```

--------------------------------

### Error Handling Example

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/python-api.md

Provides an example of how to handle potential errors during model loading.

```APIDOC
## Error Handling

### Description
Demonstrates how to catch exceptions that may occur during model loading.

### Method
```python
from vllm_mlx.models import MLXLanguageModel

try:
    model = MLXLanguageModel("invalid-model")
    model.load()
except Exception as e:
    print(f"Failed to load model: {e}")
```
```

--------------------------------

### Configure and Serve with Warm Prompts

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/warm-prompts.md

Write agent system prompts to a JSON file and then point the vLLM-mlx server to this file using the --warm-prompts argument. Ensure --continuous-batching is enabled.

```bash
# Write the agents you care about once
cat > ~/.config/vllm-mlx/agents.json <<'JSON'
[
  [{"role": "system", "content": "You are a code assistant..."}]
]
JSON

# Point the server at it
vllm-mlx serve mlx-community/Qwen3-4B-4bit \
  --continuous-batching \
  --warm-prompts ~/.config/vllm-mlx/agents.json
```

--------------------------------

### Install espeak-ng and spacy model

Source: https://github.com/waybarrios/vllm-mlx/blob/main/README.md

Installs the espeak-ng dependency via Homebrew and downloads the en_core_web_sm model for spacy.

```bash
brew install espeak-ng
python -m spacy download en_core_web_sm
```

--------------------------------

### Configure Multiple MCP Servers

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/mcp-tools.md

Set up multiple MCP servers, such as filesystem and GitHub, by defining their respective commands, arguments, and necessary environment tokens.

```json
{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"]
    },
    "github": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-github"],
      "env": {
        "GITHUB_TOKEN": "your-token"
      }
    }
  }
}
```

--------------------------------

### Install mlx-embeddings

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/embeddings.md

Install the mlx-embeddings package to enable embedding functionality. Ensure you are using version 0.0.5 or later.

```bash
pip install mlx-embeddings>=0.0.5
```

--------------------------------

### Start the server with an embedding model

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/embeddings.md

This command starts the vLLM-MLX server and pre-loads a specific embedding model for immediate use.

```APIDOC
## Start the server with an embedding model

```bash
vllm-mlx serve my-llm-model --embedding-model mlx-community/all-MiniLM-L6-v2-4bit
```

If you don't use `--embedding-model`, the embedding model is loaded lazily on the first request, but only from the built-in request-time allowlist.
```

--------------------------------

### Multilingual Text-to-Speech example

Source: https://github.com/waybarrios/vllm-mlx/blob/main/README.md

Generate and play speech in a specified language. This example shows how to use the TTS functionality for different languages.

```python
python examples/tts_multilingual.py "Hola mundo" --lang es --play
```

--------------------------------

### Pre-download Models using huggingface-cli

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/audio.md

Download models from HuggingFace using the CLI to avoid slow download times on first use. This is recommended for faster startup and deployment.

```bash
huggingface-cli download mlx-community/whisper-large-v3-mlx
huggingface-cli download mlx-community/Kokoro-82M-bf16
```

--------------------------------

### Serve Model with Reasoning Parser

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/server.md

For models that output their thinking process, use the `--reasoning-parser` option to separate reasoning from the final answer. This example shows how to use the 'qwen3' parser for Qwen3 models.

```bash
# Qwen3 models
vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3
```

--------------------------------

### Get Server Status

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/server.md

Use the GET /v1/status endpoint for real-time server statistics and per-request details. This is useful for debugging performance, tracking cache efficiency, and monitoring Metal GPU memory.

```bash
GET /v1/status
```

```bash
curl -s http://localhost:8000/v1/status | python -m json.tool
```

--------------------------------

### Serve IBM Granite Model with Tool Calling

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/tool-calling.md

Start the vllm-mlx server with an IBM Granite model, enabling auto tool choice and specifying the 'granite' parser for its output format.

```bash
# Granite 4.0
vllm-mlx serve mlx-community/granite-4.0-tiny-preview-4bit \
  --enable-auto-tool-choice --tool-call-parser granite
```

--------------------------------

### Speech Synthesis API Example

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/audio.md

Generate speech from text using the TTS API compatible endpoint. This example shows how to send a JSON payload with text, model, voice, and other parameters. The output can be saved to a file.

```bash
curl http://localhost:8000/v1/audio/speech \
  -d '{"model": "kokoro", "input": "Hello world", "voice": "af_heart"}' \
  -H "Content-Type: application/json" \
  --output speech.wav
```

--------------------------------

### Voice Separation CLI Example

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/audio.md

Demonstrates command-line usage for the audio separation feature. Options include playing the audio directly, specifying the description for separation, and setting output file names.

```bash
python examples/audio_separation_example.py meeting.mp3 --play
python examples/audio_separation_example.py song.mp3 --description music -o music.wav
```

--------------------------------

### Drums Separation CLI Example

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/audio.md

Command-line example for isolating drums from a song using SAM-Audio. Specify the input audio file, the description for separation (e.g., 'drums'), and output file names for the isolated drums and the background track.

```bash
# Isolate drums from rock song
python examples/audio_separation_example.py examples/rock_get_ready.mp3 \
  --description "drums" \
  --output drums_isolated.wav \
  --background rock_no_drums.wav
```

--------------------------------

### Benchmark vLLM-MLX with Warm Prompts

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/warm-prompts.md

Benchmark your setup to measure the impact of warm prompts. This involves running the server twice, once without warm prompts (cold) and once with warm prompts, then comparing the results from `bench-serve`.

```bash
# Cold: no warm-prompts
vllm-mlx serve MODEL --continuous-batching & 
vllm-mlx bench-serve --prompts long --concurrency 1,4 \
  --system-prompt-file my-system.txt --tag cold \
  --output cold.csv --format csv

# Warm: same server config + --warm-prompts
vllm-mlx serve MODEL --continuous-batching \
  --warm-prompts ~/.config/vllm-mlx/agents.json & 
vllm-mlx bench-serve --prompts long --concurrency 1,4 \
  --system-prompt-file my-system.txt --tag warm \
  --output warm.csv --format csv
```

--------------------------------

### Run Video Benchmarks with vLLM-MLX

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/benchmarks/video.md

Use the `vllm-mlx-bench` command with the `--video` flag to run benchmarks. Specify the model and optionally `--quick` for a shorter run or `--video-url` for a custom video.

```bash
vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video
```

```bash
vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video --quick
```

```bash
vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video --video-url https://example.com/video.mp4
```

--------------------------------

### Embeddings API

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/server.md

Example of generating embeddings for a given input text.

```python
response = client.embeddings.create(
    model="mlx-community/multilingual-e5-small-mlx",
    input="Hello world"
)
print(response.data[0].embedding[:5])  # First 5 dimensions
```

--------------------------------

### Serve Model with Reasoning Parser

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/es/reference/configuration.md

Serve a model specifically configured for reasoning tasks using the Qwen3 reasoning parser. Continuous batching is enabled.

```bash
vllm-mlx serve mlx-community/Qwen3-8B-4bit \
  --reasoning-parser qwen3 \
  --continuous-batching
```

--------------------------------

### GET /v1/audio/voices

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/audio.md

List available voices for a specified audio model.

```APIDOC
## GET /v1/audio/voices

### Description
List available voices for a model.

### Method
GET

### Endpoint
/v1/audio/voices

### Parameters
#### Query Parameters
- **model** (string) - Required - The model name or alias to list voices for.

### Request Example
```bash
curl http://localhost:8000/v1/audio/voices?model=kokoro
```

### Response
#### Success Response (200)
- **voices** (array) - A list of available voice objects, each with a `voice_id` and `name`.
```

--------------------------------

### Serve Model from Local Path

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/reference/models.md

Use this command to serve a model that has been downloaded and stored locally. Replace '/path/to/local/model' with the actual directory path.

```bash
vllm-mlx serve /path/to/local/model
```

--------------------------------

### Enable Metrics Endpoint

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/server.md

Starts the server with the Prometheus metrics endpoint enabled. This endpoint is intentionally unauthenticated.

```bash
vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit \
  --enable-metrics
```

--------------------------------

### Serve DeepSeek-R1 Model with Reasoning Parser

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/reasoning.md

Use this command to start the vLLM-MLX server with the DeepSeek-R1 reasoning parser. This parser is more lenient and can handle cases where the opening `<think>` tag might be omitted.

```bash
vllm-mlx serve mlx-community/DeepSeek-R1-Distill-Qwen-7B-4bit --reasoning-parser deepseek_r1
```

--------------------------------

### Start Server with Paged Cache

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/server.md

Activates paged caching for memory-efficient operation, suitable for production environments.

```bash
vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching --use-paged-cache
```

--------------------------------

### Start vllm-mlx Server with Model Registry

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/model-registry.md

Launches the vllm-mlx server using a specified models configuration file. This command enables multi-model serving. Global serve flags can still be applied.

```bash
vllm-mlx serve --models-config /etc/vllm-mlx/models.yaml --host 0.0.0.0 --port 8000
```

--------------------------------

### Chat Completions API - Streaming

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/server.md

Example of making a streaming chat completion request and processing the chunks.

```python
# Streaming
stream = client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": "Tell me a story"}],
    stream=True
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
```

--------------------------------

### Serve NVIDIA Nemotron Model with Tool Calling

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/tool-calling.md

Start the vllm-mlx server with an NVIDIA Nemotron model, enabling auto tool choice and specifying the 'nemotron' parser for its output format.

```bash
# Nemotron 3 Nano
vllm-mlx serve mlx-community/NVIDIA-Nemotron-3-Nano-30B-A3B-MLX-6Bit \
  --enable-auto-tool-choice --tool-call-parser nemotron
```

--------------------------------

### Chat Completions API - Non-streaming

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/server.md

Example of making a non-streaming chat completion request to the OpenAI-compatible API.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

# Non-streaming
response = client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": "Hello!"}],
    max_tokens=100
)
```

--------------------------------

### Start Server in Continuous Batching Mode

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/server.md

Enables continuous batching for handling multiple concurrent users efficiently.

```bash
vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching
```

--------------------------------

### Run Server Product-Style Workload Benchmark

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/benchmarks/README.md

Benchmark a running server using a product-style workload with quality checks and Prometheus metric deltas.

```bash
vllm-mlx bench-serve --url http://localhost:8000 
  --workload ./workload.json --output workload-results.json
```

--------------------------------

### Benchmark served model with prompts

Source: https://github.com/waybarrios/vllm-mlx/blob/main/README.md

Benchmark a served model using a list of prompts from a file. Results are saved to a CSV file.

```bash
vllm-mlx bench-serve --url http://localhost:8000 --concurrency 5 --prompts prompts.txt --output results.csv
```

--------------------------------

### Transcription Quality Example Output

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/benchmarks/audio.md

Compares the input text with the output from the Whisper-large-v3 model to demonstrate transcription accuracy.

```text
Input text:
"Welcome to this comprehensive speech to text demonstration.
This audio sample is designed to test the accuracy and speed of various speech recognition models.
The quick brown fox jumps over the lazy dog..."

Whisper-large-v3 output:
"Welcome to this comprehensive speech to text demonstration.
This audio sample is designed to test the accuracy and speed of various speech recognition models.
The quick brown fox jumps over the lazy dog..." (identical)
```

--------------------------------

### SimpleEngine Usage

Source: https://github.com/waybarrios/vllm-mlx/blob/main/docs/guides/python-api.md

Demonstrates the direct use of the SimpleEngine for advanced programmatic control.

```APIDOC
## SimpleEngine Usage

### Description
Provides direct access to the engine for advanced use cases, including starting and stopping the engine, and generating text.

### Method
```python
from vllm_mlx.engine import SimpleEngine

engine = SimpleEngine("mlx-community/Llama-3.2-3B-Instruct-4bit")
await engine.start()

output = await engine.generate(
    prompt="Hello world",
    max_tokens=100
)
print(output.text)

await engine.stop()
```
```