MLX Omni Server

https://github.com/zhutao100/mlx-omni-server
Admin
MLX Omni Server is a high-performance local inference server built on Apple's MLX framework,...
Tokens:51,847
Snippets:563
Trust Score:4.7
Update:2 months ago
Show doc for...
Tokens:
Raw
### Setup environment with pip

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/docs/development_guide.md

Simple editable installation for the project.

```bash
python3 -m pip install -e ".[all]"
```

--------------------------------

### Install and Run MLX Omni Server

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/INDEX.md

Commands to clone the repository, install dependencies, start the server, and verify connectivity.

```bash
# Install
git clone https://github.com/zhutao100/mlx-omni-server.git
cd mlx-omni-server
pip install -e ".[all]"  # Or core only: pip install -e .

# Run
mlx-omni-server --host 127.0.0.1 --port 10240

# Test
curl http://localhost:10240/v1/models
```

--------------------------------

### Get pip installation instructions

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_optional_features.md

Retrieve the command string required to install the dependencies for a specific feature.

```python
from mlx_omni_server.optional_features import install_instructions

cmd = install_instructions("images")
print(cmd)
# Output: `pip install '.[images]'`
```

--------------------------------

### Start the Server

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/README.md

Launch the server on a specific host and port.

```bash
mlx-omni-server --host 127.0.0.1 --port 10240
```

--------------------------------

### Setup environment with uv

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/docs/development_guide.md

Recommended approach for reproducible development environments.

```bash
uv sync --all-extras --dev
uv run mlx-omni-server --help
```

--------------------------------

### Install all optional features

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_optional_features.md

Installs all available optional modules including images, stt, tts, and weave.

```bash
pip install -e ".[all]"
```

--------------------------------

### Install mlx-omni-server

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/README.md

Clone the repository and install the package with all optional features.

```bash
# This repo is not published to PyPI (the PyPI name `mlx-omni-server` points to a different project).
git clone https://github.com/zhutao100/mlx-omni-server.git
cd mlx-omni-server

# Install core + all optional features (images/stt/tts)
python3 -m pip install -e ".[all]"

# Or install a subset (examples)
# python3 -m pip install -e ".[images]"
# python3 -m pip install -e ".[stt]"
# python3 -m pip install -e ".[tts]"
```

--------------------------------

### Install Optional Features

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/configuration.md

Commands to install specific dependencies for image, speech-to-text, or text-to-speech capabilities.

```bash
# Install all extras (images, STT, TTS)
pip install -e ".[all]"

# Or selectively
pip install -e ".[images]"      # Image generation (mflux)
pip install -e ".[stt]"         # Speech-to-text (mlx-whisper)
pip install -e ".[tts]"         # Text-to-speech (f5-tts-mlx + mlx-audio)
```

--------------------------------

### Install for production deployment

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_optional_features.md

Installs the package for production use, either with minimal core dependencies or all extras.

```bash
# Install minimal core
pip install mlx-omni-server

# Or install with all extras
pip install mlx-omni-server[all]
```

--------------------------------

### Install Optional Extra for TTS

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_optional_features.md

Command to install the text-to-speech dependencies.

```bash
pip install -e ".[tts]"
```

--------------------------------

### Install Optional Extra for STT

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_optional_features.md

Command to install the speech-to-text dependencies.

```bash
pip install -e ".[stt]"
```

--------------------------------

### Install Optional Extra for Images

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_optional_features.md

Command to install the image generation dependencies.

```bash
pip install -e ".[images]"
```

--------------------------------

### Instantiate Tool Call

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_schema_utils.md

Example usage of the ToolCall model.

```python
ToolCall(
    index=0,
    id="call_abc123",
    type="function",
    function=FunctionCall(
        name="get_weather",
        arguments='{"location": "NYC", "unit": "F"}'
    )
)
```

--------------------------------

### Install selective optional features

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_optional_features.md

Installs specific subsets of optional dependencies as needed.

```bash
# Just images
pip install -e ".[images]"

# Images + STT
pip install -e ".[images,stt]"

# All except weave
pip install -e ".[images,stt,tts]"
```

--------------------------------

### Handle missing feature errors

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_optional_features.md

Example of a request and the corresponding 501 error response when a feature is not installed.

```bash
curl -X POST http://localhost:10240/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{"prompt": "a dog"}'
```

```json
{
    "detail": "Image generation support is not installed. Install optional dependencies with `pip install '.[images]'`. Missing: mflux"
}
```

--------------------------------

### Configure Server via Command-Line Arguments

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/configuration.md

Examples of setting network bindings, logging levels, and file logging paths.

```bash
# Bind to localhost only, custom port, debug logging
mlx-omni-server --host 127.0.0.1 --port 8000 --log-level debug

# With file logging
mlx-omni-server --log-file --log-dir /var/log/mlx-server --log-file-format jsonl
```

--------------------------------

### Start MLX Omni Server

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/configuration.md

Basic command structure for launching the server.

```bash
mlx-omni-server [OPTIONS]
```

--------------------------------

### install_instructions(extra: str)

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_optional_features.md

Retrieves the pip install command for a specific feature.

```APIDOC
## install_instructions(extra: str)

### Description
Returns the pip install command string for the specified feature.

### Parameters
- **extra** (str) - Required - Feature name.

### Returns
- **str** - Command string (e.g., "pip install '.[images]'").
```

--------------------------------

### Initialize ResponseStreamAdapter

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_responses.md

Constructor signature and usage example for initializing the adapter.

```python
adapter = ResponseStreamAdapter(
    response_id: str,
    model: str,
    request_echo: Dict[str, Any],
)
```

```python
adapter = ResponseStreamAdapter(
    response_id="resp_abc123",
    model="mlx-community/qwen-2.5-7b",
    request_echo={"model": "mlx-community/qwen-2.5-7b", "temperature": 0.7},
)
```

--------------------------------

### Instantiate Tool

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_schema_utils.md

Example usage of the Tool model to define a weather function.

```python
Tool(
    type="function",
    function=Function(
        name="get_weather",
        description="Get weather for location",
        parameters=FunctionParameters(
            type="object",
            properties={
                "location": {"type": "string"},
                "unit": {"type": "string", "enum": ["C", "F"]}
            },
            required=["location"]
        )
    )
)
```

--------------------------------

### Run server with Uvicorn

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/docs/development_guide.md

Start the server with auto-reload enabled for development.

```bash
uvicorn mlx_omni_server.main:app --reload --host 127.0.0.1 --port 10240
```

--------------------------------

### Generate Embeddings Example

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/endpoints.md

Usage example for generating embeddings using the client library.

```python
response = client.embeddings.create(
    model="mlx-community/gte-small-mlx",
    input="The quick brown fox jumps over the lazy dog",
)
vector = response.data[0].embedding
print(f"Embedding dimension: {len(vector)}")
```

--------------------------------

### Example: Load and use a model

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_models.md

Demonstrates loading a model from Hugging Face or a local path and accessing its context window size.

```python
from mlx_omni_server.chat.models.models import load_model

# From Hugging Face
model = load_model("mlx-community/gemma-3-1b-it-4bit-DWQ")

# From local path
model = load_model("/path/to/local/model")

print(f"Model context window: {model.max_tokens}")
```

--------------------------------

### Configure Environment Variables

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/configuration.md

Setting Hugging Face cache paths, authentication tokens, and encryption keys before starting the server.

```bash
# Set custom Hugging Face cache
export HF_HOME=/mnt/models/hf_cache
export HF_TOKEN=hf_xxxxxxxxxxxxx

# Set reasoning encryption key (stable across restarts)
export MLX_OMNI_SERVER_REASONING_HMAC_KEY=my-secret-key-32-chars-or-more

mlx-omni-server
```

--------------------------------

### Example: Generate text completion

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_models.md

Demonstrates generating text using a loaded model.

```python
model = load_model("mlx-community/gemma-3-1b-it-4bit-DWQ")
output = model.generate(
    prompt="Hello, how are you?",
    max_tokens=50,
    temperature=0.7
)
print(output)
```

--------------------------------

### ChatCompletionChunk JSON Example

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_schema_utils.md

Example JSON payload for a single streaming chunk.

```json
{
    "id": "chatcmpl-abc123",
    "object": "chat.completion.chunk",
    "created": 1234567890,
    "model": "qwen-2.5-7b",
    "choices": [
        {
            "index": 0,
            "delta": {
                "role": "assistant",
                "content": "Hello"
            },
            "finish_reason": null
        }
    ]
}
```

--------------------------------

### Generate Image Example

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/endpoints.md

Demonstrates how to call the image generation client with specific parameters and access the base64 encoded result.

```python
response = client.images.generate(
    model="filipstrand/Z-Image-Turbo-mflux-4bit",
    prompt="A serene landscape with mountains and lake",
    n=1,
    size="1024x1024",
    response_format="b64_json",
)
image_b64 = response.data[0].b64_json
```

--------------------------------

### Instantiate OptionalExtra

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_optional_features.md

Example of creating an OptionalExtra instance for the image generation feature.

```python
OptionalExtra(
    name="images",
    feature_label="Image generation",
    required_modules={
        "mflux": "mflux",
    },
)
```

--------------------------------

### Transcribe Audio File

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/endpoints.md

Example usage of the client to transcribe a local audio file.

```python
with open("speech.wav", "rb") as f:
    response = client.audio.transcriptions.create(
        model="mlx-whisper",
        file=f,
        language="en",
        response_format="json",
    )
    print(response.text)
```

--------------------------------

### Check feature availability

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_optional_features.md

Verify if an optional feature is installed and its dependencies are satisfied.

```python
from mlx_omni_server.optional_features import is_available

if is_available("images"):
    print("Image generation available")
else:
    print("Image generation not installed")

if is_available("tts"):
    print("TTS available")
```

--------------------------------

### is_available(extra: str)

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_optional_features.md

Checks if an optional feature is installed and available for use.

```APIDOC
## is_available(extra: str)

### Description
Checks if an optional feature is installed and all its dependencies are available.

### Parameters
- **extra** (str) - Required - Feature name.

### Returns
- **bool** - True if all dependencies are available, False otherwise.
```

--------------------------------

### Verify module availability

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_optional_features.md

Checks if a specific module is installed and importable at runtime.

```python
import importlib.util
spec = importlib.util.find_spec("mflux")
is_installed = spec is not None
```

--------------------------------

### GET /v1/models

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/endpoints.md

Lists all available models currently accessible by the server.

```APIDOC
## GET /v1/models

### Description
List all available models.

### Method
GET

### Endpoint
/v1/models/

### Parameters
#### Query Parameters
- **include_details** (bool) - Optional - Whether to include detailed model information

### Response
#### Success Response (200)
- **object** (str) - "list"
- **data** (List[Model]) - List of available models
```

--------------------------------

### Instantiate EmbeddingRequest

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_embeddings.md

Example of creating an EmbeddingRequest instance with specific parameters.

```python
request = EmbeddingRequest(
    model="mlx-community/gte-small-mlx",
    input=["text1", "text2"],
    encoding_format="float",
    user="user123",
    # Additional custom fields are preserved
)
```

--------------------------------

### Generate embeddings example

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_embeddings.md

Demonstrates generating embeddings for both single and multiple input strings.

```python
from mlx_omni_server.embeddings.embeddings_service import EmbeddingsService
from mlx_omni_server.embeddings.schema import EmbeddingRequest

service = EmbeddingsService()

# Single text
request = EmbeddingRequest(
    model="mlx-community/gte-small-mlx",
    input="The quick brown fox jumps over the lazy dog"
)
response = await service.generate_embeddings(request)
vector = response.data[0].embedding
print(f"Embedding dimension: {len(vector)}")
print(f"First 5 values: {vector[:5]}")
print(f"Tokens used: {response.usage.prompt_tokens}")

# Multiple texts
request = EmbeddingRequest(
    model="mlx-community/gte-small-mlx",
    input=[
        "The quick brown fox",
        "Jumps over the lazy dog",
        "In the forest"
    ]
)
response = await service.generate_embeddings(request)
for i, data in enumerate(response.data):
    print(f"Text {i}: {len(data.embedding)} dimensions")
```

--------------------------------

### Resolve ModelNotFound Errors

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/configuration.md

Set environment variables for Hugging Face home directory and authentication token before starting the server.

```bash
export HF_HOME=/path/with/space
export HF_TOKEN=hf_xxxxx  # If private model
mlx-omni-server
```

--------------------------------

### Chat Completion Usage Example

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/endpoints.md

Demonstrates how to perform non-streaming and streaming chat completions using the OpenAI Python client.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:10240/v1", api_key="not-needed")

# Non-streaming
response = client.chat.completions.create(
    model="mlx-community/gemma-3-1b-it-4bit-DWQ",
    messages=[{"role": "user", "content": "Hello!"}],
    temperature=0.7,
    max_tokens=100,
)
print(response.choices[0].message.content)

# Streaming
stream = client.chat.completions.create(
    model="mlx-community/gemma-3-1b-it-4bit-DWQ",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
```

--------------------------------

### Optional Features Management

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/EXPORTS.md

Utilities for checking and installing optional feature modules.

```python
# src/mlx_omni_server/optional_features.py

@dataclass(frozen=True)
class OptionalExtra:
    name: str
    feature_label: str
    required_modules: Mapping[str, str]

def get_optional_extra(name: str) -> OptionalExtra
def install_instructions(extra: str) -> str
def missing_packages(extra: str) -> Tuple[str, ...]
def is_available(extra: str) -> bool
def not_installed_detail(extra: str, *, missing: Optional[Tuple[str, ...]] = None) -> str
def ensure_extra_available(extra: str) -> None  # Raises HTTPException(501)
```

--------------------------------

### Streaming Generation Example

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_chat_service.md

Consume a streaming chat completion response and process chunks.

```python
stream, is_replay = await chat_generation_service.stream_chat_completion(
    request,
    is_disconnected=lambda: False
)

print(f"Replay: {is_replay}")
async for item in stream:
    if item.kind == "chunk":
        delta = item.data.choices[0].delta
        if delta.content:
            print(delta.content, end="")
    elif item.kind == "error":
        print(f"Error: {item.data}")
    elif item.kind == "done":
        print("\n[DONE]")
```

--------------------------------

### ResponseResponse JSON Example

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_schema_utils.md

A sample JSON payload representing a completed response.

```json
{
    "id": "resp_abc123xyz789",
    "object": "response",
    "created_at": 1234567890,
    "model": "qwen-2.5-7b",
    "status": "completed",
    "output": [
        {
            "type": "message",
            "id": "msg_123",
            "role": "assistant",
            "content": [
                {"type": "output_text", "text": "Hello!"}
            ],
            "status": "completed"
        }
    ],
    "usage": {
        "input_tokens": 10,
        "output_tokens": 5,
        "total_tokens": 15,
        "input_tokens_details": {"cached_tokens": 0},
        "output_tokens_details": {"reasoning_tokens": 0}
    }
}
```

--------------------------------

### Triggering Cache Cleanup Task

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_chat_service.md

Example of how to initiate the background cache cleanup task using asyncio.

```python
from mlx_omni_server.chat.generation_service import background_cache_cleanup
import asyncio

# Start cleanup task
cleanup_task = asyncio.create_task(background_cache_cleanup())
```

--------------------------------

### Debug Logging Output

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/configuration.md

Example of the console output when the log level is set to debug.

```text
[DEBUG] Received chat completion request: {...}
[DEBUG] Cache miss, generating...
[INFO] Generated 150 completion tokens in 2.3s
```

--------------------------------

### Configure Environment Variables for Model Access

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/INDEX.md

Set the Hugging Face cache path and authentication token before starting the server to resolve model not found errors.

```bash
export HF_HOME=/path/to/cache
export HF_TOKEN=hf_xxxxx  # For private models
mlx-omni-server
```

--------------------------------

### Create and Poll Background Response

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/endpoints.md

Example of initiating a background task and polling for completion status.

```python
# Background response
response = client.beta.responses.create(
    model="mlx-community/qwen-2.5-7b",
    input=[{"role": "user", "content": "Write a story"}],
    background=True,
)
response_id = response.id

# Poll for completion
import time
while True:
    r = client.beta.responses.retrieve(response_id)
    if r.status == "completed":
        print(r.output[0].text)
        break
    time.sleep(1)

# Stream response events
stream = client.beta.responses.retrieve(response_id, stream=True)
for event in stream:
    print(event)
```

--------------------------------

### POST /v1/audio/speech

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_optional_features.md

Synthesizes speech from text using f5-tts-mlx and mlx-audio. Requires the 'tts' optional extra to be installed.

```APIDOC
## POST /v1/audio/speech

### Description
Synthesizes speech from text using f5-tts-mlx and mlx-audio.

### Method
POST

### Endpoint
/v1/audio/speech
```

--------------------------------

### Check Feature Availability in Endpoints

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_optional_features.md

Manual check for feature availability to return a 501 error if not installed.

```python
from fastapi import APIRouter, HTTPException
from mlx_omni_server.optional_features import is_available

router = APIRouter()

@router.post("/images/generations")
async def create_image(request):
    if not is_available("images"):
        raise HTTPException(
            status_code=501,
            detail="Image generation not installed"
        )
    # ... proceed
```

--------------------------------

### Generate Request Hash Example

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_chat_service.md

Create a SHA-256 hash for a chat completion request to facilitate caching.

```python
request = ChatCompletionRequest(
    model="test",
    messages=[{"role": "user", "content": "Hi"}]
)
hash_key = make_request_hash(request)
# Later: look up response by hash_key in cache
```

--------------------------------

### Configure Server Options

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/README.md

Run the server with custom configuration flags.

```bash
# Custom port
mlx-omni-server --port 8000

# Specific host and port
mlx-omni-server --host 127.0.0.1 --port 8000

# View all options
mlx-omni-server --help
```

--------------------------------

### Initialize local configuration

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/docs/development_guide.md

Create the local-resources.yaml file from the provided template.

```bash
cp config/local-resources.example.yaml config/local-resources.yaml
```

--------------------------------

### List available models

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_models.md

Demonstrates listing models with and without detailed configuration.

```python
from mlx_omni_server.chat.models.models_service import ModelsService

service = ModelsService()
models = service.list_models()
for model in models.data:
    print(f"  {model.id} (created: {model.created})")

# With details
models_with_details = service.list_models(include_details=True)
for model in models_with_details.data:
    print(f"  {model.id}: {model.details}")
```

--------------------------------

### Example Embedding Response

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_embeddings.md

JSON representation of a successful embedding response.

```json
{
    "object": "list",
    "data": [
        {
            "object": "embedding",
            "embedding": [0.123, 0.456, -0.789, ...],
            "index": 0
        },
        {
            "object": "embedding",
            "embedding": [0.111, 0.222, 0.333, ...],
            "index": 1
        }
    ],
    "model": "mlx-community/gte-small-mlx",
    "usage": {
        "prompt_tokens": 24,
        "total_tokens": 24
    }
}
```

--------------------------------

### Instantiate Request with Custom Parameters

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_schema_utils.md

Demonstrates how to pass vendor-specific parameters alongside standard fields in a request.

```python
# Vendor-specific params are preserved
request = ChatCompletionRequest(
    model="qwen-2.5",
    messages=[...],
    enable_thinking=True,           # Custom MLX param
    top_k=40,
    adapter_path="/path/to/lora",
    # ... more custom params
)
extras = request.get_extra_params()  # If method exists
```

--------------------------------

### Configure Model Download and Paths

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_models.md

Shows how to set the Hugging Face cache directory or specify a local path for model loading.

```bash
# Ensure HF_HOME has cached models
export HF_HOME=/path/to/model/cache

# Or use local model
model = service.get_model("/local/path/to/model")
```

--------------------------------

### GET /v1/responses/{response_id}

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/docs/apis/responses.md

Retrieve the status or details of a specific response.

```APIDOC
## GET /v1/responses/{response_id}

### Description
Retrieve the latest status of a response. When used with `stream=true`, it provides SSE lifecycle events.

### Method
GET

### Endpoint
/v1/responses/{response_id}

### Parameters
#### Path Parameters
- **response_id** (string) - Required - The unique identifier of the response.

#### Query Parameters
- **stream** (boolean) - Optional - If true, returns SSE lifecycle events.
```

--------------------------------

### GET /responses/{response_id}

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/EXPORTS.md

Retrieves the details of a specific response by its ID.

```APIDOC
## GET /responses/{response_id}

### Description
Retrieves the details of a specific response by its ID. This endpoint is also available at /v1/responses/{response_id}.

### Method
GET

### Endpoint
/responses/{response_id}

### Parameters
#### Path Parameters
- **response_id** (string) - Required - The unique identifier of the response.
```

--------------------------------

### Initialize OpenAI-Compatible Client

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/README.md

Connect to the local MLX Omni Server using the standard OpenAI Python client.

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:10240/v1",
    api_key="not-needed"
)
```

--------------------------------

### Import Main Entry Point

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/INDEX.md

Imports the FastAPI application instance and the CLI entry point.

```python
from mlx_omni_server.main import app, start

# app: FastAPI instance
# start(): CLI entry point
```

--------------------------------

### Run full test suite

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/docs/development_guide.md

Execute all tests, including integration tests.

```bash
python3 -m pytest -q
```

--------------------------------

### Interact with mlx-omni-server using OpenAI client

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/README.md

Demonstrates connecting to a local server and performing various operations including chat completions, streaming, embedding generation, and model listing.

```python
from openai import OpenAI

# Connect to local server
client = OpenAI(
    base_url="http://localhost:10240/v1",
    api_key="not-needed"
)

# Chat with caching
response = client.chat.completions.create(
    model="mlx-community/gemma-3-1b-it-4bit-DWQ",
    messages=[{"role": "user", "content": "What is 2+2?"}],
    temperature=0.7,
    seed=42,  # For caching
)
print(response.choices[0].message.content)

# Stream response
stream = client.chat.completions.create(
    model="mlx-community/qwen-2.5-7b",
    messages=[{"role": "user", "content": "Write a haiku"}],
    stream=True,
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

# Generate embeddings
embeddings = client.embeddings.create(
    model="mlx-community/gte-small-mlx",
    input=["Hello world", "Goodbye world"]
)
for data in embeddings.data:
    print(f"Embedding dimension: {len(data.embedding)}")

# Use responses API (background)
response = client.beta.responses.create(
    model="mlx-community/qwen-2.5-7b",
    input=[{"role": "user", "content": "Tell a story..."}],
    background=True,
)
print(f"Response ID: {response.id}")

# List models
models = client.models.list()
for model in models.data:
    print(f"  {model.id}")
```

--------------------------------

### Configure Multi-Instance Load Balancing

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/configuration.md

Run multiple server instances on different ports and configure an upstream load balancer to distribute traffic.

```bash
# Instance 1
mlx-omni-server --host 0.0.0.0 --port 10240

# Instance 2 (another machine or port)
mlx-omni-server --host 0.0.0.0 --port 10241

# Load balancer (nginx, haproxy, etc.)
upstream mlx_servers {
    server localhost:10240;
    server localhost:10241;
}
```

--------------------------------

### GET /v1/models/{model_id}

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/endpoints.md

Retrieves detailed information about a specific model by its ID.

```APIDOC
## GET /v1/models/{model_id}

### Description
Get information about a specific model.

### Method
GET

### Endpoint
/v1/models/{model_id}

### Parameters
#### Path Parameters
- **model_id** (str) - Required - The unique identifier of the model

#### Query Parameters
- **include_details** (bool) - Optional - Whether to include detailed model information

### Response
#### Success Response (200)
- **id** (str) - Model ID
- **object** (str) - "model"
- **created** (int) - Creation timestamp
- **owned_by** (str) - Owner of the model
- **details** (Dict[str, Any]) - Optional - Additional model details
```

--------------------------------

### GET /v1/responses/{response_id}/input_items

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/docs/apis/responses.md

Inspect resolved input items for a response.

```APIDOC
## GET /v1/responses/{response_id}/input_items

### Description
Inspects the resolved input items associated with the response.

### Method
GET

### Endpoint
/v1/responses/{response_id}/input_items

### Response
#### Success Response (200)
- **object** (string) - The object type.
- **first_id** (string) - ID of the first item.
- **last_id** (string) - ID of the last item.
- **has_more** (boolean) - Indicates if there are more items available.
```

--------------------------------

### Initialize MLX Gates for Multiple Event Loops

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_runtime.md

Demonstrates how to create separate MLX gates for distinct event loops during testing scenarios.

```python
# Test 1
loop1 = asyncio.new_event_loop()
gate1 = get_mlx_gate()  # Creates gate for loop1

# Test 2
loop2 = asyncio.new_event_loop()
gate2 = get_mlx_gate()  # Creates separate gate for loop2
```

--------------------------------

### Initialize EmbeddingsService

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_embeddings.md

Instantiate the service using default settings.

```python
service = EmbeddingsService()
```

--------------------------------

### Handling Embedding API Errors

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_embeddings.md

Examples of catching common exceptions during embedding generation.

```python
try:
    response = await service.generate_embeddings(
        EmbeddingRequest(
            model="non-existent-model",
            input="text"
        )
    )
except ValueError as e:
    print(f"Error: {e}")  # Model not found
```

```python
# Model has max_tokens constraint (e.g., 512)
try:
    response = await service.generate_embeddings(
        EmbeddingRequest(
            model="mlx-community/gte-small-mlx",
            input="x" * 10000  # Too long
        )
    )
except ValueError as e:
    print(f"Error: {e}")  # Input exceeds max length
```

```python
try:
    response = await service.generate_embeddings(
        EmbeddingRequest(
            model="mlx-community/gte-small-mlx",
            input="text",
            encoding_format="invalid"  # Only "float" or "base64"
        )
    )
except ValueError as e:
    print(f"Error: {e}")
```

--------------------------------

### ResponseStreamAdapter Initialization

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_responses.md

Initializes a new instance of the ResponseStreamAdapter.

```APIDOC
## ResponseStreamAdapter.__init__

### Description
Initializes the adapter with response metadata and the original request echo.

### Parameters
- **response_id** (str) - Required - Response ID
- **model** (str) - Required - Model ID
- **request_echo** (dict) - Required - Original request (echoed in response)

### Example
```python
adapter = ResponseStreamAdapter(
    response_id="resp_abc123",
    model="mlx-community/qwen-2.5-7b",
    request_echo={"model": "mlx-community/qwen-2.5-7b", "temperature": 0.7},
)
```
```

--------------------------------

### Conditionally Initialize Services

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_optional_features.md

Pattern for initializing services only if the corresponding optional feature is available.

```python
from mlx_omni_server.optional_features import is_available

# At startup
if is_available("images"):
    from .images.images_service import ImagesService
    images_service = ImagesService()
else:
    images_service = None

# Later
if images_service:
    # Use images service
    pass
```

--------------------------------

### Import ChatGenerationService

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_chat_service.md

Import the singleton instance of the chat generation service.

```python
from mlx_omni_server.chat.generation_service import chat_generation_service
```

--------------------------------

### Non-Streaming Generation Example

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_chat_service.md

Execute a non-streaming chat completion request and handle the result.

```python
from mlx_omni_server.chat.generation_service import chat_generation_service
from mlx_omni_server.chat.schema import ChatCompletionRequest, ChatMessage, Role

request = ChatCompletionRequest(
    model="mlx-community/gemma-3-1b-it-4bit-DWQ",
    messages=[ChatMessage(role=Role.USER, content="Hello!")]
)

result = await chat_generation_service.generate_non_stream(
    request,
    is_disconnected=lambda: False
)

if not result.is_error:
    print(f"Output: {result.payload.choices[0].message.content}")
    print(f"From cache: {result.from_cache}")
else:
    print(f"Error: {result.payload}")
```

--------------------------------

### GET /responses/{response_id}/input_items

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/EXPORTS.md

Retrieves the input items associated with a specific response.

```APIDOC
## GET /responses/{response_id}/input_items

### Description
Retrieves the input items associated with a specific response. This endpoint is also available at /v1/responses/{response_id}/input_items.

### Method
GET

### Endpoint
/responses/{response_id}/input_items

### Parameters
#### Path Parameters
- **response_id** (string) - Required - The unique identifier of the response.
```

--------------------------------

### Python Client Integration

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/README.md

Connect to the local server using the official OpenAI Python SDK.

```python
from openai import OpenAI

# Connect to local server
client = OpenAI(
    base_url="http://localhost:10240/v1",
    api_key="not-needed"
)

# Simple chat request (Chat Completions)
response = client.chat.completions.create(
    model="mlx-community/gemma-3-1b-it-4bit-DWQ",
    messages=[{"role": "user", "content": "Hello! How are you?"}]
)
print(response.choices[0].message.content)
```

--------------------------------

### ChatCompletionResponse JSON Structure

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_schema_utils.md

Example JSON payload representing a full chat completion response.

```json
{
    "id": "chatcmpl-abc123",
    "object": "chat.completion",
    "created": 1234567890,
    "model": "mlx-community/qwen-2.5-7b",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "Hello! How can I help?"
            },
            "finish_reason": "stop"
        }
    ],
    "usage": {
        "prompt_tokens": 10,
        "completion_tokens": 20,
        "total_tokens": 30,
        "prompt_tokens_details": {"cached_tokens": 0},
        "completion_tokens_details": {"reasoning_tokens": 0}
    }
}
```

--------------------------------

### responses_registry.stream_events

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_responses.md

Streams events for a given response ID, optionally starting after a specific sequence number.

```APIDOC
## stream_events(response_id: str, starting_after: Optional[int] = None) -> AsyncGenerator[ResponseStreamEvent, None]

### Description
Stream events for a response (SSE).

### Parameters
- **response_id** (str) - Required - Response ID
- **starting_after** (int) - Optional - Start after sequence number

### Yields
- **ResponseStreamEvent** (Object) - Yields event objects containing event type and data.
```

--------------------------------

### POST /v1/audio/transcriptions

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_optional_features.md

Transcribes audio to text using mlx-whisper. Requires the 'stt' optional extra to be installed.

```APIDOC
## POST /v1/audio/transcriptions

### Description
Transcribes audio to text using mlx-whisper.

### Method
POST

### Endpoint
/v1/audio/transcriptions
```

--------------------------------

### Configure Chat Completions with MLX Parameters

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/configuration.md

Use these parameters to enable reasoning and fine-tune sampling behavior in chat completion requests.

```python
response = client.chat.completions.create(
    model="mlx-community/qwen-2.5-7b",
    messages=[{"role": "user", "content": "Solve this math problem..."}],
    temperature=0.7,
    
    # Extra MLX parameters
    enable_thinking=True,
    thinking_budget=5000,
    repetition_penalty=1.1,
    top_k=40,
    min_p=0.01,
)
```

--------------------------------

### POST /v1/images/generations

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_optional_features.md

Generates images using the mflux module. Requires the 'images' optional extra to be installed.

```APIDOC
## POST /v1/images/generations

### Description
Generates images using the mflux module.

### Method
POST

### Endpoint
/v1/images/generations
```

--------------------------------

### Concurrent Request Handling

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/configuration.md

Demonstrates how multiple clients can connect concurrently using asyncio.

```python
# Multiple clients can connect
for i in range(10):
    asyncio.create_task(make_request())  # Concurrent
await asyncio.gather(...)  # Wait for all
```

--------------------------------

### Retrieve specific model information

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_models.md

Demonstrates fetching a model by ID and optionally including its configuration details.

```python
model = service.get_model("mlx-community/gemma-3-1b-it-4bit-DWQ")
if model:
    print(f"Model: {model.id}")
    print(f"Owner: {model.owned_by}")
    print(f"Created: {model.created}")
else:
    print("Model not found")

# With details
model_detailed = service.get_model(
    "mlx-community/gemma-3-1b-it-4bit-DWQ",
    include_details=True
)
if model_detailed and model_detailed.details:
    print(f"Config: {model_detailed.details}")
```

--------------------------------

### Enable Debug Logging

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_runtime.md

Configures the server to output detailed execution traces including request hashing and timing metrics.

```bash
mlx-omni-server --log-level debug
```

--------------------------------

### not_installed_detail(extra: str, *, missing: Tuple[str, ...] | None = None)

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_optional_features.md

Generates a user-friendly error message for a missing feature.

```APIDOC
## not_installed_detail(extra: str, *, missing: Tuple[str, ...] | None = None)

### Description
Generates a detailed error message explaining that a feature is not installed and how to install it.

### Parameters
- **extra** (str) - Required - Feature name.
- **missing** (tuple) - Optional - Pre-computed missing packages (auto-detect if None).

### Returns
- **str** - User-friendly error message.
```

--------------------------------

### Schema and Logging Utilities

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/EXPORTS.md

Helper functions for parameter extraction, logging configuration, and request/response middleware.

```python
# src/mlx_omni_server/schema_utils.py
def extract_extra_params(
    model: BaseModel,
    standard_fields: Collection[str],
) -> Dict[str, Any]

# src/mlx_omni_server/utils/logger.py
logger: logging.Logger
def configure_logging(...) -> dict
def default_log_dir() -> Path

# src/mlx_omni_server/middleware/logging.py
class RequestResponseLoggingMiddleware:
    async def __call__(scope, receive, send) -> None
```

--------------------------------

### Run unit tests

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/docs/development_guide.md

Execute only the unit test suite.

```bash
python3 -m pytest tests/unit -q
```

--------------------------------

### GET /v1/responses/{response_id}/input_items

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/endpoints.md

Retrieves a paginated list of input items (messages) associated with a specific response ID.

```APIDOC
## GET /v1/responses/{response_id}/input_items

### Description
List input items (messages) for a response with pagination.

### Method
GET

### Endpoint
/v1/responses/{response_id}/input_items

### Parameters
#### Path Parameters
- **response_id** (string) - Required - The ID of the response to retrieve items for.

#### Query Parameters
- **order** (string) - Optional - Sort order, either 'asc' or 'desc'.
- **after** (string) - Optional - Cursor for pagination, item ID to start after.
- **before** (string) - Optional - Cursor for pagination, item ID to start before.
- **limit** (integer) - Optional - Number of items to return, range [1-100].

### Response
#### Success Response (200)
- **object** (string) - The type of object returned (list).
- **data** (array) - List of message objects.
- **first_id** (string) - ID of the first item in the list.
- **last_id** (string) - ID of the last item in the list.
- **has_more** (boolean) - Indicates if more items are available.
```

--------------------------------

### adapter.on_done

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_responses.md

Handles the completion of the stream.

```APIDOC
## adapter.on_done

### Description
Finalizes the stream and returns completion events.

### Returns
- Iterable[ResponseStreamEvent] - Typically returns a response.completed event.

### Example
```python
events = adapter.on_done()
for event in events:
    print(event.event)
```
```

--------------------------------

### Import Response Modules

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/INDEX.md

Imports registry, adapter, and schema components for handling responses.

```python
from mlx_omni_server.responses.registry import responses_registry
from mlx_omni_server.responses.adapter import ResponseStreamAdapter
from mlx_omni_server.responses.schema import ResponseRequest, ResponseResponse
```

--------------------------------

### Stream response events

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_responses.md

Retrieves an asynchronous generator of events for a given response ID, optionally starting after a specific sequence number.

```python
async def stream_events(
    response_id: str,
    starting_after: Optional[int] = None,
) -> AsyncGenerator[ResponseStreamEvent, None]
```

```python
async for event in await responses_registry.stream_events("resp_abc123"):
    print(f"Event: {event.event}")
    print(f"Data: {event.data}")
```

--------------------------------

### Cache missing packages detection

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_optional_features.md

Uses lru_cache to store results of missing package checks, requiring a server restart if dependencies are installed.

```python
@lru_cache(maxsize=None)
def missing_packages(extra: str) -> Tuple[str, ...]:
    ...
```

--------------------------------

### Run tests with pyenv

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/docs/development_guide.md

Execute tests using a repo-local pyenv environment.

```bash
pyenv exec python3 -m pytest tests/unit
```

--------------------------------

### Enforce feature availability in routes

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_optional_features.md

Assert that a feature is installed within a route handler, raising an HTTP 501 error if dependencies are missing.

```python
from mlx_omni_server.optional_features import ensure_extra_available

@app.post("/images/generations")
async def create_image(request):
    ensure_extra_available("images")  # Raises 501 if not installed
    # ... proceed with image generation
```

```json
{
    "detail": "Image generation support is not installed. Install optional dependencies with `pip install '.[images]'`. Missing: mflux"
}
```

--------------------------------

### GET /v1/responses/{response_id}/input_items Response Structure

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/endpoints.md

The expected JSON response format for listing input items associated with a specific response ID.

```python
{
    "object": "list",
    "data": [
        {
            "id": "<item_id>",
            "type": "message",
            "role": "user|assistant|system",
            "content": "...",
        }
    ],
    "first_id": "<first_item_id>",
    "last_id": "<last_item_id>",
    "has_more": False,
}
```

--------------------------------

### Perform Chat Completions with OpenAI SDK

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/00_START_HERE.txt

Connect to the local MLX Omni Server instance to generate chat completions using the OpenAI client library.

```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:10240/v1", api_key="")
response = client.chat.completions.create(
    model="mlx-community/gemma-3-1b-it-4bit-DWQ",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)
```

--------------------------------

### Execute Tool/Function Calling

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/docs/apis/chat.md

Provide a list of tools in the standard OpenAI format to enable function calling capabilities.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:10240/v1", api_key="not-needed")

resp = client.chat.completions.create(
    model="mlx-community/Qwen3-Coder-30B-A3B-Instruct-8bit",
    messages=[{"role": "user", "content": "What's the weather in Boston today?"}],
    tools=[
        {
            "type": "function",
            "function": {
                "name": "get_current_weather",
                "description": "Get the weather in a location",
                "parameters": {
                    "type": "object",
                    "properties": {"location": {"type": "string"}},
                    "required": ["location"],
                },
            },
        }
    ],
    tool_choice="auto",
)

print(resp.choices[0].message.tool_calls)
```

--------------------------------

### Define Documentation Structure

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/README.md

Standard template for documenting modules, classes, and functions within the project.

```text
[File Title]

[Module path and overview]

[Section 1: Key Class/Function]
  - Full signature
  - Parameter table (name | type | required | default | description)
  - Return type documentation
  - Example code
  - Source file path and line reference

[Section 2: Related Data Structures]
  - Field tables
  - Cross-references to other types
  - Usage patterns

[Section 3: Constants/Enums]
  - Complete values
  - When to use each
```

--------------------------------

### Configure Reasoning HMAC Key

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/api_reference_responses.md

Set the required 32-character HMAC key via environment variable.

```bash
export MLX_OMNI_SERVER_REASONING_HMAC_KEY=<your-32-char-key>
```

--------------------------------

### Define StreamOptions model

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/_autodocs/types.md

Use this model to configure streaming behavior. Setting include_usage to true ensures the final chunk contains token usage data.

```python
class StreamOptions(BaseModel):
    include_usage: bool = False
```

--------------------------------

### Generate Speech with OpenAI SDK (Python)

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/docs/apis/audio.md

Use the OpenAI Python client to generate speech and stream the result to a file.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:10240/v1", api_key="not-needed")

response = client.audio.speech.create(
    model="mlx-community/Kokoro-82M-4bit",
    input="Hello from MLX Omni Server",
    voice="af_sky",
    response_format="wav",
)
response.stream_to_file("speech.wav")
```

--------------------------------

### Verify Server Status

Source: https://github.com/zhutao100/mlx-omni-server/blob/main/README.md

Use cURL to check if the server is running and listing models.

```bash
curl http://localhost:10240/v1/models
```
Tokens:
Raw