Xinference

https://inference.readthedocs.io/en/stable/
Admin
Xinference is an open-source platform that streamlines the operation and integration of AI models,...
Tokens:414,363
Snippets:5,195
Trust Score:8.3
Update:3 months ago
Show doc for...
Tokens:
Raw
### Launch Qwen2-MoE-Instruct (GGUFv2)

Source: https://inference.readthedocs.io/en/stable/models/builtin/llm/qwen2-moe-instruct.html

Use this command to launch the Qwen2-MoE-Instruct model with GGUFv2 format. Replace `${engine}` with your chosen engine (vLLM, llama.cpp) and `${quantization}` with the desired quantization method (e.g., q3_k_m, q4_0, q4_k_m, q5_0, q5_k_m, q6_k, q8_0, fp16).

```bash
xinference launch --model-engine ${engine} --model-name qwen2-moe-instruct --size-in-billions 14 --model-format ggufv2 --quantization ${quantization}

```

--------------------------------

### Example Segmentation Fault Output

Source: https://inference.readthedocs.io/en/stable/_sources/getting_started/troubleshooting.rst.txt

This output indicates a segmentation fault when starting xinference, often occurring after upgrading xinference with uv while vLLM is installed.

```text
root@server:/home# xinference-local --host 0.0.0.0 --port 9997
INFO 12-30 17:35:37 [__init__.py:216] Automatically detected platform cuda.
Aborted (core dumped)
```

--------------------------------

### Launch Qwen3-Instruct (GGUFv2, 30B)

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen3-instruct.rst.txt

Use this command to launch the Qwen3-Instruct model in GGUFv2 format with 30 billion parameters. Replace `${engine}` with a supported engine (e.g., vLLM, llama.cpp) and `${quantization}` with a supported quantization method.

```bash
xinference launch --model-engine ${engine} --model-name Qwen3-Instruct --size-in-billions 30 --model-format ggufv2 --quantization ${quantization}
```

--------------------------------

### Define Get Weather Function for Messages API

Source: https://inference.readthedocs.io/en/stable/_sources/models/model_abilities/tools.rst.txt

Example of defining a tool for the Anthropic client's messages API. This snippet shows the setup for calling the 'get_weather' function with the Anthropic client, specifying the model, max tokens, messages, and the tools parameter.

```python
import anthropic
import json
import uuid

client = anthropic.Anthropic(
    api_key="cannot be empty",
    base_url="http://localhost:9997"
)

response = client.messages.create(
    model="qwen3",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": "What's the weather like in Beijing?"
        }
    ],
    tools=[
        {
            "type": "function",
            "function": {
                "name": "get_weather",

```

--------------------------------

### Virtual Environment Creation Without Skipping Installed

Source: https://inference.readthedocs.io/en/stable/models/virtualenv.html

Example output showing the number of packages installed and their duration when the 'skip installed' feature is not enabled. This demonstrates a typical scenario with redundant installations.

```text
Installed 98 packages in 187ms
 + aiohappyeyeballs==2.6.1
 + aiohttp==3.12.13
 ...
 + torch==2.7.1
 ...
 + yarl==1.20.1
 + zipp==3.23.0
```

--------------------------------

### Launch Qwen3-Instruct (GGUFv2, 4B)

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen3-instruct.rst.txt

Use this command to launch the Qwen3-Instruct model in GGUFv2 format with 4 billion parameters. Replace `${engine}` with a supported engine (e.g., vLLM, llama.cpp) and `${quantization}` with a supported quantization method.

```bash
xinference launch --model-engine ${engine} --model-name Qwen3-Instruct --size-in-billions 4 --model-format ggufv2 --quantization ${quantization}
```

--------------------------------

### Launch Qwen3-Next-Instruct (FP8)

Source: https://inference.readthedocs.io/en/stable/models/builtin/llm/qwen3-next-instruct.html

Execute this command to launch the Qwen3-Next-Instruct model with FP8 format. Replace `${quantization}` with your chosen quantization method.

```bash
xinference launch --model-engine ${engine} --model-name Qwen3-Next-Instruct --size-in-billions 80 --model-format fp8 --quantization ${quantization}

```

--------------------------------

### Virtual Environment Creation With Skipping Installed

Source: https://inference.readthedocs.io/en/stable/models/virtualenv.html

Example output showing the reduced number of packages installed and their significantly shorter duration when the 'skip installed' feature is enabled. This highlights the efficiency gains.

```text
Installed 7 packages in 12ms
 + diffusers==0.29.0
 + hf-xet==1.1.5
 + huggingface-hub==0.33.2
 + importlib-metadata==8.7.0
 + pillow==11.3.0
 + typing-extensions==4.14.0
 + urllib3==2.5.0
```

--------------------------------

### Launch Qwen3-Instruct (GGUFv2, 235B)

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen3-instruct.rst.txt

Use this command to launch the Qwen3-Instruct model in GGUFv2 format with 235 billion parameters. Replace `${engine}` with a supported engine (e.g., vLLM, llama.cpp) and `${quantization}` with a supported quantization method.

```bash
xinference launch --model-engine ${engine} --model-name Qwen3-Instruct --size-in-billions 235 --model-format ggufv2 --quantization ${quantization}
```

--------------------------------

### Install Xinference with uv (Failing Example)

Source: https://inference.readthedocs.io/en/stable/getting_started/troubleshooting.html

This command demonstrates the failing installation scenario when using uv without specific version constraints.

```bash
uv pip install xinference
```

--------------------------------

### Install PyTorch with Specific CUDA Version

Source: https://inference.readthedocs.io/en/stable/_sources/getting_started/troubleshooting.rst.txt

Install a PyTorch version compatible with your CUDA driver. This example installs PyTorch 2.0.1 compiled with CUDA 11.8. Ensure your CUDA driver is at least 11.8.

```python
pip install torch==2.0.1+cu118
```

--------------------------------

### Launch Qwen3-Next-Instruct (PyTorch)

Source: https://inference.readthedocs.io/en/stable/models/builtin/llm/qwen3-next-instruct.html

Use this command to launch the Qwen3-Next-Instruct model with PyTorch format. Replace `${quantization}` with your chosen quantization method.

```bash
xinference launch --model-engine ${engine} --model-name Qwen3-Next-Instruct --size-in-billions 80 --model-format pytorch --quantization ${quantization}

```

--------------------------------

### Start Local Cluster with Debug Log Level

Source: https://inference.readthedocs.io/en/stable/getting_started/logging.html

Use the `--log-level` option to set the desired log verbosity when starting a local Xinference cluster. This example sets the log level to debug.

```bash
xinference-local --log-level debug

```

--------------------------------

### Launch Qwen3-Instruct (MLX, 4B)

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen3-instruct.rst.txt

Use this command to launch the Qwen3-Instruct model in MLX format with 4 billion parameters. Replace `${quantization}` with a supported quantization method (e.g., 4bit, 5bit, 6bit, 8bit).

```bash
xinference launch --model-engine ${engine} --model-name Qwen3-Instruct --size-in-billions 4 --model-format mlx --quantization ${quantization}
```

--------------------------------

### Launch Qwen3-Instruct (MLX, 30B)

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen3-instruct.rst.txt

Use this command to launch the Qwen3-Instruct model in MLX format with 30 billion parameters. Replace `${quantization}` with a supported quantization method (e.g., 4bit, 5bit, 6bit, 8bit).

```bash
xinference launch --model-engine ${engine} --model-name Qwen3-Instruct --size-in-billions 30 --model-format mlx --quantization ${quantization}
```

--------------------------------

### Start Xinference Local Server

Source: https://inference.readthedocs.io/en/stable/_sources/getting_started/using_xinference.rst.txt

Run a local instance of Xinference. Ensure Xinference is installed before executing this command.

```bash
xinference-local --host 0.0.0.0 --port 9997
```

--------------------------------

### Guide emotions using text with use_emo_text

Source: https://inference.readthedocs.io/en/stable/models/model_abilities/audio.html

This example demonstrates guiding emotions based on a provided text script using `use_emo_text`, which automatically converts the script into emotion vectors. `emo_alpha` is recommended to be around 0.6 or lower for natural speech.

```APIDOC
## Guide emotions with text using use_emo_text

### Description
Guides emotions based on a provided text script, which is automatically converted into emotion vectors. `emo_alpha` is recommended to be around `0.6` or lower for more natural sounding speech. Randomness can be introduced with `use_random`.

### Method
model.speech()

### Parameters
- **input** (str) - Required - The text to be synthesized.
- **prompt_speech** (bytes) - Required - The reference audio file content for voice cloning.
- **emo_alpha** (float) - Optional - The intensity factor for the emotional guidance. Recommended around `0.6` or lower.
- **use_emo_text** (bool) - Required - Enables emotion guidance based on the input text script. Defaults to `False`.
- **use_random** (bool) - Optional - Enables randomness during inference. Defaults to `False`.

### Request Example
```python
from xinference.client import Client
client = Client("http://0.0.0.0:6735")
model = client.get_model("IndexTTS2")

with open("../mp3_test_voice.mp3", "rb") as f:
    test_prompt_speech = f.read()

response = model.speech(
    input="Quick, hide! He's coming! He's coming to get us!",
    prompt_speech=test_prompt_speech,
    emo_alpha=0.6,
    use_emo_text=True,
    use_random=False
)
```
```

--------------------------------

### Launch Qwen2-VL-Instruct (GPTQ, 7B)

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen2-vl-instruct.rst.txt

Use this command to launch the Qwen2-VL-Instruct model in GPTQ format with 7 billion parameters. Replace ${engine} and ${quantization} with your chosen engine and quantization method.

```bash
xinference launch --model-engine ${engine} --model-name qwen2-vl-instruct --size-in-billions 7 --model-format gptq --quantization ${quantization}
```

--------------------------------

### Start Xinference Supervisor with Authentication

Source: https://inference.readthedocs.io/en/stable/_sources/user_guide/auth_system.rst.txt

This command enables authentication for a distributed Xinference setup by specifying the supervisor IP and the path to the authentication configuration file.

```bash
xinference-supervisor -H <supervisor_ip> --auth-config /path/to/your_json_config_file
```

--------------------------------

### Launch Qwen3-Next-Instruct (PyTorch)

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen3-next-instruct.rst.txt

Use this command to launch the Qwen3-Next-Instruct model with PyTorch format. Replace ${engine} with your chosen engine (e.g., vLLM, Transformers) and ${quantization} with the desired quantization method.

```bash
xinference launch --model-engine ${engine} --model-name Qwen3-Next-Instruct --size-in-billions 80 --model-format pytorch --quantization ${quantization}
```

--------------------------------

### Distributed Xinference Supervisor Startup with Auth

Source: https://inference.readthedocs.io/en/stable/user_guide/auth_system.html

Command to start the Xinference supervisor in a distributed setup with authentication and authorization enabled. Replace `<supervisor_ip>` with the actual supervisor IP address.

```bash
xinference-supervisor -H <supervisor_ip> --auth-config /path/to/your_json_config_file

```

--------------------------------

### Launch qwen2-audio-instruct Model

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen2-audio-instruct.rst.txt

Execute this command to launch the qwen2-audio-instruct model. Replace `${engine}` with your chosen engine (e.g., vLLM, Transformers) and `${quantization}` with your chosen quantization method.

```bash
xinference launch --model-engine ${engine} --model-name qwen2-audio-instruct --size-in-billions 7 --model-format pytorch --quantization ${quantization}
```

--------------------------------

### Inference Zero-Shot Classification with Xinference Python Client

Source: https://inference.readthedocs.io/en/stable/_sources/models/model_abilities/flexible.rst.txt

Perform zero-shot classification inference using the Xinference Python client. This example demonstrates how to get a model and call its infer method with a sentence and candidate labels.

```python
from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("nlp_structbert_zero-shot-classification_chinese-base")

labels = ['家居', '旅游', '科技', '军事', '游戏', '故事']
sentence = '世界那么大，我想去看看'
model.infer(sentence, candidate_labels=labels)
```

--------------------------------

### Launch Qwen3-Instruct (MLX, 235B)

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen3-instruct.rst.txt

Use this command to launch the Qwen3-Instruct model in MLX format with 235 billion parameters. Replace `${quantization}` with a supported quantization method (e.g., 3bit, 4bit, 5bit, 6bit, 8bit).

```bash
xinference launch --model-engine ${engine} --model-name Qwen3-Instruct --size-in-billions 235 --model-format mlx --quantization ${quantization}
```

--------------------------------

### Chat API Request using Xinference Python Client

Source: https://inference.readthedocs.io/en/stable/models/model_abilities/chat.html

Interact with the Chat API using Xinference's dedicated Python client. This example shows how to get a model and send a chat request with generation configurations.

```python
from xinference.client import RESTfulClient

client = RESTfulClient("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
model = client.get_model("<MODEL_UID>")
messages = [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the largest animal?"}]
model.chat(
    messages,
    generate_config={
      "max_tokens": 512,
      "temperature": 0.7
    }
)
```

--------------------------------

### Launch Qwen3-Omni-Instruct (PyTorch)

Source: https://inference.readthedocs.io/en/stable/models/builtin/llm/qwen3-omni-instruct.html

Execute this command to launch the Qwen3-Omni-Instruct model with the PyTorch format. Replace `${engine}` with your chosen engine (e.g., vLLM, Transformers) and `${quantization}` with your chosen quantization method.

```bash
xinference launch --model-engine ${engine} --model-name Qwen3-Omni-Instruct --size-in-billions 30 --model-format pytorch --quantization ${quantization}

```

--------------------------------

### Transcribe Audio using Xinference Python Client

Source: https://inference.readthedocs.io/en/stable/_sources/models/model_abilities/audio.rst.txt

Utilize Xinference's dedicated Python client to transcribe audio. This example shows how to connect to the Xinference server, get a model, and then call the transcription method with the audio data.

```python
from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")
with open("speech.mp3", "rb") as audio_file:
    model.transcriptions(audio=audio_file.read())
```

--------------------------------

### Launch Qwen2-MoE-Instruct (GPTQ)

Source: https://inference.readthedocs.io/en/stable/models/builtin/llm/qwen2-moe-instruct.html

Use this command to launch the Qwen2-MoE-Instruct model with GPTQ format. Replace `${engine}` with your chosen engine (vLLM, Transformers, SGLang) and `${quantization}` with the desired quantization method (Int4).

```bash
xinference launch --model-engine ${engine} --model-name qwen2-moe-instruct --size-in-billions 14 --model-format gptq --quantization ${quantization}

```

--------------------------------

### Launch Qwen3-VL-Instruct (30B, AWQ)

Source: https://inference.readthedocs.io/en/stable/models/builtin/llm/qwen3-vl-instruct.html

Use this command to launch the Qwen3-VL-Instruct model with a 30 billion parameter size in AWQ format. Replace `${engine}` and `${quantization}` with your chosen engine and quantization method.

```bash
xinference launch --model-engine ${engine} --model-name Qwen3-VL-Instruct --size-in-billions 30 --model-format awq --quantization ${quantization}

```

--------------------------------

### Install Xinference with pip

Source: https://inference.readthedocs.io/en/stable/_sources/getting_started/troubleshooting.rst.txt

Use 'pip install xinference' for a simpler installation that resolves to specific compatible versions of its dependencies.

```bash
pip install xinference
```

--------------------------------

### Launch Qwen3-Next-Instruct (FP8)

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen3-next-instruct.rst.txt

Execute this command to launch the Qwen3-Next-Instruct model with FP8 format. Ensure to replace ${engine} with your selected engine (e.g., vLLM) and ${quantization} with the appropriate FP8 quantization method.

```bash
xinference launch --model-engine ${engine} --model-name Qwen3-Next-Instruct --size-in-billions 80 --model-format fp8 --quantization ${quantization}
```

--------------------------------

### Install and Build Frontend

Source: https://inference.readthedocs.io/en/stable/_sources/development/contributing_environment.rst.txt

Install frontend dependencies and build the frontend for Xinference. The '--force' option can be used for 'npm install' if it fails.

```bash
npm install
npm run build
```

--------------------------------

### cURL Example

Source: https://inference.readthedocs.io/en/stable/models/model_abilities/embed.html

Example of how to call the Embeddings API using cURL.

```APIDOC
```bash
curl -X 'POST' \
  'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/embeddings' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{ 
    "model": "<MODEL_UID>", 
    "input": "What is the capital of China?"
  }'
```
```

--------------------------------

### Launch Qwen3-VL-Instruct (8B, AWQ)

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen3-vl-instruct.rst.txt

Use this command to launch the 8 billion parameter Qwen3-VL-Instruct model with AWQ quantization. Replace ${engine} with your chosen engine and ${quantization} with the desired quantization method (e.g., 4bit, 8bit).

```bash
xinference launch --model-engine ${engine} --model-name Qwen3-VL-Instruct --size-in-billions 8 --model-format awq --quantization ${quantization}
```

--------------------------------

### Launch Qwen2-MoE-Instruct (PyTorch)

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen2-moe-instruct.rst.txt

Use this command to launch the Qwen2-MoE-Instruct model with PyTorch format. Replace `${engine}` with your chosen engine (e.g., vLLM, Transformers, SGLang) and `${quantization}` with the desired quantization method (none for this format).

```bash
xinference launch --model-engine ${engine} --model-name qwen2-moe-instruct --size-in-billions 14 --model-format pytorch --quantization ${quantization}
```

--------------------------------

### Installing and Building Frontend Assets

Source: https://inference.readthedocs.io/en/stable/development/contributing_environment.html

Installs frontend dependencies and builds the frontend for Xinference. Navigate to 'inference/xinference/ui/web/ui' directory first. The '--force' option can be used if the install command fails.

```bash
npm install
npm run build

```

--------------------------------

### Install Xinference with Transformers Quantization Dependencies

Source: https://inference.readthedocs.io/en/stable/_sources/getting_started/installation.rst.txt

Install Xinference with specific dependencies for AWQ or GPTQ formats when using the transformers engine. This requires manual installation due to special options and potential dependency conflicts. Consider running 'pip install "xinference[all]"' afterwards to resolve any potential downgrades.

```bash
pip install "xinference[transformers_quantization]" --no-build-isolation
```

--------------------------------

### Launch Qwen2.5-7B-Instruct (GPTQ)

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen2.5-instruct.rst.txt

This command launches the 7B GPTQ version of Qwen2.5 Instruct. Remember to substitute ${engine} and ${quantization} with your choices.

```bash
xinference launch --model-engine ${engine} --model-name qwen2.5-instruct --size-in-billions 7 --model-format gptq --quantization ${quantization}
```

--------------------------------

### Install Xinference

Source: https://inference.readthedocs.io/en/stable/getting_started/using_xinference.html

Installs the Xinference package, which includes the command-line tool and Python client.

```shell
pip install xinference

```

--------------------------------

### Launch Qwen3-VL-Instruct (235B, AWQ)

Source: https://inference.readthedocs.io/en/stable/models/builtin/llm/qwen3-vl-instruct.html

Use this command to launch the Qwen3-VL-Instruct model with a 235 billion parameter size in AWQ format. Replace `${engine}` and `${quantization}` with your chosen engine and quantization method.

```bash
xinference launch --model-engine ${engine} --model-name Qwen3-VL-Instruct --size-in-billions 235 --model-format awq --quantization ${quantization}

```

--------------------------------

### Launch Qwen3-VL-Instruct (30B, FP8)

Source: https://inference.readthedocs.io/en/stable/models/builtin/llm/qwen3-vl-instruct.html

Use this command to launch the Qwen3-VL-Instruct model with a 30 billion parameter size in FP8 format. Replace `${engine}` and `${quantization}` with your chosen engine and quantization method.

```bash
xinference launch --model-engine ${engine} --model-name Qwen3-VL-Instruct --size-in-billions 30 --model-format fp8 --quantization ${quantization}

```

--------------------------------

### Text-to-Image Output Example

Source: https://inference.readthedocs.io/en/stable/models/model_abilities/image.html

This is an example of the JSON output received after a successful text-to-image generation request.

```json
{
  "created": 1697536913,
  "data": [
    {
      "url": "/home/admin/.xinference/image/605d2f545ac74142b8031455af31ee33.jpg",
      "b64_json": null
    }
  ]
}

```

--------------------------------

### Embedding Model Output Example

Source: https://inference.readthedocs.io/en/stable/user_guide/client_api.html

This is an example of the output received when creating an embedding using the Xinference client.

```json
{'object': 'list',
 'model': 'da2a511c-6ccc-11ee-ad07-22c9969c1611-1-0',
 'data': [{'index': 0,
 'object': 'embedding',
 'embedding': [-0.014207549393177032,
    -0.01832585781812668,
    0.010556723922491074,
    ...
    -0.021243810653686523,
    -0.03009396605193615,
    0.05420297756791115]}],
 'usage': {'prompt_tokens': 37, 'total_tokens': 37}}


```

--------------------------------

### Launch Qwen3-Next-Instruct (MLX)

Source: https://inference.readthedocs.io/en/stable/models/builtin/llm/qwen3-next-instruct.html

Use this command to launch the Qwen3-Next-Instruct model with MLX format. Replace `${quantization}` with your chosen quantization method (e.g., 4bit, 5bit, 6bit, 8bit).

```bash
xinference launch --model-engine ${engine} --model-name Qwen3-Next-Instruct --size-in-billions 80 --model-format mlx --quantization ${quantization}

```

--------------------------------

### Launch Qwen2.5-Coder-7B-Instruct GGUFv2

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen2.5-coder-instruct.rst.txt

Use this command to launch the 7B Instruct model in GGUFv2 format. Replace ${engine} with your chosen engine (vLLM, llama.cpp), and ${quantization} with one of the supported quantization methods (q2_k, q3_k_m, q4_0, q4_k_m, q5_0, q5_k_m, q6_k, q8_0).

```bash
xinference launch --model-engine ${engine} --model-name qwen2.5-coder-instruct --size-in-billions 7 --model-format ggufv2 --quantization ${quantization}
```

--------------------------------

### Xinference Python Client Example

Source: https://inference.readthedocs.io/en/stable/models/model_abilities/embed.html

Example of how to call the Embeddings API using the Xinference Python client.

```APIDOC
```python
from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")
input = "What is the capital of China?"
model.create_embedding(input)
```
```

--------------------------------

### Launch Qwen2.5-Instruct 1.5B Model

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen2.5-instruct.rst.txt

Use this command to launch the Qwen2.5-Instruct model with 1.5 billion parameters. Replace ${engine} with your chosen engine (e.g., vLLM, Transformers, SGLang) and ${quantization} with your selected quantization method.

```bash
xinference launch --model-engine ${engine} --model-name qwen2.5-instruct --size-in-billions 1_5 --model-format pytorch --quantization ${quantization}
```

--------------------------------

### OpenAI Python Client Example

Source: https://inference.readthedocs.io/en/stable/models/model_abilities/embed.html

Example of how to call the Embeddings API using the OpenAI Python client.

```APIDOC
```python
import openai

client = openai.Client(
  api_key="cannot be empty",
  base_url="http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1"
)
client.embeddings.create(
  model=model_uid,
  input=["What is the capital of China?"]
)
```
```

--------------------------------

### Install pre-commit

Source: https://inference.readthedocs.io/en/stable/development/contributing_codebase.html

Install the pre-commit tool to automatically run code formatting and style checks before committing changes.

```bash
pip install pre-commit

```

--------------------------------

### Launch Qwen2.5-Instruct 0.5B Model

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen2.5-instruct.rst.txt

Use this command to launch the Qwen2.5-Instruct model with 0.5 billion parameters. Replace ${engine} with your chosen engine (e.g., vLLM, Transformers, SGLang) and ${quantization} with your selected quantization method.

```bash
xinference launch --model-engine ${engine} --model-name qwen2.5-instruct --size-in-billions 0_5 --model-format pytorch --quantization ${quantization}
```

--------------------------------

### Install Xinference

Source: https://inference.readthedocs.io/en/stable/_sources/getting_started/installation_npu.rst.txt

Installs the Xinference package using pip. This is the final step for setting up Xinference for Ascend NPU.

```bash
pip3 install xinference
```

--------------------------------

### Launch Qwen3-VL-Instruct (30B, PyTorch)

Source: https://inference.readthedocs.io/en/stable/models/builtin/llm/qwen3-vl-instruct.html

Use this command to launch the Qwen3-VL-Instruct model with a 30 billion parameter size in PyTorch format. Replace `${engine}` and `${quantization}` with your chosen engine and quantization method.

```bash
xinference launch --model-engine ${engine} --model-name Qwen3-VL-Instruct --size-in-billions 30 --model-format pytorch --quantization ${quantization}

```

--------------------------------

### Launch Qwen2.5-Coder-1.5B-Instruct GGUFv2

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen2.5-coder-instruct.rst.txt

Use this command to launch the 1.5B Instruct model in GGUFv2 format. Replace ${engine} with your chosen engine (vLLM, llama.cpp), and ${quantization} with one of the supported quantization methods (q2_k, q3_k_m, q4_0, q4_k_m, q5_0, q5_k_m, q6_k, q8_0).

```bash
xinference launch --model-engine ${engine} --model-name qwen2.5-coder-instruct --size-in-billions 1_5 --model-format ggufv2 --quantization ${quantization}
```

--------------------------------

### Chat API Response Example

Source: https://inference.readthedocs.io/en/stable/models/model_abilities/chat.html

An example of the JSON response structure received from a Chat API completion request.

```json
{
  "id": "chatcmpl-8d76b65a-bad0-42ef-912d-4a0533d90d61",
  "model": "<MODEL_UID>",
  "object": "chat.completion",
  "created": 1688919187,
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The largest animal that has been scientifically measured is the blue whale, which has a maximum length of around 23 meters (75 feet) for adult animals and can weigh up to 150,000 pounds (68,000 kg). However, it is important to note that this is just an estimate and that the largest animal known to science may be larger still. Some scientists believe that the largest animals may not have a clear \"size\" in the same way that humans do, as their size can vary depending on the environment and the stage of their life."
      },
      "finish_reason": "None"
    }
  ],
  "usage": {
    "prompt_tokens": -1,
    "completion_tokens": -1,
    "total_tokens": -1
  }
}
```

--------------------------------

### Launch Qwen2-MoE-Instruct (GPTQ)

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen2-moe-instruct.rst.txt

Execute this command to launch the Qwen2-MoE-Instruct model in GPTQ format. Specify your engine and replace `${quantization}` with a GPTQ quantization method like Int4.

```bash
xinference launch --model-engine ${engine} --model-name qwen2-moe-instruct --size-in-billions 14 --model-format gptq --quantization ${quantization}
```

--------------------------------

### Translation API Output Example

Source: https://inference.readthedocs.io/en/stable/_sources/models/model_abilities/audio.rst.txt

Example JSON output for the audio translation API, showing the translated text.

```json
{
  "text": "Hello, my name is Wolfgang and I come from Germany. Where are you heading today?"
}
```

--------------------------------

### Launch Qwen2-MoE-Instruct (GGUFv2)

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen2-moe-instruct.rst.txt

This command launches the Qwen2-MoE-Instruct model using the GGUFv2 format. Choose your engine and select a quantization method from the available options (e.g., q4_k_m, q8_0).

```bash
xinference launch --model-engine ${engine} --model-name qwen2-moe-instruct --size-in-billions 14 --model-format ggufv2 --quantization ${quantization}
```

--------------------------------

### Install Xinference with Virtual Environment Support

Source: https://inference.readthedocs.io/en/stable/_sources/models/virtualenv.rst.txt

Install the necessary packages for Xinference, specifically enabling virtual environment functionality.

```bash
pip install 'xinference[all]'
# or virtualenv
pip install 'xinference[virtualenv]'
```

--------------------------------

### Install Xinference with MLX Backend

Source: https://inference.readthedocs.io/en/stable/_sources/getting_started/installation.rst.txt

Installs Xinference with the MLX-lm backend, designed for efficient LLM execution on Apple silicon.

```bash
pip install "xinference[mlx]"
```

--------------------------------

### Launch Qwen2 Instruct (MLX, 1.5B)

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen2-instruct.rst.txt

Use this command to launch the Qwen2 Instruct model with 1.5 billion parameters in MLX format. Replace ${engine} and ${quantization} with your chosen engine and quantization method.

```bash
xinference launch --model-engine ${engine} --model-name qwen2-instruct --size-in-billions 1_5 --model-format mlx --quantization ${quantization}
```

--------------------------------

### Launch Qwen2.5-1.5B-Instruct (GPTQ)

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen2.5-instruct.rst.txt

Use this command to launch the 1.5B GPTQ version of Qwen2.5 Instruct. Ensure to replace ${engine} and ${quantization} with appropriate values.

```bash
xinference launch --model-engine ${engine} --model-name qwen2.5-instruct --size-in-billions 1_5 --model-format gptq --quantization ${quantization}
```

--------------------------------

### Install Python and Node.js in Conda Environment

Source: https://inference.readthedocs.io/en/stable/_sources/development/contributing_environment.rst.txt

Install Python version 3.12 and Node.js within the activated Conda environment.

```bash
conda install python=3.12
conda install nodejs
```

--------------------------------

### Install pre-commit Hooks

Source: https://inference.readthedocs.io/en/stable/_sources/development/contributing_codebase.rst.txt

Install pre-commit hooks to automatically run code formatting and style checks before committing changes.

```bash
pip install pre-commit
```

--------------------------------

### Launch Qwen2-VL-Instruct (AWQ, 72B)

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen2-vl-instruct.rst.txt

Use this command to launch the Qwen2-VL-Instruct model in AWQ format with 72 billion parameters. Substitute ${engine} and ${quantization} with your selected engine and quantization method.

```bash
xinference launch --model-engine ${engine} --model-name qwen2-vl-instruct --size-in-billions 72 --model-format awq --quantization ${quantization}
```

--------------------------------

### OpenAI Client Embedding Response Example

Source: https://inference.readthedocs.io/en/stable/user_guide/client_api.html

This is an example of the response received when creating embeddings using the OpenAI client with Xinference.

```python
CreateEmbeddingResponse(data=[Embedding(embedding=[-0.014207549393177032, -0.01832585781812668, 0.010556723922491074, ..., -0.021243810653686523, -0.03009396605193615, 0.05420297756791115], index=0, object='embedding')], model='bge-small-en-v1.5-1-0', object='list', usage=Usage(prompt_tokens=37, total_tokens=37))


```

--------------------------------

### Launch Qwen2-0.5B-Instruct-GGUF (GGUFv2)

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen2-instruct.rst.txt

Execute this command to launch the Qwen2-0.5B-Instruct model in GGUFv2 format. Replace ${engine} and ${quantization} with your chosen engine and quantization method.

```bash
xinference launch --model-engine ${engine} --model-name qwen2-instruct --size-in-billions 0_5 --model-format ggufv2 --quantization ${quantization}
```

--------------------------------

### Launch Qwen3-VL-Instruct (235B, FP8)

Source: https://inference.readthedocs.io/en/stable/models/builtin/llm/qwen3-vl-instruct.html

Use this command to launch the Qwen3-VL-Instruct model with a 235 billion parameter size in FP8 format. Replace `${engine}` and `${quantization}` with your chosen engine and quantization method.

```bash
xinference launch --model-engine ${engine} --model-name Qwen3-VL-Instruct --size-in-billions 235 --model-format fp8 --quantization ${quantization}

```

--------------------------------

### Start Xinference Supervisor

Source: https://inference.readthedocs.io/en/stable/_sources/getting_started/using_xinference.rst.txt

Command to start the Xinference supervisor on a designated server. Replace ${supervisor_host} with the actual host address.

```bash
xinference-supervisor -H "${supervisor_host}"
```

--------------------------------

### Install Xinference with Helm

Source: https://inference.readthedocs.io/en/stable/_sources/getting_started/using_kubernetes.rst.txt

Installs the Xinference Helm chart into the specified Kubernetes namespace. Replace `<helm_charts_version>` with the desired chart version.

```bash
helm install xinference xinference/xinference -n xinference --version <helm_charts_version>
```

--------------------------------

### Launch Qwen2.5 Instruct (14B, GGUFv2)

Source: https://inference.readthedocs.io/en/stable/_sources/models/builtin/llm/qwen2.5-instruct.rst.txt

Use this command to launch the 14B parameter Qwen2.5 Instruct model in GGUFv2 format. Replace ${engine} with your chosen engine (e.g., vLLM, llama.cpp) and ${quantization} with a supported quantization method.

```bash
xinference launch --model-engine ${engine} --model-name qwen2.5-instruct --size-in-billions 14 --model-format ggufv2 --quantization ${quantization}
```
Tokens:
Raw