### Base64 Encoded Video Input Example (Partial)

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/multi_modal/multimodal_inputs.md

Illustrates the setup for sending video data as Base64 encoded data. This snippet is incomplete and requires further implementation.

```python
from openai import OpenAI
from lmdeploy.vl.utils import encode_video_base64

client = OpenAI(api_key='EMPTY', base_url='http://localhost:23333/v1')
model_name = client.models.list().data[0].id


```

--------------------------------

### Install Qwen2-VL Utilities

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/multi_modal/qwen2_vl.md

Install the necessary Python package for Qwen2-VL models. This is a prerequisite for using these models with LMDeploy.

```shell
pip install qwen_vl_utils
```

--------------------------------

### Install LMDeploy and VLMEvalKit

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/benchmark/evaluate_with_vlmevalkit.md

Install the necessary libraries for LMDeploy and VLMEvalKit. It's recommended to use separate virtual environments.

```shell
pip install lmdeploy

git clone https://github.com/open-compass/VLMEvalKit.git
cd VLMEvalKit && pip install -e .
```

--------------------------------

### Custom Dockerfile for LLaVA

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/multi_modal/api_server_vl.md

Example Dockerfile to set up a custom environment for LLaVA, including installing specific Python packages and dependencies. This is useful when the official Docker image lacks required libraries.

```docker
FROM openmmlab/lmdeploy:latest

RUN apt-get update && apt-get install -y python3 python3-pip git

WORKDIR /app

RUN pip3 install --upgrade pip
RUN pip3 install timm
RUN pip3 install git+https://github.com/haotian-liu/LLaVA.git --no-deps

COPY . .

CMD ["lmdeploy", "serve", "api_server", "liuhaotian/llava-v1.6-34b"]
```

--------------------------------

### Install Triton for PyTorchEngine

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/llm/pipeline.md

Installs the Triton library, a prerequisite for using the PyTorchEngine with LMDeploy.

```shell
pip install triton>=2.1.0
```

--------------------------------

### Start Ray Head Node

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/advance/pytorch_multinodes.md

Initialize the Ray cluster by starting the head node. This node will coordinate the cluster.

```bash
ray start --head --port=$DRIVER_PORT
```

--------------------------------

### Install LMDeploy

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/quantization/kv_quant.md

Install the LMDeploy library to enable KV quantization and inference capabilities. This is a prerequisite for using the quantization features.

```shell
pip install lmdeploy
```

--------------------------------

### Install debuginfo libraries

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/advance/debug_turbomind.md

Install the debuginfo packages for glibc and python3. These packages contain the necessary symbol information for debugging.

```bash
rpm -ivh glibc-debuginfo-common-2.17-325.el7.x86_64.rpm
rpm -ivh glibc-debuginfo-2.17-325.el7.x86_64.rpm
rpm -ivh python3-debuginfo-3.6.8-21.el7.x86_64.rpm

```

--------------------------------

### Single Video Input Example

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/multi_modal/multimodal_inputs.md

Provides an example of sending a video input along with a text prompt. Note that native video input is supported only for specific models.

```python
from openai import OpenAI

client = OpenAI(api_key='EMPTY', base_url='http://localhost:23333/v1')
model_name = client.models.list().data[0].id

video_url = 'https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-VL/space_woaudio.mp4'

response = client.chat.completions.create(
    model=model_name,
    messages=[
        {
            'role': 'user',
            'content': [
                {
                    'type': 'video_url',
                    'video_url': {
                        'url': video_url,
                    },
                },
                {
                    'type': 'text',
                    'text': "What's in this video?",
                },
            ],
        }
    ],
    temperature=0.8,
    top_p=0.8,
    max_completion_tokens=256,
)
print(response.choices[0].message.content)
```

--------------------------------

### Qwen2.5 Multi-Tool Calling Example

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/llm/api_server_tools.md

This Python script demonstrates multi-tool calling with Qwen2.5. It defines two tools, `get_current_temperature` and `get_temperature_date`, and shows how to make a chat completion request that utilizes these tools. The example includes setting up the OpenAI client, defining tool specifications, and processing the tool calls.

```python
from openai import OpenAI
import json

def get_current_temperature(location: str, unit: str = "celsius"):
    """Get current temperature at a location.

    Args:
        location: The location to get the temperature for, in the format "City, State, Country".
        unit: The unit to return the temperature in. Defaults to "celsius". (choices: ["celsius", "fahrenheit"])

    Returns:
        the temperature, the location, and the unit in a dict
    """
    return {
        "temperature": 26.1,
        "location": location,
        "unit": unit,
    }


def get_temperature_date(location: str, date: str, unit: str = "celsius"):
    """Get temperature at a location and date.

    Args:
        location: The location to get the temperature for, in the format "City, State, Country".
        date: The date to get the temperature for, in the format "Year-Month-Day".
        unit: The unit to return the temperature in. Defaults to "celsius". (choices: ["celsius", "fahrenheit"])

    Returns:
        the temperature, the location, the date and the unit in a dict
    """
    return {
        "temperature": 25.9,
        "location": location,
        "date": date,
        "unit": unit,
    }

def get_function_by_name(name):
    if name == "get_current_temperature":
        return get_current_temperature
    if name == "get_temperature_date":
        return get_temperature_date

tools = [{
    'type': 'function',
    'function': {
        'name': 'get_current_temperature',
        'description': 'Get current temperature at a location.',
        'parameters': {
            'type': 'object',
            'properties': {
                'location': {
                    'type': 'string',
                    'description': "The location to get the temperature for, in the format 'City, State, Country'."
                },
                'unit': {
                    'type': 'string',
                    'enum': [
                        'celsius',
                        'fahrenheit'
                    ],
                    'description': "The unit to return the temperature in. Defaults to 'celsius'."
                }
            },
            'required': [
                'location'
            ]
        }
    }
}, {
    'type': 'function',
    'function': {
        'name': 'get_temperature_date',
        'description': 'Get temperature at a location and date.',
        'parameters': {
            'type': 'object',
            'properties': {
                'location': {
                    'type': 'string',
                    'description': "The location to get the temperature for, in the format 'City, State, Country'."
                },
                'date': {
                    'type': 'string',
                    'description': "The date to get the temperature for, in the format 'Year-Month-Day'."
                },
                'unit': {
                    'type': 'string',
                    'enum': [
                        'celsius',
                        'fahrenheit'
                    ],
                    'description': "The unit to return the temperature in. Defaults to 'celsius'."
                }
            },
            'required': [
                'location',
                'date'
            ]
        }
    }
}]
messages = [{'role': 'user', 'content': "Today is 2024-11-14, What's the temperature in San Francisco now? How about tomorrow?"}]

client = OpenAI(api_key='YOUR_API_KEY', base_url='http://0.0.0.0:23333/v1')
model_name = client.models.list().data[0].id
response = client.chat.completions.create(
    model=model_name,
    messages=messages,
    temperature=0.8,
    top_p=0.8,
    stream=False,
    tools=tools)
print(response.choices[0].message.tool_calls)
messages.append(response.choices[0].message)

for tool_call in response.choices[0].message.tool_calls:
    tool_call_args = json.loads(tool_call.function.arguments)
    tool_call_result =  get_function_by_name(tool_call.function.name)(**tool_call_args)
    messages.append({
        'role': 'tool',
        'name': tool_call.function.name,
        'content': tool_call_result,
        'tool_call_id': tool_call.id
    })

response = client.chat.completions.create(
    model=model_name,
    messages=messages,
    temperature=0.8,
    top_p=0.8,
    stream=False,
    tools=tools)
print(response.choices[0].message.content)

```

--------------------------------

### Install LMDeploy and Download Benchmark Data

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/benchmark/benchmark.md

Installs the lmdeploy package, clones the repository, checks out the correct version tag, and downloads the necessary test dataset for benchmarking.

```shell
pip install lmdeploy
# clone the repo to get the benchmark script
git clone --depth=1 https://github.com/InternLM/lmdeploy
cd lmdeploy
# switch to the tag corresponding to the installed version:
git fetch --tags
# Check the installed lmdeploy version: 
pip show lmdeploy | grep Version
# Then, check out the corresponding tag (replace <version> with the version string):
git checkout <version>
# download the test dataset
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
```

--------------------------------

### Install Qwen2.5-VL Dependencies

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/multi_modal/qwen2_5_vl.md

Install the latest transformers library and qwen-vl-utils with decord support for faster video loading. Ensure transformers version is 4.49.0 or higher.

```shell
# Qwen2.5-VL requires the latest transformers (transformers >= 4.49.0)
pip install git+https://github.com/huggingface/transformers
# It's highly recommended to use `[decord]` feature for faster video loading.
pip install qwen-vl-utils[decord]==0.0.8
```

--------------------------------

### Install InternVL Dependencies

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/multi_modal/internvl.md

Install the 'timm' package and 'flash-attn' for InternVL model compatibility. Ensure the 'flash-attn' whl package matches your environment.

```shell
pip install timm
# It is recommended to find the whl package that matches the environment from the releases on https://github.com/Dao-AILab/flash-attention.
pip install flash-attn
```

--------------------------------

### Successful API Server Logs

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/multi_modal/qwen2_vl.md

Example log output indicating that the LMDeploy API server has started successfully and is accessible.

```text
HINT:    Please open  http://0.0.0.0:23333   in a browser for detailed api usage!!!
HINT:    Please open  http://0.0.0.0:23333   in a browser for detailed api usage!!!
HINT:    Please open  http://0.0.0.0:23333   in a browser for detailed api usage!!!
INFO:     Started server process [2439]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on  http://0.0.0.0:23333  (Press CTRL+C to quit)

```

--------------------------------

### Start api_server with Proxy URL

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/llm/proxy_server.md

Starts an api_server instance and configures it to use the proxy service by specifying the --proxy-url. This allows users to access api_server services through the proxy node.

```shell
lmdeploy serve api_server InternLM/internlm2-chat-1_8b --proxy-url http://0.0.0.0:8000
```

--------------------------------

### Interact with LLM Server using OpenAI Python Package

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/get_started/get_started.md

Example of using the OpenAI Python package to interact with the launched LLM server. Ensure you have `openai` installed and configure the `base_url` correctly.

```python
from openai import OpenAI
client = OpenAI(
    api_key='YOUR_API_KEY',
    base_url="http://0.0.0.0:23333/v1"
)
model_name = client.models.list().data[0].id
response = client.chat.completions.create(
  model=model_name,
  messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": " provide three suggestions about time management"},
  ],
    temperature=0.8,
    top_p=0.8
)
print(response)
```

--------------------------------

### Install LMDeploy and OpenCompass

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/benchmark/evaluate_with_opencompass.md

Installs the necessary libraries for model evaluation. It's recommended to use separate virtual environments.

```shell
pip install lmdeploy
pip install "opencompass[full]"

# Download the lmdeploy source code, which will be used in subsequent steps to access eval script and configuration
git clone --depth=1 https://github.com/InternLM/lmdeploy.git
```

--------------------------------

### Launch API Server with CLI

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/multi_modal/qwen2_vl.md

Start the LMDeploy API server directly from the command line for the Qwen2-VL model.

```shell
lmdeploy serve api_server Qwen/Qwen2-VL-2B-Instruct
```

--------------------------------

### Text Input Example

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/multi_modal/multimodal_inputs.md

Demonstrates sending a simple text-based query to the LMDeploy API server. Ensure the API server is running.

```python
from openai import OpenAI

client = OpenAI(api_key='EMPTY', base_url='http://localhost:23333/v1')
model_name = client.models.list().data[0].id

response = client.chat.completions.create(
    model=model_name,
    messages=[
        {
            'role': 'user',
            'content': [
                {
                    'type': 'text',
                    'text': 'Who are you?',
                }
            ],
        }
    ],
    temperature=0.8,
    top_p=0.8,
)
print(response.choices[0].message.content)
```

--------------------------------

### Install Python FFI Targets

Source: https://github.com/internlm/lmdeploy/blob/main/CMakeLists.txt

Installs the '_turbomind' and '_xgrammar' targets for the Python FFI. The installation destination depends on whether the build is called from setup.py.

```cmake
if (BUILD_PY_FFI)
  if (CALL_FROM_SETUP_PY)
    install(TARGETS _turbomind DESTINATION ${CMAKE_INSTALL_PREFIX})
    install(TARGETS _xgrammar DESTINATION ${CMAKE_INSTALL_PREFIX})
  else()
    install(TARGETS _turbomind DESTINATION ${CMAKE_SOURCE_DIR}/lmdeploy/lib)
    install(TARGETS _xgrammar DESTINATION ${CMAKE_SOURCE_DIR}/lmdeploy/lib)
  endif()
endif ()
```

--------------------------------

### Install Decord for InternLM-XComposer-2.5

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/multi_modal/xcomposer2d5.md

Install the decord package, which is a required dependency for InternLM-XComposer-2.5.

```shell
pip install decord
```

--------------------------------

### Install Flash-Attention 3 for Eagle 3

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/advance/spec_decoding.md

Install the Flash-Attention 3 library from source. This is a prerequisite for using the Eagle 3 speculative decoding method.

```shell
git clone --depth=1 https://github.com/Dao-AILab/flash-attention.git
cd flash-attention/hopper
python setup.py install
```

--------------------------------

### Install FlashMLA for Deepseek MTP

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/advance/spec_decoding.md

Install the FlashMLA library from source. This is required for using the Deepseek MTP speculative decoding method.

```shell
git clone https://github.com/deepseek-ai/FlashMLA.git flash-mla
cd flash-mla
git submodule update --init --recursive
pip install -v .
```

--------------------------------

### TurboMind 1.0 Configuration Example

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/inference/turbomind_config.md

Example of a complete config.ini file for a llama-2-7b-chat model using TurboMind 1.0. This includes model attributes and inference parameters.

```toml
[llama]
model_name = "llama2"
tensor_para_size = 1
head_num = 32
kv_head_num = 32
vocab_size = 32000
num_layer = 32
inter_size = 11008
norm_eps = 1e-06
attn_bias = 0
start_id = 1
end_id = 2
session_len = 4104
weight_type = "fp16"
rotary_embedding = 128
rope_theta = 10000.0
size_per_head = 128
group_size = 0
max_batch_size = 32
max_context_token_num = 4
step_length = 1
cache_max_entry_count = 48
cache_chunk_size = 1
use_context_fmha = 1
quant_policy = 0
max_position_embeddings = 2048
use_dynamic_ntk = 0
use_logn_attn = 0

```

--------------------------------

### Install flash-attn for Qwen Quantization

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/faq.md

Quantizing 'qwen' models requires the 'flash-attn' library. If you encounter 'ModuleNotFoundError: No module named 'flash_attn'', install it manually.

```shell
pip install flash-attn
```

--------------------------------

### Build lmdeploy Wheel

Source: https://github.com/internlm/lmdeploy/blob/main/builder/windows/README.md

Installs the build package and then builds the lmdeploy wheel. Ensure all prerequisites are met before running.

```powershell
pip install build
python -m build --wheel
```

--------------------------------

### TurboMind 2.x Llama2 Configuration Example

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/inference/turbomind_config.md

Example configuration for a Llama2 model using TurboMind 2.x. This includes model attributes and inference parameters.

```toml
[llama]
model_name = "llama2"
tensor_para_size = 1
head_num = 32
kv_head_num = 32
vocab_size = 32000
num_layer = 32
inter_size = 11008
norm_eps = 1e-06
attn_bias = 0
start_id = 1
end_id = 2
session_len = 4104
weight_type = "fp16"
rotary_embedding = 128
rope_theta = 10000.0
size_per_head = 128
group_size = 0
max_batch_size = 64
max_context_token_num = 1
step_length = 1
cache_max_entry_count = 0.5
cache_block_seq_len = 128
cache_chunk_size = 1
enable_prefix_caching = false
quant_policy = 0
max_position_embeddings = 2048
rope_scaling_factor = 0.0
use_logn_attn = 0

```

--------------------------------

### Launch API Server with CLI

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/multi_modal/minicpmv.md

Start the LMDeploy API server directly from the command line, specifying the model to serve.

```shell
lmdeploy serve api_server openbmb/MiniCPM-V-2_6
```

--------------------------------

### KV Cache Block Memory Example (Llama2-7B)

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/inference/turbomind_config.md

Example calculation of k/v block memory for a Llama2-7B model using 'half' precision.

```text
128 * 32 * 32 * 128 * 2 * sizeof(half) = 64MB

```

--------------------------------

### Start LMDeploy Server with Metrics Enabled (DP=1)

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/advance/metrics.md

Use this command to start your LMDeploy API server with the metrics system enabled for single-process deployments. The metrics endpoint will be available at the default port.

```bash
lmdeploy serve api_server Qwen/Qwen2.5-7B-Instruct --enable-metrics
```

--------------------------------

### Start Online Serving with Quantized Model

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/quantization/llm_compressor.md

Encapsulate a quantized model as a service using LMDeploy's api_server. This command starts the service with the TurboMind backend.

```shell
lmdeploy serve api_server ./qwen3_30b_a3b_4bit --backend turbomind
```

--------------------------------

### Start API Server with Custom Reasoning Parser

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/llm/api_server_reasoning.md

After defining a custom parser, use this command to start the LMDeploy API server, enabling the custom reasoning parser by specifying its registered name with the `--reasoning-parser` argument.

```bash
lmdeploy serve api_server $model_path --reasoning-parser example
```

--------------------------------

### Start API Server with Reasoning Parser

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/llm/api_server_reasoning.md

Use this command to start the LMDeploy API server for models that support reasoning, specifying the appropriate reasoning parser. This enables the server to process and expose reasoning content.

```bash
lmdeploy serve api_server deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B --reasoning-parser deepseek-r1
```

--------------------------------

### Start API Server for InternLM2

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/advance/structed_output.md

Start the lmdeploy API server with the PyTorch backend for the specified model. This is a prerequisite for using the API server client.

```shell
lmdeploy serve api_server internlm/internlm2-chat-1_8b --backend pytorch
```

--------------------------------

### Start Evaluation with VLMEvalKit

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/benchmark/evaluate_with_vlmevalkit.md

Run the evaluation script in VLMEvalKit. Ensure the `--model` parameter matches the `<task_name>` defined in your configuration.

```shell
cd VLMEvalKit
python run.py --data OCRBench --model <task_name> --api-nproc 16 --reuse --verbose
```

--------------------------------

### Python Example: Wolfram Alpha Tool Call

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/llm/api_server_tools.md

Demonstrates how to use the 'Wolfram Alpha' tool with Llama 3.1 via LMDeploy's API server. Requires a Wolfram Alpha API key and the OpenAI Python client. The example shows a two-turn conversation where the model first generates a query and then processes the Wolfram Alpha API response.

```python
from openai import OpenAI
import requests


def request_llama3_1_service(messages):
    client = OpenAI(api_key='YOUR_API_KEY',
                    base_url='http://0.0.0.0:23333/v1')
    model_name = client.models.list().data[0].id
    response = client.chat.completions.create(
        model=model_name,
        messages=messages,
        temperature=0.8,
        top_p=0.8,
        stream=False)
    return response.choices[0].message.content


# The role of "system" MUST be specified, including the required tools
messages = [
    {
        "role": "system",
        "content": "Environment: ipython\nTools: wolfram_alpha\n\n Cutting Knowledge Date: December 2023\nToday Date: 23 Jul 2024\n\nYou are a helpful Assistant." # noqa
    },
    {
        "role": "user",
        "content": "Can you help me solve this equation: x^3 - 4x^2 + 6x - 24 = 0"  # noqa
    }
]

# send request to the api_server of llama3.1-70b and get the response
# the "assistant_response" is supposed to be:
# <|python_tag|>wolfram_alpha.call(query="solve x^3 - 4x^2 + 6x - 24 = 0")
assistant_response = request_llama3_1_service(messages)
print(assistant_response)

# Call the API of Wolfram Alpha with the query generated by the model
app_id = 'YOUR-Wolfram-Alpha-API-KEY'
params = {
    "input": assistant_response,
    "appid": app_id,
    "format": "plaintext",
    "output": "json",
}

wolframalpha_response = requests.get(
    "https://api.wolframalpha.com/v2/query",
    params=params
)
wolframalpha_response = wolframalpha_response.json()

# Append the contents obtained by the model and the wolframalpha's API
# to "messages", and send it again to the api_server
messages += [
    {
        "role": "assistant",
        "content": assistant_response
    },
    {
        "role": "ipython",
        "content": wolframalpha_response
    }
]

assistant_response = request_llama3_1_service(messages)
print(assistant_response)
```

--------------------------------

### List Served Models (cURL)

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/llm/api_server.md

Example of how to list all served models using cURL.

```APIDOC
## GET /v1/models

### Description
Retrieves a list of all models currently served by the API server.

### Method
GET

### Endpoint
/v1/models

### Response
#### Success Response (200)
- **(response body)** - A JSON object containing a list of available models.
```

--------------------------------

### Start LMDeploy Proxy Server (DP > 1)

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/advance/metrics.md

This command starts the proxy server for distributed deployments (DP > 1). Configure the server port, routing strategy, and serving strategy as needed.

```bash
# Proxy server
lmdeploy serve proxy --server-port 8000 --routing-strategy 'min_expected_latency' --serving-strategy Hybrid --log-level INFO
```

--------------------------------

### Single Image Input Example

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/multi_modal/multimodal_inputs.md

Shows how to send a request with a single image and a text prompt. The image is referenced via a URL.

```python
from openai import OpenAI

client = OpenAI(api_key='EMPTY', base_url='http://localhost:23333/v1')
model_name = client.models.list().data[0].id

response = client.chat.completions.create(
    model=model_name,
    messages=[
        {
            'role': 'user',
            'content': [
                {
                    'type': 'image_url',
                    'image_url': {
                        'url': 'https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg',
                    },
                },
                {
                    'type': 'text',
                    'text': 'Describe this image.',
                },
            ],
        }
    ],
    temperature=0.8,
    top_p=0.8,
)
print(response.choices[0].message.content)
```

--------------------------------

### Get API Server Help

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/multi_modal/minicpmv.md

Display the help message for the `lmdeploy serve api_server` command to review available arguments and options.

```shell
lmdeploy serve api_server -h
```

--------------------------------

### Local Image File Input Example

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/multi_modal/multimodal_inputs.md

Shows how to use a local image file as input using the `file://` scheme. Ensure the path is correctly specified.

```python
from openai import OpenAI

client = OpenAI(api_key='EMPTY', base_url='http://localhost:23333/v1')
model_name = client.models.list().data[0].id

response = client.chat.completions.create(
    model=model_name,
    messages=[
        {
            'role': 'user',
            'content': [
                {
                    'type': 'image_url',
                    'image_url': {
                        'url': 'file:///path/to/your/image.jpg',
                    },
                },
                {'type': 'text', 'text': 'Describe this image.'},
            ],
        }
    ],
)
print(response.choices[0].message.content)
```

--------------------------------

### Get glibc and python3 versions

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/advance/debug_turbomind.md

Obtain the installed versions of glibc and python3 on your system. This information is crucial for downloading the correct debug symbols.

```bash
rpm -qa | grep glibc

```

```bash
rpm -qa | grep python3

```

--------------------------------

### Pipeline API for Structured Output

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/advance/structed_output.md

Use the pipeline API with a GenerationConfig specifying a JSON schema for guided decoding. Ensure the lmdeploy library is installed.

```python
from lmdeploy import pipeline
from lmdeploy.messages import GenerationConfig, PytorchEngineConfig

model = 'internlm/internlm2-chat-1_8b'
guide = {
    'type': 'object',
    'properties': {
        'name': {
            'type': 'string'
        },
        'skills': {
            'type': 'array',
            'items': {
                'type': 'string',
                'maxLength': 10
            },
            'minItems': 3
        },
        'work history': {
            'type': 'array',
            'items': {
                'type': 'object',
                'properties': {
                    'company': {
                        'type': 'string'
                    },
                    'duration': {
                        'type': 'string'
                    }
                },
                'required': ['company']
            }
        }
    },
    'required': ['name', 'skills', 'work history']
}
pipe = pipeline(model, backend_config=PytorchEngineConfig(), log_level='INFO')
gen_config = GenerationConfig(
    response_format=dict(type='json_schema', json_schema=dict(name='test', schema=guide)))
response = pipe(['Make a self introduction please.'], gen_config=gen_config)
print(response)
```

--------------------------------

### Completions API (Python Client)

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/llm/api_server.md

Example of how to use the `/v1/completions` endpoint with the Python APIClient.

```APIDOC
## POST /v1/completions

### Description
Generates text completions for a given model and prompt.

### Method
POST

### Endpoint
/v1/completions

### Parameters
#### Request Body
- **model** (string) - Required - The name of the model to use.
- **prompt** (string) - Required - The prompt to generate completions for.

### Request Example
```python
from lmdeploy.serve.openai.api_client import APIClient
api_client = APIClient('http://{server_ip}:{server_port}')
model_name = api_client.available_models[0]
for item in api_client.completions_v1(model=model_name, prompt='hi'):
    print(item)
```

### Response
#### Success Response (200)
- **item** (object) - Each item represents a chunk of the streamed response.
```

--------------------------------

### Start LMDeploy API Server

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/intergration/claude_code.md

Launch an LMDeploy API server with a specified model. For tool calling, include a tool parser.

```bash
lmdeploy serve api_server Qwen/Qwen3.5-35B-A3B --backend pytorch --server-port 23333
```

```bash
lmdeploy serve api_server <model> \
  --server-port 23333 \
  --tool-call-parser <parser-name>
```

--------------------------------

### Single Round Tool Invocation Example

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/llm/api_server_tools.md

Demonstrates a single round of tool invocation for getting current weather information. Ensure the API server is running before executing.

```python
from openai import OpenAI

tools = [
  {
    "type": "function",
    "function": {
      "name": "get_current_weather",
      "description": "Get the current weather in a given location",
      "parameters": {
        "type": "object",
        "properties": {
          "location": {
            "type": "string",
            "description": "The city and state, e.g. San Francisco, CA",
          },
          "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
        },
        "required": ["location"],
      },
    }
  }
]
messages = [{"role": "user", "content": "What's the weather like in Boston today?"}]

client = OpenAI(api_key='YOUR_API_KEY',base_url='http://0.0.0.0:23333/v1')
model_name = client.models.list().data[0].id
response = client.chat.completions.create(
    model=model_name,
    messages=messages,
    temperature=0.8,
    top_p=0.8,
    stream=False,
    tools=tools)
print(response)
```

--------------------------------

### Integrate with OpenAI Client

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/multi_modal/phi3.md

Example of interacting with the LMDeploy API endpoint using the OpenAI Python package. Ensure the openai package is installed and the base URL is correctly set.

```python
from openai import OpenAI

client = OpenAI(api_key='YOUR_API_KEY', base_url='http://0.0.0.0:23333/v1')
model_name = client.models.list().data[0].id
response = client.chat.completions.create(
    model=model_name,
    messages=[
        {
        'role':
        'user',
        'content': [
            {
            'type': 'text',
            'text': 'Describe the image please',
            },
            {
            'type': 'image_url',
            'image_url': {
                'url':
                'https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg',
            },
            },
        ],
        }
    ],
    temperature=0.8,
    top_p=0.8)
print(response)
```

--------------------------------

### Get Node Status via curl

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/llm/proxy_server.md

Retrieves the status of all registered api_server service nodes connected to the proxy. This is useful for monitoring the health and availability of your distributed inference setup.

```shell
curl -X 'GET' \
  'http://localhost:8000/nodes/status' \
  -H 'accept: application/json'
```

--------------------------------

### Start Monitoring Stack with Docker Compose

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/advance/metrics.md

Execute this command in the lmdeploy/monitoring directory to launch Prometheus and Grafana services in the background.

```bash
docker compose up
```

--------------------------------

### Launch API Server with Docker

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/multi_modal/internvl.md

Run the API server using a pre-built Docker image. Mount the Hugging Face cache and expose the necessary port. Replace <secret> with your actual token.

```shell
docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HUGGING_FACE_HUB_TOKEN=<secret>" \
    -p 23333:23333 \
    --ipc=host \
    openmmlab/lmdeploy:internvl \
    lmdeploy serve api_server OpenGVLab/InternVL2-8B
```

--------------------------------

### Start LMDeploy API Server

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/quantization/w4a16.md

Launch the LMDeploy API server with a specified model and backend. Ensure the model path and format are correct.

```shell
lmdeploy serve api_server ./internlm2_5-7b-chat-4bit --backend turbomind --model-format awq
```

--------------------------------

### Offline Reward Model Inference with Python

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/supported_models/reward_models.md

Perform offline inference to get reward scores for a given input using the lmdeploy pipeline. Ensure transformers and lmdeploy are installed, and the model path is correctly specified.

```python
from transformers import AutoTokenizer
from lmdeploy import pipeline, PytorchEngineConfig

model_path = "internlm/internlm2-1_8b-reward"
chat = [
    {"role": "system", "content": "Please reason step by step, and put your final answer within \boxed{}."},
    {"role": "user", "content": "Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?"},
    {"role": "assistant", "content": "To determine how much Janet makes from selling the duck eggs at the farmers' market, we need to follow these steps:

1. Calculate the total number of eggs laid by the ducks each day.
2. Determine how many eggs Janet eats and bakes for herself each day.
3. Find out how many eggs are left to be sold.
4. Calculate the revenue from selling the remaining eggs at $2 per egg.

Let's start with the first step:

1. Janet's ducks lay 16 eggs per day.

Next, we calculate how many eggs Janet eats and bakes for herself each day:

2. Janet eats 3 eggs for breakfast every morning.
3. Janet bakes 4 eggs for her friends every day.

So, the total number of eggs Janet eats and bakes for herself each day is:
\[ 3 + 4 = 7 \text{ eggs} \]

Now, we find out how many eggs are left to be sold:
\[ 16 - 7 = 9 \text{ eggs} \]

Finally, we calculate the revenue from selling the remaining eggs at $2 per egg:
\[ 9 \times 2 = 18 \text{ dollars} \]

Therefore, Janet makes 18 dollars every day at the farmers' market."
}
]

tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

conversation_str = tokenizer.apply_chat_template(
    chat,
    tokenize=False,
    add_generation_prompt=False
)

input_ids = tokenizer.encode(
    conversation_str,
    add_special_tokens=False
)


if __name__ == '__main__':
    engine_config = PytorchEngineConfig(tp=tp)
    with pipeline(model_path, backend_config=engine_config) as pipe:
        score = pipe.get_reward_score(input_ids)
        print(f'score: {score}')
```

--------------------------------

### Call /v1/messages with Tool Use

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/llm/api_server_anthropic.md

This example demonstrates how to call the /v1/messages endpoint with tool-use capabilities enabled. Configure the 'tools' and 'tool_choice' parameters as needed.

```bash
curl http://{server_ip}:{server_port}/v1/messages \
  -H "content-type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "internlm-chat-7b",
    "max_tokens": 128,
    "messages": [{"role": "user", "content": "Find lmdeploy docs"}],
    "tools": [{
      "name": "search",
      "description": "Search docs",
      "input_schema": {
        "type": "object",
        "properties": {
          "query": {"type": "string"}
        },
        "required": ["query"]
      }
    }],
    "tool_choice": {"type": "auto"}
  }'
```

--------------------------------

### Start Claude Code

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/intergration/claude_code.md

Start Claude Code with the specified model after configuring the settings.

```bash
claude --model Qwen/Qwen3.5-35B-A3B
```

--------------------------------

### Install fast_hadamard_transform

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/quantization/kv_quant.md

Install the optional dependency for best performance with TurboQuant. This package accelerates the quantization process.

```shell
pip install fast_hadamard_transform
```

--------------------------------

### Install LMDeploy via Docker

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/multi_modal/llava.md

Use this command to pull the latest LMDeploy Docker image for installation.

```shell
docker pull openmmlab/lmdeploy:latest
```

--------------------------------

### Multiple Image Input Example

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/multi_modal/multimodal_inputs.md

Demonstrates sending multiple images along with a text prompt for comparison. Both images are provided via URLs.

```python
from openai import OpenAI

client = OpenAI(api_key='EMPTY', base_url='http://localhost:23333/v1')
model_name = client.models.list().data[0].id

response = client.chat.completions.create(
    model=model_name,
    messages=[
        {
            'role': 'user',
            'content': [
                {
                    'type': 'image_url',
                    'image_url': {
                        'url': 'https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg',
                    },
                },
                {
                    'type': 'image_url',
                    'image_url': {
                        'url': 'https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg',
                    },
                },
                {
                    'type': 'text',
                    'text': 'Compare these two images. What are the similarities and differences?',
                },
            ],
        }
    ],
    temperature=0.8,
    top_p=0.8,
)
print(response.choices[0].message.content)
```

--------------------------------

### Start LMDeploy API Server with Distributed Settings (DP > 1)

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/advance/metrics.md

Launch the API server for distributed deployments, specifying tensor parallelism (TP) and data parallelism (DP) settings. Ensure the proxy URL is correctly set.

```bash
# API server
LMDEPLOY_DP_MASTER_ADDR=127.0.0.1 \
LMDEPLOY_DP_MASTER_PORT=29555 \
lmdeploy serve api_server \
    Qwen/Qwen2.5-7B-Instruct \
    --backend pytorch \
    --tp 2 \
    --dp 2 \
    --proxy-url http://0.0.0.0:8000 \
    --nnodes 1 \
    --node-rank 0 \
    --enable-metrics
```

--------------------------------

### Install LMDeploy

Source: https://github.com/internlm/lmdeploy/blob/main/README.md

Install LMDeploy using pip in a conda environment. Ensure Python version compatibility (3.10-3.13).

```shell
conda create -n lmdeploy python=3.12 -y
conda activate lmdeploy
pip install lmdeploy
```

--------------------------------

### Benchmark Guided Decoding (Regex Schema)

Source: https://github.com/internlm/lmdeploy/blob/main/benchmark/README.md

Measure the overhead of guided decoding with a regex schema and a specific pattern.

```bash
python3 benchmark_guided.py \
    ShareGPT_V3_unfiltered_cleaned_split.json \
    Qwen/Qwen2.5-7B-Instruct \
    --response-format regex_schema \
    --regex-schema '[A-Z][a-z]+ lives in [A-Z][a-z]+\. '
```

--------------------------------

### Benchmark Guided Decoding (JSON Object)

Source: https://github.com/internlm/lmdeploy/blob/main/benchmark/README.md

Measure the overhead of guided decoding with JSON object response format.

```bash
python3 benchmark_guided.py \
    ShareGPT_V3_unfiltered_cleaned_split.json \
    Qwen/Qwen2.5-7B-Instruct \
    --response-format json_object
```

--------------------------------

### Generation Configuration Settings

Source: https://github.com/internlm/lmdeploy/blob/main/autotest/configs/README.md

This example demonstrates the `gen_config` block for request and evaluation sampling parameters. It includes settings like temperature, reasoning effort, and top-p.

```yaml
gen_config:
  temperature: 0.6
  reasoning-effort: high
  top-p: 0.95
  top-k: 50
  min-p: 0.0
  chat-template-kwargs:
    enable_thinking: true

```

--------------------------------

### Benchmark Guided Decoding (JSON Schema)

Source: https://github.com/internlm/lmdeploy/blob/main/benchmark/README.md

Measure the overhead of guided decoding with JSON schema, comparing it against a baseline.

```bash
python3 benchmark_guided.py \
    ShareGPT_V3_unfiltered_cleaned_split.json \
    Qwen/Qwen2.5-7B-Instruct \
    --response-format json_schema \
    --concurrency 64
```

--------------------------------

### Launch API Server with Docker

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/multi_modal/qwen2_vl.md

Deploy the LMDeploy API server for Qwen2-VL using a Docker container, mounting Hugging Face cache and exposing the necessary port.

```shell
docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HUGGING_FACE_HUB_TOKEN=<secret>" \
    -p 23333:23333 \
    --ipc=host \
    openmmlab/lmdeploy:qwen2vl \
    lmdeploy serve api_server Qwen/Qwen2-VL-2B-Instruct
```

--------------------------------

### Start API Server for Quantized Model

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/quantization/w8a8.md

Launch an API server to serve a quantized model. This command makes the model accessible via RESTful APIs compatible with OpenAI's interfaces.

```shell
lmdeploy serve api_server ./internlm2_5-7b-chat-int8 --backend pytorch
```

--------------------------------

### Launch API Server with Docker

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/llm/api_server.md

Deploy an OpenAI-compatible server using the official LMDeploy Docker image. Ensure to replace `<secret>` with your Hugging Face token.

```shell
docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HUGGING_FACE_HUB_TOKEN=<secret>" \
    -p 23333:23333 \
    --ipc=host \
    openmmlab/lmdeploy:latest \
    lmdeploy serve api_server internlm/internlm2_5-7b-chat
```

--------------------------------

### Serve LLM Model with Maca Device (CLI)

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/get_started/maca/get_started.md

Use the `lmdeploy serve api_server` command with the `--device maca` flag to serve a LLM model. This is the command-line interface for starting the API server.

```bash
lmdeploy serve api_server --backend pytorch --device maca internlm/internlm2_5-7b-chat
```

--------------------------------

### Install LMDeploy and Transfer Engine

Source: https://github.com/internlm/lmdeploy/blob/main/lmdeploy/pytorch/disagg/README.md

Install the inference engine with all features and the transfer engine. Ensure lmdeploy is version 0.7.0 or higher.

```shell
pip install lmdeploy[all] >= 0.7.0
pip install dlslime>=0.0.2
```

--------------------------------

### Basic Pipeline Initialization and Inference

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/llm/pipeline.md

Initializes a pipeline with a default configuration and performs inference on a list of prompts.

```python
from lmdeploy import pipeline

pipe = pipeline('internlm/internlm2_5-7b-chat')
response = pipe(['Hi, pls intro yourself', 'Shanghai is'])
print(response)
```

--------------------------------

### Start API Server with Docker Compose

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/multi_modal/llava.md

After configuring `docker-compose.yml`, use this command to start the LMDeploy API server in detached mode.

```shell
docker-compose up -d
```

--------------------------------

### Install RDMA Development Libraries

Source: https://github.com/internlm/lmdeploy/blob/main/lmdeploy/pytorch/disagg/README.md

Install necessary development libraries for RDMA on Ubuntu or CentOS systems. These are required for RDMA connection troubleshooting.

```shell
# on Ubuntu
sudo apt install libibverbs-dev
# on CentOS
sudo yum install ibverbs-devel
```

--------------------------------

### Basic Image Description with VLM Pipeline

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/multi_modal/internvl.md

Demonstrates the fundamental usage of the VLM pipeline for describing a single image. Ensure the 'lmdeploy' library is installed.

```python
from lmdeploy import pipeline
from lmdeploy.vl import load_image

pipe = pipeline('OpenGVLab/InternVL2-8B')

image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')
response = pipe((f'describe this image', image))
print(response)
```

--------------------------------

### Install Specific LMDeploy Version from Zip Archive

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/get_started/installation.md

Installs a specific version of LMDeploy by providing a URL to a zip archive of a tagged release.

```shell
pip install https://github.com/InternLM/lmdeploy/archive/refs/tags/v0.11.0.zip
```

--------------------------------

### Create Docker Container

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/advance/pytorch_multinodes.md

Run this command on each node to create a consistent environment using Docker. Ensure the model path is mounted correctly.

```bash
docker run -it \
    --network host \
    -v $MODEL_PATH:$CONTAINER_MODEL_PATH \
    openmmlab/lmdeploy:latest
```

--------------------------------

### Benchmark Guided Decoding (Custom JSON Schema File)

Source: https://github.com/internlm/lmdeploy/blob/main/benchmark/README.md

Measure the overhead of guided decoding using a custom JSON schema file.

```bash
python3 benchmark_guided.py \
    ShareGPT_V3_unfiltered_cleaned_split.json \
    Qwen/Qwen2.5-7B-Instruct \
    --response-format json_schema \
    --json-schema-path my_schema.json
```

--------------------------------

### Install DeepSeek-VL2 Dependencies

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/multi_modal/deepseek_vl2.md

Install the official DeepSeek-VL2 GitHub repository and required third-party libraries. Note the specific version constraint for the transformers library.

```bash
pip install git+https://github.com/deepseek-ai/DeepSeek-VL2.git --no-deps
pip install attrdict timm 'transformers<4.48.0'
```

--------------------------------

### Serve an LLM Model

Source: https://github.com/internlm/lmdeploy/blob/main/docs/en/get_started/get_started.md

Launches an OpenAI-compatible server for LLM models. Specify a custom port using `--server-port`. Options align with engine configuration.

```shell
lmdeploy serve api_server internlm/internlm2_5-7b-chat
```