### Install Kimi-VL Dependencies

Source: https://context7.com/moonshotai/kimi-vl/llms.txt

Set up the Python environment by creating a conda environment and installing core dependencies including PyTorch, Transformers, Pillow, Tiktoken, Accelerate, Blobfile, and OpenAI. Optionally, install flash-attn for memory optimization.

```bash
conda create -n kimi-vl python=3.10 -y
conda activate kimi-vl

pip install torch==2.5.1 torchvision==0.20.1 transformers==4.51.3
pip install pillow tiktoken accelerate blobfile openai

pip install flash-attn --no-build-isolation
```

--------------------------------

### Setup Kimi-VL Environment

Source: https://github.com/moonshotai/kimi-vl/blob/main/README.md

Create and activate a conda environment for Kimi-VL and install dependencies.

```bash
conda create -n kimi-vl python=3.10 -y
conda activate kimi-vl
pip install -r requirements.txt
```

--------------------------------

### Install Flash Attention

Source: https://github.com/moonshotai/kimi-vl/blob/main/README.md

Install flash-attn for improved memory usage and inference speed. This is recommended if you encounter Out-of-Memory errors.

```bash
pip install flash-attn --no-build-isolation
```

--------------------------------

### Load Kimi-VL-A3B-Instruct with Flash Attention

Source: https://github.com/moonshotai/kimi-vl/blob/main/README.md

Load the Kimi-VL-A3B-Instruct model with flash attention enabled for optimized performance. This requires flash-attn to be installed.

```python
# If flash-attn has been installed, it is recommended to set torch_dtype=torch.bfloat16 and attn_implementation="flash_attention_2"
# to save memory and speed up inference
# model = AutoModelForCausalLM.from_pretrained(
#     model_path,
#     torch_dtype=torch.bfloat16,
#     device_map="auto",
#     trust_remote_code=True,
#     attn_implementation="flash_attention_2"
# )
```

--------------------------------

### Deploy Kimi-VL with vLLM OpenAI-Compatible Server

Source: https://context7.com/moonshotai/kimi-vl/llms.txt

Starts a vLLM server providing an OpenAI-compatible API endpoint for Kimi-VL models. Configuration options include model name, tensor parallelism, token limits, and multimodal prompt limits.

```bash
# Deploy Kimi-VL-Thinking-2506 as OpenAI-compatible server
vllm serve moonshotai/Kimi-VL-A3B-Thinking-2506 \
    --served-model-name kimi-vl-thinking-2506 \
    --trust-remote-code \
    --tensor-parallel-size 1 \
    --max-num-batched-tokens 32768 \
    --max-model-len 32768 \
    --limit-mm-per-prompt image=64

# Alternative: Deploy Kimi-VL-Instruct for general tasks
vllm serve moonshotai/Kimi-VL-A3B-Instruct \
    --served-model-name kimi-vl \
    --trust-remote-code \
    --tensor-parallel-size 1 \
    --max-num-batched-tokens 32768 \
    --max-model-len 32768 \
    --limit-mm-per-prompt image=64

# Extended context configuration (128K tokens)
vllm serve moonshotai/Kimi-VL-A3B-Thinking-2506 \
    --served-model-name kimi-vl-thinking-2506 \
    --trust-remote-code \
    --tensor-parallel-size 1 \
    --max-num-batched-tokens 131072 \
    --max-model-len 131072 \
    --limit-mm-per-prompt image=256
```

--------------------------------

### Load Model and Process Multiple Images for Kimi-VL-A3B-Thinking-2506

Source: https://github.com/moonshotai/kimi-vl/blob/main/README.md

Loads the Kimi-VL-A3B-Thinking-2506 model and processor, prepares multiple images and a text prompt, and generates a response. This example uses a higher max_new_tokens value and a temperature setting for generation.

```python
import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoProcessor

model_path = "moonshotai/Kimi-VL-A3B-Thinking-2506"
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)
# If flash-attn has been installed, it is recommended to set torch_dtype=torch.bfloat16 and attn_implementation="flash_attention_2"
# to save memory and speed up inference
# model = AutoModelForCausalLM.from_pretrained(
#     model_path,
#     torch_dtype=torch.bfloat16,
#     device_map="auto",
#     trust_remote_code=True,
#     attn_implementation="flash_attention_2"
# )
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)

image_paths = ["./figures/demo1.png", "./figures/demo2.png"]
images = [Image.open(path) for path in image_paths]
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image_path} for image_path in image_paths
        ] + [{"type": "text", "text": "Please infer step by step who this manuscript belongs to and what it records"}],
    },
]
text = processor.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
inputs = processor(images=images, text=text, return_tensors="pt", padding=True, truncation=True).to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=32768, temperature=0.8)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
response = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)[0]
print(response)
```

--------------------------------

### Load Kimi-VL-A3B-Instruct Model

Source: https://github.com/moonshotai/kimi-vl/blob/main/README.md

Load the Kimi-VL-A3B-Instruct model and processor using Hugging Face Transformers. Ensure you have the correct Python, PyTorch, and Transformers versions installed.

```python
import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoProcessor

model_path = "moonshotai/Kimi-VL-A3B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)
```

--------------------------------

### Offline Inference with vLLM

Source: https://github.com/moonshotai/kimi-vl/blob/main/README.md

Performs offline inference using vLLM. This example demonstrates how to load the model and processor, prepare multimodal inputs, and generate text. Refer to vLLM documentation for more advanced usage.

```python
from PIL import Image
from transformers import AutoProcessor
from vllm import LLM, SamplingParams

model_path = "moonshotai/Kimi-VL-A3B-Instruct"  # or "moonshotai/Kimi-VL-A3B-Thinking-2506"
llm = LLM(
    model_path,
    trust_remote_code=True,
)

processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)

image_path = "./figures/demo.png"
image = Image.open(image_path)
messages = [
    {"role": "user", "content": [{"type": "image", "image": image_path}, {"type": "text", "text": "What is the dome building in the picture? Think step by step."}]}
]
text = processor.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
outputs = llm.generate([{"prompt": text, "multi_modal_data": {"image": image}}], sampling_params = SamplingParams(max_tokens=512))

print("-" * 50)
for o in outputs:
    generated_text = o.outputs[0].text
    print(generated_text)
    print("-" * 50)
```

--------------------------------

### Serve Kimi-VL-A3B-Instruct Model with vLLM

Source: https://github.com/moonshotai/kimi-vl/blob/main/README.md

Deploy the Kimi-VL-A3B-Instruct model using vLLM. This command specifies the model name and configuration for serving.

```bash
vllm serve moonshotai/Kimi-VL-A3B-Instruct --served-model-name kimi-vl --trust-remote-code --tensor-parallel-size 1 --max-num-batched-tokens 32768 --max-model-len 32768 --limit-mm-per-prompt image=64
```

--------------------------------

### Serve Kimi-VL with vLLM OpenAI-Compatible Server

Source: https://github.com/moonshotai/kimi-vl/blob/main/README.md

Command to serve Kimi-VL using vLLM's OpenAI-Compatible Server. Options for longer context windows and handling more input images are provided as comments.

```bash
# If you need a longer context window, you can set --max-model-len and --max-num-batched-tokens to 131072
# If you need more input images, you can set --limit-mm-per-prompt image=256 or 512


```

--------------------------------

### Load Model and Process Image for Kimi-VL-A3B-Instruct

Source: https://github.com/moonshotai/kimi-vl/blob/main/README.md

Loads the Kimi-VL-A3B-Instruct model and processor, prepares an image and text prompt, and generates a response. Ensure the image path is correct.

```python
from PIL import Image
from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)

image_path = "./figures/demo.png"
image = Image.open(image_path)
messages = [
    {"role": "user", "content": [{"type": "image", "image": image_path}, {"type": "text", "text": "What is the dome building in the picture? Think step by step."}]}
]
text = processor.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
inputs = processor(images=image, text=text, return_tensors="pt", padding=True, truncation=True).to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=512)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
response = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)[0]
print(response)
```

--------------------------------

### Serve Kimi-VL-A3B-Thinking-2506 Model with vLLM

Source: https://github.com/moonshotai/kimi-vl/blob/main/README.md

Use this command to serve the Kimi-VL-A3B-Thinking-2506 model. Ensure to set appropriate parameters like tensor-parallel-size and token limits.

```bash
vllm serve moonshotai/Kimi-VL-A3B-Thinking-2506 --served-model-name kimi-vl-thinking-2506 --trust-remote-code --tensor-parallel-size 1 --max-num-batched-tokens 32768 --max-model-len 32768 --limit-mm-per-prompt image=64
```

--------------------------------

### Run LLaMA-Factory Fine-tuning Commands

Source: https://context7.com/moonshotai/kimi-vl/llms.txt

Execute LoRA fine-tuning using the specified YAML configuration. For multi-GPU full fine-tuning, use DeepSpeed with the appropriate configuration file.

```bash
# Run single-GPU LoRA fine-tuning (requires ~50GB VRAM)
llamafactory-cli train kimi_vl_lora.yaml
```

```bash
# Run multi-GPU full fine-tuning with DeepSpeed ZeRO-2
deepspeed --num_gpus 8 llamafactory-cli train kimi_vl_full.yaml
```

--------------------------------

### vLLM Offline Inference with Kimi-VL

Source: https://context7.com/moonshotai/kimi-vl/llms.txt

Initializes vLLM with a Kimi-VL model, prepares a single image and prompt, and generates text using vLLM's optimized batch inference capabilities.

```python
from PIL import Image
from transformers import AutoProcessor
from vllm import LLM, SamplingParams

# Initialize vLLM with Kimi-VL
model_path = "moonshotai/Kimi-VL-A3B-Instruct"  # or "moonshotai/Kimi-VL-A3B-Thinking-2506"
llm = LLM(
    model_path,
    trust_remote_code=True,
)
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)

# Prepare image and prompt
image_path = "./figures/demo.png"
image = Image.open(image_path)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image_path},
            {"type": "text", "text": "What is the dome building in the picture? Think step by step."}
        ]
    }
]

# Generate using vLLM
text = processor.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
outputs = llm.generate(
    [{"prompt": text, "multi_modal_data": {"image": image}}],
    sampling_params=SamplingParams(max_tokens=512)
)

# Print results
print("-" * 50)
for o in outputs:
    generated_text = o.outputs[0].text
    print(generated_text)
    print("-" * 50)
```

--------------------------------

### Single Image Inference with Kimi-VL-A3B-Instruct

Source: https://context7.com/moonshotai/kimi-vl/llms.txt

Load the Kimi-VL-A3B-Instruct model and processor for general vision-language tasks. Prepare a single image and a user message, then process and generate a response. Ensure the image path is correct.

```python
import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoProcessor

# Load model and processor
model_path = "moonshotai/Kimi-VL-A3B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)

# Prepare image and message
image_path = "./figures/demo.png"
image = Image.open(image_path)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image_path},
            {"type": "text", "text": "What is the dome building in the picture? Think step by step."}
        ]
    }
]

# Process and generate response
text = processor.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
inputs = processor(images=image, text=text, return_tensors="pt", padding=True, truncation=True).to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=512)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
response = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)[0]
print(response)
```

--------------------------------

### LLaMA-Factory Configuration for Kimi-VL LoRA Fine-tuning

Source: https://context7.com/moonshotai/kimi-vl/llms.txt

Save this configuration as kimi_vl_lora.yaml for LoRA fine-tuning. It specifies model paths, LoRA parameters, training settings, and dataset configuration. Optimized for single-GPU training with ~50GB VRAM.

```yaml
model_name_or_path: moonshotai/Kimi-VL-A3B-Instruct
trust_remote_code: true

# LoRA configuration
finetuning_type: lora
lora_target: all
lora_rank: 8
lora_alpha: 16
lora_dropout: 0.1

# Training parameters
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 1.0e-4
num_train_epochs: 3
max_length: 4096

# Optimization for 50GB VRAM single-GPU training
bf16: true
gradient_checkpointing: true

# Dataset configuration
dataset: your_multimodal_dataset
template: kimi_vl

# Output
output_dir: ./output/kimi_vl_lora
```

--------------------------------

### Optimized Single Image Inference with Flash Attention

Source: https://context7.com/moonshotai/kimi-vl/llms.txt

Utilize Flash Attention for reduced memory usage and faster inference with the Kimi-VL-A3B-Instruct model. Load the model with `attn_implementation='flash_attention_2'` and set `torch_dtype` to `torch.bfloat16`. Generate responses with a recommended temperature for the Instruct model.

```python
import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoProcessor

model_path = "moonshotai/Kimi-VL-A3B-Instruct"

# Load with flash attention for optimized inference
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
    attn_implementation="flash_attention_2"
)
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)

# Prepare inputs
image_path = "./figures/demo.png"
image = Image.open(image_path)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image_path},
            {"type": "text", "text": "Describe this image in detail."}
        ]
    }
]

# Generate with recommended temperature for Instruct model
text = processor.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
inputs = processor(images=image, text=text, return_tensors="pt", padding=True, truncation=True).to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=512, temperature=0.2)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
response = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)[0]
print(response)
```

--------------------------------

### OpenAI Python Client API Integration with Kimi-VL

Source: https://context7.com/moonshotai/kimi-vl/llms.txt

Connects to a locally deployed Kimi-VL vLLM server using the OpenAI Python client. It demonstrates loading, encoding an image to base64, building a multimodal message, and calling the chat completions API.

```python
import base64
from PIL import Image
from io import BytesIO
from openai import OpenAI

# Initialize OpenAI client pointing to local vLLM server
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="token-abc123",
)

# Load and encode image as base64
image_path = "./figures/demo.png"
image = Image.open(image_path).convert("RGB")
buffered = BytesIO()
image.save(buffered, format="JPEG")
img_b64_str = base64.b64encode(buffered.getvalue()).decode("utf-8")
base64_image_url = f"data:image/jpeg;base64,{img_b64_str}"

# Build multimodal message with image and text
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": base64_image_url}},
            {"type": "text", "text": "What is the dome building in the picture? Think step by step."}
        ]
    }
]

# Call the API
completion = client.chat.completions.create(
    model="kimi-vl-thinking-2506",  # or "kimi-vl" for Instruct model
    messages=messages
)

print(completion.choices[0].message)
```

--------------------------------

### Call Kimi-VL API with Image and Text Prompt

Source: https://github.com/moonshotai/kimi-vl/blob/main/README.md

This Python script demonstrates how to interact with the Kimi-VL API. It includes base64 encoding of an image and constructing a messages payload for a chat completion request.

```python
import base64
from PIL import Image
from io import BytesIO
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="token-abc123",
)

image_path = "./figures/demo.png"
image = Image.open(image_path).convert("RGB")

buffered = BytesIO()
image.save(buffered, format="JPEG")
img_b64_str = base64.b64encode(buffered.getvalue()).decode("utf-8")
base64_image_url = f"data:image/jpeg;base64,{img_b64_str}"

messages = [
    {"role": "user", "content": [{"type": "image_url", "image_url": {"url": base64_image_url}}, {"type": "text", "text": "What is the dome building in the picture? Think step by step."}]}
]

completion = client.chat.completions.create(
  model="kimi-vl-thinking-2506", # or kimi-vl
  messages=messages
)

print(completion.choices[0].message)
```

--------------------------------

### Load Kimi-VL Thinking Model and Perform Multi-Image Inference

Source: https://context7.com/moonshotai/kimi-vl/llms.txt

Loads the Kimi-VL Thinking model and processor, prepares multiple images, builds a multi-image message, and generates a response with an extended token limit.

```python
model_path = "moonshotai/Kimi-VL-A3B-Thinking-2506"
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)

# Load multiple images
image_paths = ["./figures/demo1.png", "./figures/demo2.png"]
images = [Image.open(path) for path in image_paths]

# Build multi-image message
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image_path} for image_path in image_paths
        ] + [{"type": "text", "text": "Please infer step by step who this manuscript belongs to and what it records"}],
    },
]

# Generate with extended token limit and recommended temperature for Thinking model
text = processor.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
inputs = processor(images=images, text=text, return_tensors="pt", padding=True, truncation=True).to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=32768, temperature=0.8)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
response = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)[0]
print(response)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.