### Serving OvisOCR2 with vLLM Source: https://huggingface.co/ATH-MaaS/OvisOCR2/blame/65c619d374b55d4152e85150fc1b003700bc1f0c/model.safetensors Instructions for installing vLLM, starting the server, and querying it via an OpenAI-compatible API. ```bash # Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "ATH-MaaS/OvisOCR2" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ATH-MaaS/OvisOCR2", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' ``` -------------------------------- ### Install vLLM Source: https://huggingface.co/ATH-MaaS/OvisOCR2?local-app=vllm Install the vLLM package via pip. ```bash # Install vLLM from pip: pip install vllm ``` -------------------------------- ### Start vLLM Server Source: https://huggingface.co/ATH-MaaS/OvisOCR2?local-app=vllm Launch the vLLM server for the OvisOCR2 model. ```bash # Start the vLLM server: vllm serve "ATH-MaaS/OvisOCR2" ``` -------------------------------- ### Install SGLang Source: https://huggingface.co/ATH-MaaS/OvisOCR2?local-app=sglang Install the SGLang package using pip. ```bash # Install SGLang from pip: pip install sglang ``` -------------------------------- ### Launch SGLang Server Source: https://huggingface.co/ATH-MaaS/OvisOCR2?local-app=sglang Start the SGLang server for the OvisOCR2 model on a specified port. ```bash # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "ATH-MaaS/OvisOCR2" \ --host 0.0.0.0 \ --port 30000 ``` -------------------------------- ### Install Transformers Library Source: https://huggingface.co/ATH-MaaS/OvisOCR2/colab Install or upgrade the transformers library to ensure compatibility with the model. ```bash !pip install -U transformers ``` -------------------------------- ### Serving OvisOCR2 with SGLang Source: https://huggingface.co/ATH-MaaS/OvisOCR2/blame/65c619d374b55d4152e85150fc1b003700bc1f0c/model.safetensors Setup and execution commands for SGLang, including local server launch and Docker container deployment. ```bash # Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "ATH-MaaS/OvisOCR2" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ATH-MaaS/OvisOCR2", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' ``` ```bash docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "ATH-MaaS/OvisOCR2" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ATH-MaaS/OvisOCR2", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' ``` -------------------------------- ### Install Inference Dependencies Source: https://huggingface.co/ATH-MaaS/OvisOCR2/blame/65c619d374b55d4152e85150fc1b003700bc1f0c/README.md Install the required packages for running inference with OvisOCR2. ```bash pip install "vllm==0.22.1" pillow ``` -------------------------------- ### Parse document with OvisOCR2 Source: https://huggingface.co/ATH-MaaS/OvisOCR2/blob/d844a45160ff88ed5d51e8c13d7484e50fd2f801/README.md Basic usage example for initializing the OvisOCR2 parser and processing an image. ```python parser = OvisOCR2Parser("ATH-MaaS/OvisOCR2") page_image = Image.open("test1.jpg") markdown = parser.parse([page_image], filter_imgtags=False)[0] save_renderable_markdown_with_visual_regions(markdown, page_image, "output") ``` -------------------------------- ### Parse images and save markdown with visual regions Source: https://huggingface.co/ATH-MaaS/OvisOCR2/blob/65c619d374b55d4152e85150fc1b003700bc1f0c/README.md Defines a function to extract visual regions from markdown and save them as images, followed by an example of initializing the OvisOCR2 parser and processing an image. ```python def save_renderable_markdown_with_visual_regions( markdown: str, page_image: Image.Image, output_dir: str, ) -> None: output_dir = Path(output_dir) images_dir = output_dir / "images" images_dir.mkdir(parents=True, exist_ok=True) width, height = page_image.size for left, top, right, bottom in BBOX_IMAGE_PATTERN.findall(markdown): x1 = max(0, min(width, round(int(left) * width / 1000))) y1 = max(0, min(height, round(int(top) * height / 1000))) x2 = max(0, min(width, round(int(right) * width / 1000))) y2 = max(0, min(height, round(int(bottom) * height / 1000))) if x2 <= x1 or y2 <= y1: continue crop_path = images_dir / f"bbox_{left}_{top}_{right}_{bottom}.jpg" page_image.crop((x1, y1, x2, y2)).convert("RGB").save(crop_path) (output_dir / "output.md").write_text(markdown, encoding="utf-8") parser = OvisOCR2Parser("ATH-MaaS/OvisOCR2") page_image = Image.open("test1.jpg") markdown = parser.parse([page_image], filter_imgtags=False)[0] save_renderable_markdown_with_visual_regions(markdown, page_image, "output") ``` -------------------------------- ### Run OvisOCR2 with Docker Source: https://huggingface.co/ATH-MaaS/OvisOCR2/blame/77bfe9462d1e6f8965ee6698f08ea8ede580912c/.gitattributes Launch the OvisOCR2 model server using the sglang Docker image. ```bash docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "ATH-MaaS/OvisOCR2" \ --host 0.0.0.0 \ --port 30000 ``` -------------------------------- ### Using OvisOCR2 with Transformers Source: https://huggingface.co/ATH-MaaS/OvisOCR2/blame/65c619d374b55d4152e85150fc1b003700bc1f0c/model.safetensors Demonstrates high-level pipeline usage and direct model loading with AutoProcessor and AutoModelForMultimodalLM. ```python # Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="ATH-MaaS/OvisOCR2") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages) ``` ```python # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("ATH-MaaS/OvisOCR2") model = AutoModelForMultimodalLM.from_pretrained("ATH-MaaS/OvisOCR2") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) ``` -------------------------------- ### Preprocessor Configuration (preprocessor_config.json) Source: https://huggingface.co/ATH-MaaS/OvisOCR2/commit/77bfe9462d1e6f8965ee6698f08ea8ede580912c Configuration for image processing, including patch sizes and normalization parameters. ```json { "size": { "longest_edge": 16777216, "shortest_edge": 65536 }, "patch_size": 16, "temporal_patch_size": 2, "merge_size": 2, "image_mean": [ 0.5, 0.5, 0.5 ], "image_std": [ 0.5, 0.5, 0.5 ], "processor_class": "Qwen3VLProcessor", "image_processor_type": "Qwen2VLImageProcessorFast" } ``` -------------------------------- ### Model Configuration (config.json) Source: https://huggingface.co/ATH-MaaS/OvisOCR2/commit/77bfe9462d1e6f8965ee6698f08ea8ede580912c Defines the architecture, text configuration, and vision configuration for the Qwen3_5-based model. ```json { "architectures": [ "Qwen3_5ForConditionalGeneration" ], "image_token_id": 248056, "model_type": "qwen3_5", "text_config": { "attention_bias": false, "attention_dropout": 0.0, "attn_output_gate": true, "dtype": "bfloat16", "eos_token_id": 248044, "full_attention_interval": 4, "head_dim": 256, "hidden_act": "silu", "hidden_size": 1024, "initializer_range": 0.02, "intermediate_size": 3584, "layer_types": [ "linear_attention", "linear_attention", "linear_attention", "full_attention", "linear_attention", "linear_attention", "linear_attention", "full_attention", "linear_attention", "linear_attention", "linear_attention", "full_attention", "linear_attention", "linear_attention", "linear_attention", "full_attention", "linear_attention", "linear_attention", "linear_attention", "full_attention", "linear_attention", "linear_attention", "linear_attention", "full_attention" ], "linear_conv_kernel_dim": 4, "linear_key_head_dim": 128, "linear_num_key_heads": 16, "linear_num_value_heads": 16, "linear_value_head_dim": 128, "max_position_embeddings": 262144, "mlp_only_layers": [], "model_type": "qwen3_5_text", "mtp_num_hidden_layers": 1, "mtp_use_dedicated_embeddings": false, "num_attention_heads": 8, "num_hidden_layers": 24, "num_key_value_heads": 2, "rms_norm_eps": 1e-06, "tie_word_embeddings": true, "use_cache": true, "vocab_size": 248320, "mamba_ssm_dtype": "float32", "rope_parameters": { "mrope_interleaved": true, "mrope_section": [ 11, 11, 10 ], "rope_type": "default", "rope_theta": 10000000, "partial_rotary_factor": 0.25 } }, "tie_word_embeddings": true, "transformers_version": "4.57.0.dev0", "video_token_id": 248057, "vision_config": { "deepstack_visual_indexes": [], "depth": 12, "hidden_act": "gelu_pytorch_tanh", "hidden_size": 768, "in_channels": 3, "initializer_range": 0.02, "intermediate_size": 3072, "model_type": "qwen3_5", "num_heads": 12, "num_position_embeddings": 2304, "patch_size": 16, "spatial_merge_size": 2, "temporal_patch_size": 2 }, "vision_end_token_id": 248054, "vision_start_token_id": 248053 } ``` -------------------------------- ### Preprocessor Configuration Source: https://huggingface.co/ATH-MaaS/OvisOCR2/blob/65c619d374b55d4152e85150fc1b003700bc1f0c/preprocessor_config.json Configuration settings for the OvisOCR2 preprocessor. ```json { "size": { "longest_edge": 16777216, "shortest_edge": 65536 }, "patch_size": 16, "temporal_patch_size": 2, "merge_size": 2, "image_mean": [ 0.5, 0.5, 0.5 ], "image_std": [ 0.5, 0.5, 0.5 ], "processor_class": "Qwen3VLProcessor", "image_processor_type": "Qwen2VLImageProcessorFast" } ``` -------------------------------- ### OvisOCR2 Configuration Source: https://huggingface.co/ATH-MaaS/OvisOCR2/blob/65c619d374b55d4152e85150fc1b003700bc1f0c/video_preprocessor_config.json JSON configuration for the video preprocessor. ```json { "size": { "longest_edge": 25165824, "shortest_edge": 4096 }, "patch_size": 16, "temporal_patch_size": 2, "merge_size": 2, "image_mean": [ 0.5, 0.5, 0.5 ], "image_std": [ 0.5, 0.5, 0.5 ], "processor_class": "Qwen3VLProcessor", "video_processor_type": "Qwen3VLVideoProcessor" } ``` -------------------------------- ### Video Preprocessor Configuration Source: https://huggingface.co/ATH-MaaS/OvisOCR2/commit/77bfe9462d1e6f8965ee6698f08ea8ede580912c Configuration for video processing, including patch sizes and normalization parameters. ```json { "size": { "longest_edge": 25165824, "shortest_edge": 4096 }, "patch_size": 16, "temporal_patch_size": 2, "merge_size": 2, "image_mean": [ 0.5, 0.5, 0.5 ], "image_std": [ 0.5, 0.5, 0.5 ], "processor_class": "Qwen3VLProcessor", "video_processor_type": "Qwen3VLVideoProcessor" } ``` -------------------------------- ### Query vLLM via OpenAI-compatible API Source: https://huggingface.co/ATH-MaaS/OvisOCR2?local-app=vllm Send a request to the local vLLM server using curl to perform image description. ```bash curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ATH-MaaS/OvisOCR2", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' ``` -------------------------------- ### Tokenizer Configuration JSON Source: https://huggingface.co/ATH-MaaS/OvisOCR2/blame/65c619d374b55d4152e85150fc1b003700bc1f0c/tokenizer_config.json Configuration object defining tokenizer behavior, special tokens, and prompt templates for the Qwen2Tokenizer. ```json { "chat_template": "{% if add_generation_prompt %} {{- '<|im_start|>assistant\n' }} {%- if enable_thinking is defined and enable_thinking is true %} {{- '\n' }} {%- else %} {{- '\n\n\n\n' }} {%- endif %} {%- endif %}", "clean_up_tokenization_spaces": false, "eos_token": "<|im_end|>", "errors": "replace", "model_max_length": 262144, "pad_token": "<|endoftext|>", "split_special_tokens": false, "tokenizer_class": "Qwen2Tokenizer", "unk_token": null, "add_bos_token": false, "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?[\p{L}\p{M}]+|\p{N}| ?[^\s\p{L}\p{M}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+", "extra_special_tokens": { "audio_bos_token": "<|audio_start|>", "audio_eos_token": "<|audio_end|>", "audio_token": "<|audio_pad|>", "image_token": "<|image_pad|>", "video_token": "<|video_pad|>", "vision_bos_token": "<|vision_start|>", "vision_eos_token": "<|vision_end|>" } } ``` -------------------------------- ### Jinja2 Tool Call and Response Formatting Source: https://huggingface.co/ATH-MaaS/OvisOCR2/commit/77bfe9462d1e6f8965ee6698f08ea8ede580912c Template logic for serializing tool calls, parameters, and responses into a specific XML-like format for model consumption. ```jinja2 {%- if tool_call.function is defined %} + {%- set tool_call = tool_call.function %} + {%- endif %} + {%- if loop.first %} + {%- if content|trim %} + {{- '\n\n\n\n' }} + {%- else %} + {{- '\n\n' }} + {%- endif %} + {%- else %} + {{- '\n\n\n' }} + {%- endif %} + {%- if tool_call.arguments is defined %} + {%- for args_name, args_value in tool_call.arguments|items %} + {{- '\n' }} + {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %} + {{- args_value }} + {{- '\n\n' }} + {%- endfor %} + {%- endif %} + {{- '\n' }} + {%- endfor %} + {%- endif %} + {{- '<|im_end|>\n' }} + {%- elif message.role == "tool" %} + {%- if loop.previtem and loop.previtem.role != "tool" %} + {{- '<|im_start|>user' }} + {%- endif %} + {{- '\n\n' }} + {{- content }} + {{- '\n' }} + {%- if not loop.last and loop.nextitem.role != "tool" %} + {{- '<|im_end|>\n' }} + {%- elif loop.last %} + {{- '<|im_end|>\n' }} + {%- endif %} + {%- else %} + {{- raise_exception('Unexpected message role.') }} + {%- endif %} + {%- endfor %} + {%- if add_generation_prompt %} + {{- '<|im_start|>assistant\n' }} + {%- if enable_thinking is defined and enable_thinking is true %} + {{- '\n' }} + {%- else %} + {{- '\n\n\n\n' }} + {%- endif %} + {%- endif %} ``` -------------------------------- ### POST /v1/chat/completions Source: https://huggingface.co/ATH-MaaS/OvisOCR2/blame/65c619d374b55d4152e85150fc1b003700bc1f0c/tokenizer.json Sends a multimodal prompt (text and image) to the OvisOCR2 model hosted on an OpenAI-compatible server to receive a generated response. ```APIDOC ## POST /v1/chat/completions ### Description Sends a chat completion request to the model, supporting both text and image inputs. ### Method POST ### Endpoint /v1/chat/completions ### Request Body - **model** (string) - Required - The model identifier (e.g., "ATH-MaaS/OvisOCR2") - **messages** (array) - Required - A list of message objects containing role and content (text or image_url) ### Request Example { "model": "ATH-MaaS/OvisOCR2", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] } ``` -------------------------------- ### Tokenizer Configuration JSON Source: https://huggingface.co/ATH-MaaS/OvisOCR2/raw/main/tokenizer_config.json The complete tokenizer configuration file defining chat templates, special tokens, and model-specific parameters. ```json { "chat_template": "{%- for message in messages %} {%- if message.role == \"system\" %} {{- '<|im_start|>system\n' + message.content + '<|im_end|>\n' }} {%- elif message.role == \"user\" %} {{- '<|im_start|>user\n' + message.content + '<|im_end|>\n' }} {%- elif message.role == \"assistant\" %} {{- '<|im_start|>assistant\n' }} {%- if enable_thinking is defined and enable_thinking is true %} {%- if message.content is mapping %} {{- '\n' + message.content.thought + '\n\n' + message.content.response }} {%- else %} {{- message.content }} {%- endif %} {%- else %} {%- if message.content is mapping %} {{- message.content.response }} {%- else %} {{- message.content }} {%- endif %} {%- endif %} {%- if not loop.last %} {{- '<|im_end|>\n' }} {%- else %} {%- if add_generation_prompt %} {{- '' }} {%- else %} {{- '<|im_end|>\n' }} {%- endif %} {%- endif %} {%- elif message.role == \"tool\" %} {%- if loop.previtem and loop.previtem.role != \"tool\" %} {{- '<|im_start|>user' }} {%- endif %} {{- '\n\n' }} {{- content }} {{- '\n' }} {%- if not loop.last and loop.nextitem.role != \"tool\" %} {{- '<|im_end|>\n' }} {%- elif loop.last %} {{- '<|im_end|>\n' }} {%- endif %} {%- else %} {{- raise_exception('Unexpected message role.') }} {%- endif %} {%- endfor %} {%- if add_generation_prompt %} {{- '<|im_start|>assistant\n' }} {%- if enable_thinking is defined and enable_thinking is true %} {{- '\n' }} {%- else %} {{- '\n\n\n\n' }} {%- endif %} {%- endif %}", "clean_up_tokenization_spaces": false, "eos_token": "<|im_end|>", "errors": "replace", "model_max_length": 262144, "pad_token": "<|endoftext|>", "split_special_tokens": false, "tokenizer_class": "Qwen2Tokenizer", "unk_token": null, "add_bos_token": false, "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+", "extra_special_tokens": { "audio_bos_token": "<|audio_start|>", "audio_eos_token": "<|audio_end|>", "audio_token": "<|audio_pad|>", "image_token": "<|image_pad|>", "video_token": "<|video_pad|>", "vision_bos_token": "<|vision_start|>", "vision_eos_token": "<|vision_end|>" } } ``` -------------------------------- ### Load OvisOCR2 Directly with Transformers Source: https://huggingface.co/ATH-MaaS/OvisOCR2/blame/77bfe9462d1e6f8965ee6698f08ea8ede580912c/model.safetensors Manually load the processor and model to handle chat templates and generation. ```python # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("ATH-MaaS/OvisOCR2") model = AutoModelForMultimodalLM.from_pretrained("ATH-MaaS/OvisOCR2") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) ``` -------------------------------- ### Run Inference with AutoModel Source: https://huggingface.co/ATH-MaaS/OvisOCR2/colab Load the model and processor directly for more granular control over the inference process. ```python # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("ATH-MaaS/OvisOCR2") model = AutoModelForMultimodalLM.from_pretrained("ATH-MaaS/OvisOCR2") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) ``` -------------------------------- ### Run Inference with Pipeline Source: https://huggingface.co/ATH-MaaS/OvisOCR2/colab Use the high-level pipeline helper to perform image-to-text inference. ```python # Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="ATH-MaaS/OvisOCR2") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages) ``` -------------------------------- ### POST /v1/chat/completions Source: https://huggingface.co/ATH-MaaS/OvisOCR2/blame/77bfe9462d1e6f8965ee6698f08ea8ede580912c/.gitattributes An OpenAI-compatible chat completions endpoint provided by vLLM or SGLang servers to perform multimodal inference. ```APIDOC ## POST /v1/chat/completions ### Description Sends a chat request with text and image content to the model server to receive a generated response. ### Method POST ### Endpoint /v1/chat/completions ### Request Body - **model** (string) - Required - The model identifier, e.g., "ATH-MaaS/OvisOCR2" - **messages** (array) - Required - A list of message objects containing role and content (text or image_url) ### Request Example { "model": "ATH-MaaS/OvisOCR2", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] } ``` -------------------------------- ### Tokenizer Configuration Parameters Source: https://huggingface.co/ATH-MaaS/OvisOCR2/blob/d844a45160ff88ed5d51e8c13d7484e50fd2f801/tokenizer_config.json JSON configuration defining tokenizer settings, special tokens, and vision-related tokens for OvisOCR2. ```json { "clean_up_tokenization_spaces": false, "eos_token": "<|im_end|>", "errors": "replace", "model_max_length": 262144, "pad_token": "<|endoftext|>", "split_special_tokens": false, "tokenizer_class": "Qwen2Tokenizer", "unk_token": null, "add_bos_token": false, "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\ \ \\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\ \ ]*|\\s*[\ \ ]+|\\s+(?!\\S)|\\s+", "extra_special_tokens": { "audio_bos_token": "<|audio_start|>", "audio_eos_token": "<|audio_end|>", "audio_token": "<|audio_pad|>", "image_token": "<|image_pad|>", "video_token": "<|video_pad|>", "vision_bos_token": "<|vision_start|>", "vision_eos_token": "<|vision_end|>" } } ``` -------------------------------- ### Tokenizer Configuration and Chat Template Source: https://huggingface.co/ATH-MaaS/OvisOCR2/blob/65c619d374b55d4152e85150fc1b003700bc1f0c/tokenizer_config.json Configuration defining special tokens and the chat template logic for processing system messages, user queries, and tool calls. ```json "lstrip": false, | "normalized": false, | "rstrip": false, | "single_word": false, | "special": true | }, | "248074": { | "content": "", | "lstrip": false, | "normalized": false, | "rstrip": false, | "single_word": false, | "special": true | }, | "248075": { | "content": "", | "lstrip": false, | "normalized": false, | "rstrip": false, | "single_word": false, | "special": true | }, | "248076": { | "content": "<|audio_pad|>", | "lstrip": false, | "normalized": false, | "rstrip": false, | "single_word": false, | "special": true | } | }, | "additional_special_tokens": [ | "<|im_start|>", | "<|im_end|>", | "<|object_ref_start|>", | "<|object_ref_end|>", | "<|box_start|>", | "<|box_end|>", | "<|quad_start|>", | "<|quad_end|>", | "<|vision_start|>", | "<|vision_end|>", | "<|vision_pad|>", | "<|image_pad|>", | "<|video_pad|>" | ], | "bos_token": null, | "chat_template": "{%- set image_count = namespace(value=0) %}\n{%- set video_count = namespace(value=0) %}\n{%- macro render_content(content, do_vision_count, is_system_content=false) %}\n {%- if content is string %}\n {{- content }}\n {%- elif content is iterable and content is not mapping %}\n {%- for item in content %}\n {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}\n {%- if is_system_content %}\n {{- raise_exception('System message cannot contain images.') }}\n {%- endif %}\n {%- if do_vision_count %}\n {%- set image_count.value = image_count.value + 1 %}\n {%- endif %}\n {%- if add_vision_id %}\n {{- 'Picture ' ~ image_count.value ~ ': ' }}\n {%- endif %}\n {{- '<|vision_start|><|image_pad|><|vision_end|>' }}\n {%- elif 'video' in item or item.type == 'video' %}\n {%- if is_system_content %}\n {{- raise_exception('System message cannot contain videos.') }}\n {%- endif %}\n {%- if do_vision_count %}\n {%- set video_count.value = video_count.value + 1 %}\n {%- endif %}\n {%- if add_vision_id %}\n {{- 'Video ' ~ video_count.value ~ ': ' }}\n {%- endif %}\n {{- '<|vision_start|><|video_pad|><|vision_end|>' }}\n {%- elif 'text' in item %}\n {{- item.text }}\n {%- else %}\n {{- raise_exception('Unexpected item type in content.') }}\n {%- endif %}\n {%- endfor %}\n {%- elif content is none or content is undefined %}\n {{- '' }}\n {%- else %}\n {{- raise_exception('Unexpected content type.') }}\n {%- endif %}\n{%- endmacro %}\n{%- if not messages %}\n {{- raise_exception('No messages provided.') }}\n{%- endif %}\n{%- if tools and tools is iterable and tools is not mapping %}\n {{- '<|im_start|>system\\n' }}\n {{- \"# Tools\\n\\nYou have access to the following functions:\\n\\n\" }}\n {%- for tool in tools %}\n {{- \"\\n\" }}\n {{- tool | tojson }}\n {%- endfor %}\n {{- \"\\n\" }}\n {{- '\\n\\nIf you choose to call a function ONLY reply in the following format with NO suffix:\\n\\n\\n\\n\\nvalue_1\\n\\n\\nThis is the value for the second parameter\\nthat can span\\nmultiple lines\\n\\n\\n\\n\\n\\nReminder:\\n- Function calls MUST follow the specified format: an inner block must be nested within XML tags\\n- Required parameters MUST be specified\\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\\n' }}\n {%- if messages[0].role == 'system' %}\n {%- set content = render_content(messages[0].content, false, true)|trim %}\n {%- if content %}\n {{- '\\n\\n' + content }}\n {%- endif %}\n {%- endif %}\n {{- '<|im_end|>\\n' }}\n{%- else %}\n {%- if messages[0].role == 'system' %}\n {%- set content = render_content(messages[0].content, false, true)|trim %}\n {{- '<|im_start|>system\\n' + content + '<|im_end|>\\n' }}\n {%- endif %}\n{%- endif %}\n{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}\n{%- for message in messages[::-1] %}\n {%- set index = (messages|length - 1) - loop.index0 %}\n {%- if ns.multi_step_tool and message.role == \"user\" %}\n {%- set content = render_content(message.content, false)|trim %}\n {%- if not(content.startswith('') and content.endswith('')) %}\n {%- set ns.multi_step_tool = false %}\n {%- set ns.last_query_index = index %}\n {%- endif %}\n {%- endif %}\n{%- endfor %}\n{%- if ns.multi_step_tool %}\n {{- raise_exception('No user query found in messages.') }}\n{%- endif %}\n{%- for message in messages %}\n {%- set content = render_content(message.content, true)|trim %}\n " ``` -------------------------------- ### Running OvisOCR2 with Docker Source: https://huggingface.co/ATH-MaaS/OvisOCR2/blame/65c619d374b55d4152e85150fc1b003700bc1f0c/model.safetensors Command to run the model using the Docker model runner. ```bash docker model run hf.co/ATH-MaaS/OvisOCR2 ``` -------------------------------- ### Query OvisOCR2 via OpenAI-compatible API Source: https://huggingface.co/ATH-MaaS/OvisOCR2/blame/77bfe9462d1e6f8965ee6698f08ea8ede580912c/.gitattributes Send a POST request to the running server to perform inference on an image. ```bash curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ATH-MaaS/OvisOCR2", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' ``` -------------------------------- ### Configure Git LFS Attributes Source: https://huggingface.co/ATH-MaaS/OvisOCR2/blame/77bfe9462d1e6f8965ee6698f08ea8ede580912c/.gitattributes Define file patterns to be tracked by Git LFS in the .gitattributes file. ```text *.7z filter=lfs diff=lfs merge=lfs -text *.arrow filter=lfs diff=lfs merge=lfs -text *.bin filter=lfs diff=lfs merge=lfs -text *.bz2 filter=lfs diff=lfs merge=lfs -text *.ckpt filter=lfs diff=lfs merge=lfs -text *.ftz filter=lfs diff=lfs merge=lfs -text *.gz filter=lfs diff=lfs merge=lfs -text *.h5 filter=lfs diff=lfs merge=lfs -text *.joblib filter=lfs diff=lfs merge=lfs -text *.lfs.* filter=lfs diff=lfs merge=lfs -text *.mlmodel filter=lfs diff=lfs merge=lfs -text *.model filter=lfs diff=lfs merge=lfs -text *.msgpack filter=lfs diff=lfs merge=lfs -text *.npy filter=lfs diff=lfs merge=lfs -text *.npz filter=lfs diff=lfs merge=lfs -text *.onnx filter=lfs diff=lfs merge=lfs -text *.ot filter=lfs diff=lfs merge=lfs -text *.parquet filter=lfs diff=lfs merge=lfs -text *.pb filter=lfs diff=lfs merge=lfs -text *.pickle filter=lfs diff=lfs merge=lfs -text *.pkl filter=lfs diff=lfs merge=lfs -text *.pt filter=lfs diff=lfs merge=lfs -text *.pth filter=lfs diff=lfs merge=lfs -text *.rar filter=lfs diff=lfs merge=lfs -text *.safetensors filter=lfs diff=lfs merge=lfs -text saved_model/**/* filter=lfs diff=lfs merge=lfs -text *.tar.* filter=lfs diff=lfs merge=lfs -text *.tar filter=lfs diff=lfs merge=lfs -text *.tflite filter=lfs diff=lfs merge=lfs -text *.tgz filter=lfs diff=lfs merge=lfs -text *.wasm filter=lfs diff=lfs merge=lfs -text *.xz filter=lfs diff=lfs merge=lfs -text *.zip filter=lfs diff=lfs merge=lfs -text *.zst filter=lfs diff=lfs merge=lfs -text *tfevents* filter=lfs diff=lfs merge=lfs -text tokenizer.json filter=lfs diff=lfs merge=lfs -text performance.png filter=lfs diff=lfs merge=lfs -text performance_omnidocbench_v16.png filter=lfs diff=lfs merge=lfs -text performance_puredocbench.png filter=lfs diff=lfs merge=lfs -text ``` -------------------------------- ### OvisOCR2 Inference Implementation Source: https://huggingface.co/ATH-MaaS/OvisOCR2/commit/77bfe9462d1e6f8965ee6698f08ea8ede580912c A parser class for OvisOCR2 that handles model initialization, prompt formatting, and image parsing with vLLM. ```python from PIL import Image from vllm import LLM, SamplingParams class OvisOCR2Parser: def __init__(self, model_name_or_path: str): self.model = LLM( model=model_name_or_path, tensor_parallel_size=1, gpu_memory_utilization=0.8, gdn_prefill_backend="triton" ) prompt = '\nExtract all readable content from the image in natural human reading order and output the result as a single Markdown document. For charts or images, represent them using an HTML image tag: <' + 'img src="images/bbox_{left}_{top}_{right}_{bottom}.jpg" />, where left, top, right, bottom are bounding box coordinates scaled to [0, 1000). Format formulas as LaTeX. Format tables as HTML: ...
. Transcribe all other text as standard Markdown. Preserve the original text without translation or paraphrasing.' self.prompt = self.model.get_tokenizer().apply_chat_template( [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": prompt}]}], tokenize=False, add_generation_prompt=True, enable_thinking=False ) self.sampling_params = SamplingParams( max_tokens=16384, temperature=0.0 ) def _clean_truncated_repeats( self, text: str, min_text_len: int = 8000, max_period: int = 200, min_period: int = 1, min_repeat_chars: int = 100, min_repeat_times: int = 5 ) -> str: n = len(text) if n < min_text_len: return text max_period = min(max_period, n - 1) for unit_len in range(min_period, max_period + 1): if text[n - 1] != text[n - 1 - unit_len]: continue match_len = 1 idx = n - 2 while idx >= unit_len and text[idx] == text[idx - unit_len]: match_len += 1 idx -= 1 total_len = match_len + unit_len repeat_times = total_len // unit_len tail_len = total_len % unit_len if repeat_times >= min_repeat_times and total_len >= min_repeat_chars: return text[: n - total_len + unit_len] + text[n - tail_len:] return text def parse(self, images: list[Image.Image], filter_imgtags: bool = True) -> list[str]: vllm_inputs = [ { "prompt": self.prompt, "multi_modal_data": {"image": image}, "mm_processor_kwargs": { "images_kwargs": { "min_pixels": 448 * 448, "max_pixels": 2880 * 2880 } } } for image in images ] outputs = self.model.generate(vllm_inputs, self.sampling_params) markdowns = [] for output in outputs: text = output.outputs[0].text.strip() if filter_imgtags: text = "\n\n".join( block for block in text.split("\n\n") if not block.strip().startswith('", "lstrip": false, "normalized": false, "rstrip": false, "single_word": false, "special": true }, "248045": { "content": "<|im_start|>", "lstrip": false, "normalized": false, "rstrip": false, "single_word": false, "special": true }, "248046": { "content": "<|im_end|>", "lstrip": false, "normalized": false, "rstrip": false, "single_word": false, "special": true }, "248047": { ```