GLM-OCR

https://github.com/zai-org/glm-ocr
Admin
GLM-OCR is a multimodal OCR model for complex document understanding that achieves state-of-the-art...
Tokens:86,038
Snippets:613
Trust Score:7.8
License:Apache-2.0
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### Install and start the self-hosted server

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/self-host/README.md

Install the self-hosted server package and start the server on the GPU machine. The server listens on 0.0.0.0:5002 by default with the API endpoint at /glmocr/parse.

```bash
pip install "glmocr[selfhosted,server]"
```

```bash
python -m glmocr.server --config config.yaml
```

--------------------------------

### Install the client SDK

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/self-host/README.md

Install the client-side SDK on any machine without GPU.

```bash
pip install glmocr
```

--------------------------------

### Flask service: install, start, and curl API

Source: https://github.com/zai-org/glm-ocr/blob/main/README.md

Install the optional server dependency, start the Flask service, and call the /glmocr/parse endpoint. The images field can be a string or a list; a list is treated as pages of a single document, and multiple independent documents require separate requests.

```bash
pip install "glmocr[server]"
```

```bash
# Start service
python -m glmocr.server

# With debug logging
python -m glmocr.server --log-level DEBUG

# Call API
curl -X POST http://localhost:5002/glmocr/parse \
  -H "Content-Type: application/json" \
  -d '{"images": ["./example/source/code.png"]}'
```

--------------------------------

### Install GLM-OCR SDK from source

Source: https://github.com/zai-org/glm-ocr/blob/main/README.md

Install the GLM-OCR SDK from source for development. Clones the repository, creates a Python 3.12 virtual environment with uv, and installs the package in editable mode.

```bash
git clone https://github.com/zai-org/glm-ocr.git
cd glm-ocr
uv venv --python 3.12 --seed && source .venv/bin/activate
uv pip install -e .
```

--------------------------------

### Install GLM-OCR SDK and transformers from source

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/mlx-deploy/README.md

Installs the GLM-OCR SDK in editable mode and installs the transformers library from source, which is required by the SDK.

```bash
pip install -e .

# Install transformers from source (required by the SDK)
pip install git+https://github.com/huggingface/transformers.git
```

--------------------------------

### Install glmocr and Set ZHIPU_API_KEY

Source: https://github.com/zai-org/glm-ocr/blob/main/skills/sdk/SKILL.md

Install the glmocr package and set the ZHIPU_API_KEY environment variable, either by exporting it or appending it to a .env file in the working directory. This setup is required once before using the SDK.

```bash
pip install glmocr
```

```bash
export ZHIPU_API_KEY=sk-xxx
# or add to .env file in working directory:
echo "ZHIPU_API_KEY=sk-xxx" >> .env
```

--------------------------------

### Install Ollama on macOS/Linux

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/ollama-deploy/README.md

Downloads and installs Ollama using the official install script.

```bash
curl -fsSL https://ollama.ai/install.sh | sh
```

--------------------------------

### Install GLM-OCR SDK with pip extras

Source: https://github.com/zai-org/glm-ocr/blob/main/README.md

Install the GLM-OCR SDK with the lightest installation matching your scenario: cloud/MaaS with local images/PDFs (fastest), self-hosted pipeline for layout detection, or Flask service support.

```bash
pip install glmocr
```

```bash
pip install "glmocr[selfhosted]"
```

```bash
pip install "glmocr[server]"
```

--------------------------------

### Quick Start: Specify GPUs and use vLLM

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/multi-gpu-deploy/README.md

Launches the script with the vLLM engine on specified GPU IDs (0,1,2,3).

```bash
python examples/multi-gpu-deploy/launch.py -i ./images -o ./output --engine vllm --gpus 0,1,2,3
```

--------------------------------

### File Structure

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/multi-gpu-deploy/README.md

Directory layout of the multi-GPU deployment example.

```text
examples/multi-gpu-deploy/
├── launch.py        # Entry point and CLI argument parser
├── coordinator.py   # Orchestration: GPU detection, engine/worker lifecycle
├── engine.py        # Engine service management and progress tracking
├── worker.py        # Worker process: GLM-OCR pipeline execution
├── gpu_utils.py     # GPU detection, port checking, file sharding
├── README.md        # This file (English)
└── README_zh.md     # Chinese documentation
```

--------------------------------

### Quick Start: Use all available GPUs with sglang (default)

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/multi-gpu-deploy/README.md

Launches the multi-GPU deployment script with default settings (sglang engine) on all available GPUs.

```bash
python examples/multi-gpu-deploy/launch.py -i ./images -o ./output -m /path/to/GLM-OCR
```

--------------------------------

### Install mlx-vlm from git for GLM-OCR support

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/mlx-deploy/README.md

Install mlx-vlm from git to get GLM-OCR support when the PyPI release lacks it, fixing the 'model glm_ocr not found' import error.

```bash
pip install git+https://github.com/Blaizzy/mlx-vlm.git

```

--------------------------------

### Quick Start: Custom model path and VRAM threshold

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/multi-gpu-deploy/README.md

Launches the script with a custom model path and a minimum free VRAM threshold of 20000 MB.

```bash
python examples/multi-gpu-deploy/launch.py -i ./images -o ./output -m /path/to/GLM-OCR --min-free-mb 20000
```

--------------------------------

### Install self-hosted extra

Source: https://github.com/zai-org/glm-ocr/blob/main/README.md

Install the self-hosted extra required for deploying GLM-OCR locally with vLLM or SGLang. Provides the complete pipeline: layout detection, parallel region OCR, and result formatting.

```bash
pip install "glmocr[selfhosted]"
```

--------------------------------

### Verify Ollama Installation

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/ollama-deploy/README.md

Checks the installed Ollama version.

```bash
ollama --version
```

--------------------------------

### Parse Images with CLI

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/ollama-deploy/README.md

Command-line examples for parsing a single image, specifying an output directory, and enabling debug logging.

```bash
# Parse a single image
glmocr parse examples/source/code.png --config config.yaml

# Parse with custom output directory
glmocr parse examples/source/code.png --output ./results/

# Enable debug logging
glmocr parse examples/source/code.png --log-level DEBUG
```

--------------------------------

### ShareGPT format dataset example for multimodal SFT

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/finetune/README.md

Example of a single sample in ShareGPT format for multimodal SFT. Each <image> tag must correspond to an entry in the images list, and image paths are relative to the data/ directory.

```json
[
  {
    "messages": [
      {
        "role": "user",
        "content": "<image>Code Generation:"
      },
      {
        "role": "assistant",
        "content": "```rdkit\nC1=CC=CC=C1\n```"
      }
    ],
    "images": [
      "cosyn_chemical/example.png"
    ]
  }
]
```

--------------------------------

### Install vLLM for GLM-OCR

Source: https://github.com/zai-org/glm-ocr/blob/main/README.md

Install vLLM either via Docker image or pip. Choose the method that fits your environment.

```bash
docker pull vllm/vllm-openai:v0.19.0-ubuntu2404
```

```bash
pip install -U "vllm>=0.19.0"
```

--------------------------------

### Install SGLang for GLM-OCR

Source: https://github.com/zai-org/glm-ocr/blob/main/README.md

Install SGLang either via Docker image or pip. Choose the method that fits your environment.

```bash
docker pull lmsysorg/sglang:v0.5.10
```

```bash
pip install "sglang>=0.5.10"
```

--------------------------------

### Use Python API

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/ollama-deploy/README.md

Python example for initializing GlmOcr with a custom config file, parsing an image, and saving the result.

```python
from glmocr import GlmOcr

# Initialize with custom config
with GlmOcr(config_path="config.yaml") as parser:
    result = parser.parse("image.png")
    print(result.markdown_result)
    result.save(output_dir="./results")
```

--------------------------------

### Install mlx-vlm from git

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/mlx-deploy/README.md

Install mlx-vlm from the git repository because the glm_ocr model architecture is not yet available in the latest PyPI release 0.3.10. Once a new release ships with GLM-OCR support, you can switch to `pip install mlx-vlm`.

```bash
pip install git+https://github.com/Blaizzy/mlx-vlm.git
```

--------------------------------

### Start Ollama Service

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/ollama-deploy/README.md

Starts the Ollama service if it is not already running. The service defaults to http://localhost:11434.

```bash
ollama serve
```

--------------------------------

### Launch mlx-vlm server

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/mlx-deploy/README.md

Starts the mlx-vlm server with an OpenAI-compatible API on http://localhost:8080. The first run downloads model weights from Hugging Face (mlx-community/GLM-OCR-bf16).

```bash
mlx_vlm.server --trust-remote-code
```

--------------------------------

### Basic usage

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/multi-gpu-deploy/README.md

Runs the launcher with input and output directories.

```bash
python examples/multi-gpu-deploy/launch.py -i /data/documents -o /data/results
```

--------------------------------

### Configure API key with config_setup.py

Source: https://github.com/zai-org/glm-ocr/blob/main/skills/glmocr/SKILL.md

Configures the ZHIPU_API_KEY for the GLM-OCR skill. Run this once before using the OCR commands.

```bash
python scripts/config_setup.py setup --api-key YOUR_KEY
```

--------------------------------

### Configure SDK with .env file

Source: https://github.com/zai-org/glm-ocr/blob/main/agent.md

Place this .env file in the working-directory ancestry to configure the SDK. Real environment variables always take priority over values from this file.

```dotenv
# .env
ZHIPU_API_KEY=sk-my-secret-key
GLMOCR_MODE=maas
GLMOCR_LOG_LEVEL=DEBUG
```

--------------------------------

### Install LLaMA-Factory and upgrade transformers

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/finetune/README.md

Clones the LLaMA-Factory repository, installs it with metrics dependencies, and upgrades transformers to a version compatible with GLM-OCR. The default transformers 5.0.0 does not support GLM-OCR.

```bash
git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e .
pip install -r requirements/metrics.txt
```

```bash
llamafactory-cli version
```

```bash
pip install --upgrade transformers
```

--------------------------------

### Response Format JSON Example

Source: https://github.com/zai-org/glm-ocr/blob/main/skills/glmocr-handwriting/SKILL.md

Example JSON response. The text field contains extracted Markdown for display; layout_details holds layout analysis, and error provides details on failure.

```json
{
  "ok": true,
  "text": "Recognized handwritten text in Markdown...",
  "layout_details": [...],
  "result": null,
  "error": null,
  "source": "/path/to/file",
  "source_type": "file",
  "raw_result_included": false
}
```

--------------------------------

### Custom engine arguments

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/multi-gpu-deploy/README.md

Runs the launcher with custom engine arguments, here setting the static memory fraction for sglang.

```bash
# sglang with custom memory fraction
python examples/multi-gpu-deploy/launch.py \
  -i /data/documents \
  -o /data/results \
  --engine-args "--mem-fraction-static 0.85"
```

--------------------------------

### CLI Reference Usage

Source: https://github.com/zai-org/glm-ocr/blob/main/skills/glmocr-handwriting/SKILL.md

CLI usage reference. Requires either --file-url or --file; optional flags are --output, --pretty, and --include-raw.

```bash
python {baseDir}/scripts/glm_ocr_cli.py (--file-url URL | --file PATH) [--output FILE] [--pretty] [--include-raw]
```

--------------------------------

### json_result structure example

Source: https://github.com/zai-org/glm-ocr/blob/main/skills/sdk/SKILL.md

Example showing the structure of `json_result`: a list of pages, each containing a list of region dicts with fields index, label, content, and bbox_2d. `bbox_2d` is [x1, y1, x2, y2] normalised to 0-1000 scale. Region labels include title, text, table, figure, formula, header, footer, page_number, reference, seal.

```json
[
  [
    {
      "index": 0,
      "label": "title",
      "content": "Annual Report 2024",
      "bbox_2d": [100, 50, 900, 120]
    },
    {
      "index": 1,
      "label": "table",
      "content": "| Q1 | Q2 |\n|---|---|\n| 120 | 145 |",
      "bbox_2d": [100, 140, 900, 400]
    }
  ]
]
```

--------------------------------

### Check Model Status with Ollama

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/ollama-deploy/README.md

Commands to list installed models, view model details, and check running models.

```bash
# List installed models
ollama list

# View model details
ollama show glm-ocr:latest

# Check running models
ollama ps
```

--------------------------------

### Specify custom port for mlx-vlm server

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/mlx-deploy/README.md

Starts the mlx-vlm server on a custom port (9090) instead of the default 8080.

```bash
mlx_vlm.server --trust-remote-code --port 9090
```

--------------------------------

### Single GPU Training with llamafactory-cli

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/finetune/README.md

Launch GLM-OCR full fine-tuning on a single GPU. Set CUDA_VISIBLE_DEVICES to control which GPU is used; replace path/to/glm_ocr_full_sft.yaml with your configuration file. The checkpoint is saved to the output_dir specified in the YAML.

```bash
DISABLE_VERSION_CHECK=1 CUDA_VISIBLE_DEVICES=0 \
  llamafactory-cli train path/to/glm_ocr_full_sft.yaml
```

--------------------------------

### Parse a document with the CLI

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/self-host/README.md

Use the CLI to parse a document. You can either pass the config file explicitly or set the ZHIPU_API_KEY environment variable.

```bash
glmocr parse document.png --config config.yaml
```

```bash
export ZHIPU_API_KEY=any-string
glmocr parse document.png
```

--------------------------------

### Custom config YAML

Source: https://github.com/zai-org/glm-ocr/blob/main/examples/multi-gpu-deploy/README.md

Runs the launcher with a custom GLM-OCR configuration YAML file.

```bash
python examples/multi-gpu-deploy/launch.py \
  -i /data/documents \
  -o /data/results \
  --config my_config.yaml
```

--------------------------------

### Save control: --no-layout-vis and --output

Source: https://github.com/zai-org/glm-ocr/blob/main/skills/sdk/SKILL.md

Control output size by skipping layout visualization images with `--no-layout-vis`; can combine with `--output` to specify destination directory.

```bash
# Skip layout visualization images (faster, smaller output)
glmocr parse image.png --no-layout-vis

# Parse and save only JSON + Markdown, skip layout vis
glmocr parse image.png --no-layout-vis --output ./results/
```

--------------------------------

### GLM-OCR API response format

Source: https://github.com/zai-org/glm-ocr/blob/main/skills/glmocr/SKILL.md

Example JSON response from the GLM-OCR API. The 'text' field contains the extracted text in Markdown for display.

```json
{
  "ok": true,
  "text": "# Extracted text in Markdown...",
  "layout_details": [[...]],
  "result": { "raw_api_response": "..." },
  "error": null,
  "source": "/path/to/file.jpg",
  "source_type": "file"
}
```

--------------------------------

### API key not configured error

Source: https://github.com/zai-org/glm-ocr/blob/main/skills/glmocr/SKILL.md

Error message shown when ZHIPU_API_KEY is not configured. Display this exact error to the user and guide them to configure the key.

```text
Error: ZHIPU_API_KEY not configured. Get your API key at: https://www.bigmodel.cn/usercenter/proj-mgmt/apikeys
```

--------------------------------

### Configuration priority order

Source: https://github.com/zai-org/glm-ocr/blob/main/skills/sdk/SKILL.md

Shows the priority order for configuration sources, where constructor arguments override environment variables, which override .env file, which overrides config.yaml, which overrides built-in defaults.

```text
Constructor kwargs  >  os.environ  >  .env file  >  config.yaml  >  built-in defaults
```
Tokens:
Raw