### Install and Setup Docext

Source: https://github.com/nanonets/docext/blob/main/_autodocs/QUICK-REFERENCE.md

Install the docext package using pip. Set the VLM_MODEL_URL environment variable and start the vLLM server in a separate terminal. Finally, launch the Gradio UI or start the server programmatically.

```bash
# Install
pip install docext

# Set environment
export VLM_MODEL_URL="http://localhost:8000/v1"

# Start vLLM server (separate terminal)
vllm serve Qwen/Qwen2.5-VL-7B-Instruct --port 8000

# Start Gradio UI
docext --server_port 7860

# Or start everything in Python
from docext.core.vllm import VLLMServer
server = VLLMServer("hosted_vllm/Qwen/...")
server.run_in_background()
```

--------------------------------

### Local Development Setup

Source: https://github.com/nanonets/docext/blob/main/_autodocs/README.md

Start a local VLLM server and configure Docext for local development.

```bash
vllm serve Qwen/Qwen2.5-VL-7B-Instruct --host 0.0.0.0 --port 8000
export VLM_MODEL_URL="http://localhost:8000/v1"
docext --server_port 7860
```

--------------------------------

### Local Development with vLLM

Source: https://github.com/nanonets/docext/blob/main/_autodocs/04-configuration.md

This example shows how to start a vLLM server and then run Docext for local development. It includes setting the VLM_MODEL_URL and configuring Docext parameters.

```bash
# Start vLLM server in separate terminal
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype bfloat16

# In another terminal, start docext
export VLM_MODEL_URL="http://localhost:8000/v1"
docext \
  --model_name hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct \
  --vlm_server_host 127.0.0.1 \
  --vlm_server_port 8000 \
  --server_port 7860 \
  --gpu_memory_utilization 0.9 \
  --max_img_size 2048
```

--------------------------------

### Clone Repository and Install Package

Source: https://github.com/nanonets/docext/blob/main/_autodocs/06-quick-start.md

Clone the docext repository and install the package using pip. Use the development installation for active development.

```bash
# Clone the repository
git clone https://github.com/nanonets/docext.git
cd docext

# Install package
pip install -e .

# Or for development
pip install -e ".[dev]"
```

--------------------------------

### Install Development Dependencies

Source: https://github.com/nanonets/docext/blob/main/contribution.md

Steps to set up the development environment, including creating a virtual environment, activating it, and installing project dependencies with pre-commit hooks.

```bash
uv venv --python=3.11
source .venv/bin/activate
pip install -e .
uv pip install pre-commit
pre-commit install
```

--------------------------------

### Install Docext using uv

Source: https://github.com/nanonets/docext/blob/main/EXT_README.md

Installs the docext package using uv for environment management and package installation. It covers creating a virtual environment and installing from PyPI or source.

```bash
# create a virtual environment
## install uv if not installed
curl -LsSf https://astral.sh/uv/install.sh | sh
## create a virtual environment with python 3.11
uv venv --python=3.11
source .venv/bin/activate

# Install from PyPI
uv pip install docext

# Or install from source
git clone https://github.com/nanonets/docext.git
cd docext
uv pip install -e .
```

--------------------------------

### Start docext Application

Source: https://github.com/nanonets/docext/blob/main/_autodocs/06-quick-start.md

Start the docext web interface. Set the VLM_MODEL_URL environment variable to point to your vLLM server and specify the server port.

```bash
# In terminal 2: Start the web interface
export VLM_MODEL_URL="http://localhost:8000/v1"
docext --server_port 7860
```

--------------------------------

### Install Docext Package

Source: https://github.com/nanonets/docext/blob/main/pdf2markdown.ipynb

Installs the docext package using uv pip. Ensure necessary directories are created beforehand.

```python
# install the docext package
## UV tries to access these files
!mkdir /backend-container
!mkdir /backend-container/containers
!touch /backend-container/containers/build.constraints
!touch /backend-container/containers/requirements.constraints
!uv pip install docext -q
```

--------------------------------

### Install and Run Docext Application

Source: https://github.com/nanonets/docext/blob/main/docext.ipynb

Clones the Docext repository, installs its dependencies, and runs the application. Configure parameters like model length, GPU utilization, and image limits as needed.

```python
!git clone https://github.com/NanoNets/docext
%cd docext
# install the repo dependencies
!pip install -e . -q
# run the app
!python -m docext.app.app --max_model_len 8000 --gpu_memory_utilization 0.9 --max_num_imgs 1 --max_img_size 1024 --concurrency_limit 1
```

--------------------------------

### Benchmark Configuration Example

Source: https://github.com/nanonets/docext/blob/main/_autodocs/QUICK-REFERENCE.md

Example YAML configuration file for benchmarking tasks, datasets, and models. Includes model-specific API URLs and default templates.

```yaml
# configs/benchmark_config.yaml
tasks:
  - KIE
  - OCR

datasets:
  - nanonets_kie
  - docvqa

models:
  - qwen

qwen:
  name: "hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct"
  api_url: "http://localhost:8000/v1"

KIE_default_template: "Extract these fields: {fields}"
OCR_default_template: "Read all text"

cache_dir: "./cache"
max_workers: 4
ignore_cache: false
```

--------------------------------

### Clone and Install Docext

Source: https://github.com/nanonets/docext/blob/main/_autodocs/06-quick-start.md

Clone the Docext repository and install it in editable mode using pip.

```bash
git clone https://github.com/nanonets/docext.git
cd docext
pip install -e .
```

--------------------------------

### VLLMServer Configuration Example

Source: https://github.com/nanonets/docext/blob/main/_autodocs/03-types.md

Demonstrates how to configure and instantiate a VLLMServer with specific parameters. Ensure 'model_name' uses the 'hosted_vllm/' prefix.

```python
from docext.core.vllm import VLLMServer

config = {
    "model_name": "hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct",
    "host": "0.0.0.0",
    "port": 8000,
    "max_model_len": 10000,
    "gpu_memory_utilization": 0.9,
    "max_num_imgs": 5,
    "vllm_start_timeout": 300,
    "dtype": "bfloat16"
}

server = VLLMServer(**config)
```

--------------------------------

### Start vLLM Server

Source: https://github.com/nanonets/docext/blob/main/_autodocs/06-quick-start.md

Start the vLLM server to serve the language model. Ensure the model name, host, port, and data type are correctly specified.

```bash
# In terminal 1: Start vLLM server
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype bfloat16
```

--------------------------------

### Install docext using uv

Source: https://github.com/nanonets/docext/blob/main/PDF2MD_README.md

Install the docext package using uv within a virtual environment. Ensure Python 3.11 is used.

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh

## create a virtual environment with python 3.11
uv venv --python=3.11
source .venv/bin/activate

# Install from PyPI
uv pip install docext

# Or install from source
git clone https://github.com/nanonets/docext.git
cd docext
uv pip install -e .
```

--------------------------------

### Clone Repository and Install Dependencies

Source: https://github.com/nanonets/docext/blob/main/docext/benchmark/README.md

Clone the IDP Leaderboard repository and install the required Python packages using pip.

```bash
git clone https://github.com/NanoNets/docext.git
cd docext
pip install -r requirements.txt
```

--------------------------------

### Start vLLM Server

Source: https://github.com/nanonets/docext/blob/main/_autodocs/01-core-api-reference.md

Starts the vLLM server as a background subprocess. Ensure the server is started before making requests. Use `wait_for_server` to confirm readiness.

```python
from docext.core.vllm import VLLMServer

server = VLLMServer(
    model_name="hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct",
    port=8000,
    max_model_len=10000
)

server.start_server()
server.wait_for_server(timeout=300)

# Now make requests to http://localhost:8000
```

--------------------------------

### High-Throughput Production Configuration for Docext

Source: https://github.com/nanonets/docext/blob/main/_autodocs/04-configuration.md

Configure Docext for high-throughput production environments. This example sets a remote VLM_MODEL_URL, increases concurrency, and adjusts vLLM specific parameters like start timeout.

```bash
export VLM_MODEL_URL="http://gpu-server:8000/v1"
docext \
  --model_name hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct \
  --vlm_server_host gpu-server \
  --vlm_server_port 8000 \
  --server_port 7860 \
  --gpu_memory_utilization 0.95 \
  --max_model_len 15000 \
  --max_img_size 2048 \
  --concurrency_limit 10 \
  --vllm_start_timeout 300 \
  --no-share
```

--------------------------------

### Start the docext API Server

Source: https://github.com/nanonets/docext/blob/main/EXT_README.md

Starts the docext API server. Use the --concurrency_limit flag to adjust the number of parallel requests the server can handle.

```bash
# increase the concurrency limit to process more requests in parallel, default is 1
python -m docext.app.app --concurrency_limit 10
```

--------------------------------

### Start docext web interface

Source: https://github.com/nanonets/docext/blob/main/PDF2MD_README.md

Launch the Gradio-based web interface for document processing. Use default or custom configurations for model and image size.

```bash
# Start the web interface with default configs
python -m docext.app.app --model_name hosted_vllm/nanonets/Nanonets-OCR-s

# Start the web interface with custom configs
python -m docext.app.app --model_name hosted_vllm/nanonets/Nanonets-OCR-s --max_img_size 1024 --concurrency_limit 16 # `--help` for more options
```

--------------------------------

### API Response Example Usage

Source: https://github.com/nanonets/docext/blob/main/_autodocs/03-types.md

Example of how to call a sync_request function and access the content from the response.

```python
response = sync_request(messages, "hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct")
content = response["choices"][0]["message"]["content"]
# content = '{"invoice_number": "INV-001", "date": "2025-06-24"}'
```

--------------------------------

### Initialize vLLM Server and Launch Gradio Interface

Source: https://github.com/nanonets/docext/blob/main/_autodocs/02-application-ui.md

Use this function to start the vLLM server and launch the Gradio web interface. Ensure all parameters are correctly configured for your environment.

```python
from docext.app.app import main

main(
    model_name="hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct",
    host="127.0.0.1",
    port=8000,
    gradio_port=7860,
    max_model_len=10000,
    gpu_memory_utilization=0.9,
    max_num_imgs=5,
    vllm_start_timeout=300,
    max_img_size=2048,
    concurrency_limit=2,
    share=False,
    dtype="bfloat16",
    max_gen_tokens=10000
)
```

--------------------------------

### Start vLLM Server

Source: https://github.com/nanonets/docext/blob/main/_autodocs/README.md

Initiate a vLLM server in a background thread. Configure the model name and port for the server.

```python
from docext.core.vllm import VLLMServer

server = VLLMServer(
    model_name="hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct",
    port=8000
)
thread = server.run_in_background()
```

--------------------------------

### Run the Benchmark Script

Source: https://github.com/nanonets/docext/blob/main/docext/benchmark/README.md

Execute the main benchmark script to start the evaluation process. Ensure the configuration file is set up correctly.

```bash
python docext/benchmark/benchmark.py
```

--------------------------------

### Example .env file for Docext Configuration

Source: https://github.com/nanonets/docext/blob/main/_autodocs/04-configuration.md

A sample .env file demonstrating how to configure VLM_MODEL_URL, API_KEY, and LOG_LEVEL for Docext. This file is loaded by python-dotenv.

```dotenv
VLM_MODEL_URL=http://localhost:8000/v1
API_KEY=sk-example-key
LOG_LEVEL=debug
```

--------------------------------

### Configure vLLM Server Start Timeout

Source: https://github.com/nanonets/docext/blob/main/_autodocs/04-configuration.md

Set the timeout in seconds for waiting for the vLLM server to start. This prevents indefinite waiting if the server fails to initialize.

```bash
# Wait up to 5 minutes for server
docext --vllm_start_timeout 300

# Wait only 2 minutes
docext --vllm_start_timeout 120
```

--------------------------------

### Use Vendor-Hosted OpenAI Model

Source: https://github.com/nanonets/docext/blob/main/EXT_README.md

Example of how to use a vendor-hosted model, specifically 'gpt-4o' from OpenAI. Requires the OPENAI_API_KEY environment variable to be set.

```bash
export OPENROUTER_API_KEY=sk-...
python -m docext.app.app --model_name "openrouter/meta-llama/llama-4-maverick:free"
```

--------------------------------

### Example VLM API Field Definitions

Source: https://github.com/nanonets/docext/blob/main/_autodocs/03-types.md

Provides an example of a list of field definitions, specifying 'invoice_number' and 'invoice_date' with their respective descriptions.

```python
fields = [
    {
        "name": "invoice_number",
        "description": "The unique invoice identifier from the header"
    },
    {
        "name": "invoice_date",
        "description": "Date the invoice was issued (YYYY-MM-DD format)"
    }
]
```

--------------------------------

### Run Docext Application Server

Source: https://github.com/nanonets/docext/blob/main/pdf2markdown.ipynb

Launches the docext application server with specified model and resource configurations. This command is used to start the service for document conversion.

```python
!python -m docext.app.app --model_name hosted_vllm/nanonets/Nanonets-OCR-s --gpu_memory_utilization 0.95 --max_model_len 10000 --max_gen_tokens 8000 --concurrency_limit 1 --max_num_imgs 1 --dtype float16 --server_port 9998
```

--------------------------------

### Run Docext Application

Source: https://github.com/nanonets/docext/blob/main/_autodocs/02-application-ui.md

Execute the docext application either as an installed console script or by running it as a Python module.

```bash
# As console script (installed via setup.py)
docext

# As module
python -m docext
```

--------------------------------

### Project Documentation Structure

Source: https://github.com/nanonets/docext/blob/main/_autodocs/MANIFEST.md

Illustrates the hierarchical organization of the project's documentation files, starting from the README.md.

```markdown
README.md (Overview)
├── QUICK-REFERENCE.md (Cheat sheet)
├── 01-core-api-reference.md (Details)
├── 02-application-ui.md (Details)
├── 03-types.md (Definitions)
├── 04-configuration.md (Setup)
├── 05-benchmark-api.md (Evaluation)
├── 06-quick-start.md (Examples)
└── 07-module-index.md (Hierarchy)
```

--------------------------------

### Run Docext Tests

Source: https://github.com/nanonets/docext/blob/main/_autodocs/06-quick-start.md

Install test dependencies and run all tests using pytest. Includes options for running with coverage.

```bash
# Install test dependencies
pip install pytest pytest-cov

# Run all tests
pytest tests/

# Run with coverage
pytest --cov=docext tests/
```

--------------------------------

### Benchmark Configuration Structure

Source: https://github.com/nanonets/docext/blob/main/_autodocs/05-benchmark-api.md

Example YAML structure for configuring benchmark tasks, datasets, models, and data sources. Includes placeholders for API keys and paths.

```yaml
# Task configuration
tasks:
  - KIE
  - OCR
  - VQA
  - CLASSIFICATION
  - TABLE

# Dataset selection
datasets:
  - nanonets_kie
  - nanonets_cls
  - docvqa

# Model selection
models:
  - model_1
  - model_2

model_1:
  name: "gpt-4-vision"
  api_key: "${OPENAI_API_KEY}"

model_2:
  name: "hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct"
  api_url: "http://localhost:8000/v1"

# Default templates/prompts
KIE_default_template: "Extract the following fields..."
OCR_default_template: "Recognize all text..."
VQA_default_template: "Answer this question..."
CLASSIFICATION_default_template: "Classify this document..."
TABLE_default_template: "Extract this table..."

# Data source configuration
docile:
  annot_path: "/path/to/docile/annot.json"
  annotations_root: "/path/to/docile/annotations"
  pdf_root: "/path/to/docile/pdfs"

handwritten_forms:
  hf_name: "dataset/handwritten-forms"
  test_split: "test"

# Benchmark options
cache_dir: "./docext_benchmark_cache"
max_samples_per_dataset: 100
max_workers: 4
ignore_cache: false
```

--------------------------------

### Start vLLM Server Programmatically

Source: https://github.com/nanonets/docext/blob/main/_autodocs/06-quick-start.md

Instantiate and run a vLLM server in the background using Python. This allows for programmatic control over the server's model, host, port, and other configurations. Ensure to stop the server when done.

```python
from docext.core.vllm import VLLMServer

# Create server instance
server = VLLMServer(
    model_name="hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct",
    host="0.0.0.0",
    port=8000,
    max_model_len=10000,
    gpu_memory_utilization=0.9,
    max_num_imgs=5,
    dtype="bfloat16"
)

# Start in background
thread = server.run_in_background()

# Now make requests to http://localhost:8000

# Cleanup when done
server.stop_server()

```

--------------------------------

### Running Docext with a Cloud Provider (OpenAI)

Source: https://github.com/nanonets/docext/blob/main/_autodocs/04-configuration.md

Example of configuring Docext to use a cloud provider like OpenAI. Requires setting the OPENAI_API_KEY environment variable and specifying the model name.

```bash
export OPENAI_API_KEY="sk-..."
docext \
  --model_name gpt-4-vision \
  --server_port 7860 \
  --no-share
```

--------------------------------

### Extraction Specification Example (Dict and DataFrame)

Source: https://github.com/nanonets/docext/blob/main/_autodocs/03-types.md

Illustrates how to define an extraction specification using both Python dictionaries and a Pandas DataFrame.

```python
# Dict format
spec = {
    "fields": [
        {"name": "invoice_number"},
        {"name": "total_amount"}
    ],
    "tables": [
        {"name": "line_items"}
    ]
}

# Dataframe format
import pandas as pd
spec_df = pd.DataFrame({
    "type": ["field", "field", "table"],
    "name": ["invoice_number", "total_amount", "line_items"],
    "description": ["...", "...", "..."]
})
```

--------------------------------

### VLLMServer.run_in_background

Source: https://github.com/nanonets/docext/blob/main/_autodocs/README.md

Starts the VLLM server in a separate thread and waits for it to become ready.

```APIDOC
## VLLMServer.run_in_background

### Description
Starts the VLLM server in a separate thread and waits for it to become ready.

### Purpose
Start in thread + wait
```

--------------------------------

### Manage VLLM Server

Source: https://github.com/nanonets/docext/blob/main/_autodocs/QUICK-REFERENCE.md

Start, check health, and stop a VLLM server using `VLLMServer`. The server can run in the background or be explicitly controlled.

```python
from docext.core.vllm import VLLMServer
from docext.app.utils import check_vllm_healthcheck

# Start server
server = VLLMServer(
    model_name="hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct",
    port=8000,
    max_model_len=10000,
    gpu_memory_utilization=0.9
)
server.run_in_background()  # Blocks until ready

# Or just start it
server.start_server()
is_ready = server.wait_for_server(timeout=300)

# Health check
is_ready = check_vllm_healthcheck("localhost", 8000)

# Shutdown
server.stop_server()
```

--------------------------------

### Use Smaller Model for Speed

Source: https://github.com/nanonets/docext/blob/main/_autodocs/README.md

Improve processing speed by selecting a smaller model via the --model_name argument. Example uses a Qwen model.

```bash
--model_name hosted_vllm/Qwen/Qwen2.5-VL-3B-Instruct
```

--------------------------------

### Example VLM API Message with Image

Source: https://github.com/nanonets/docext/blob/main/_autodocs/03-types.md

Illustrates a user message containing both text and an image URL, formatted according to the VLM API message structure.

```python
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Extract the invoice number from this document"
            },
            {
                "type": "image_url",
                "image_url": {
                    "url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEA..."
                }
            },
            {
                "type": "text",
                "text": "Return as JSON: {'invoice_number': '...'}"
            }
        ]
    }
]
```

--------------------------------

### Docext Configuration for Multiple Document Processing

Source: https://github.com/nanonets/docext/blob/main/_autodocs/04-configuration.md

Configure Docext for processing multiple documents simultaneously. This example increases the concurrency limit and the maximum number of images per request.

```bash
export VLM_MODEL_URL="http://localhost:8000/v1"
docext \
  --model_name hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct \
  --concurrency_limit 4 \
  --max_num_imgs 5 \
  --max_img_size 2048 \
  --gpu_memory_utilization 0.9
```

--------------------------------

### main

Source: https://github.com/nanonets/docext/blob/main/_autodocs/02-application-ui.md

Entry point that initializes the vLLM server (if needed) and launches the Gradio interface. It configures the model, server host and port, Gradio port, and various processing parameters.

```APIDOC
## Function: main

### Description
Entry point that initializes the vLLM server (if needed) and launches the Gradio interface.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **model_name** (str) - Required - VLM model identifier (hosted_vllm/, ollama/, or cloud provider)
- **host** (str) - Required - Host for vLLM/Ollama server binding
- **port** (int) - Required - Port for vLLM/Ollama server
- **gradio_port** (int) - Required - Port for Gradio web interface (typically 7860)
- **max_model_len** (int) - Required - Maximum context length in tokens
- **gpu_memory_utilization** (float) - Required - GPU memory fraction (0.0-1.0)
- **max_num_imgs** (int) - Required - Maximum images per single model request
- **vllm_start_timeout** (int) - Required - Timeout in seconds for vLLM startup
- **max_img_size** (int) - Required - Target image dimension for resizing (pixels)
- **concurrency_limit** (int) - Required - Maximum concurrent processing requests
- **share** (bool) - Required - If True, disables Gradio sharing (inverted logic)
- **dtype** (str) - Required - Model data type: "bfloat16" or "float16"
- **max_gen_tokens** (int) - Required - Maximum tokens to generate per request

### Request Example
```python
from docext.app.app import main

main(
    model_name="hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct",
    host="127.0.0.1",
    port=8000,
    gradio_port=7860,
    max_model_len=10000,
    gpu_memory_utilization=0.9,
    max_num_imgs=5,
    vllm_start_timeout=300,
    max_img_size=2048,
    concurrency_limit=2,
    share=False,
    dtype="bfloat16",
    max_gen_tokens=10000
)
```

### Response
Success Response (200)
None

#### Response Example
None
```

--------------------------------

### VLLMServer.start_server

Source: https://github.com/nanonets/docext/blob/main/_autodocs/README.md

Launches the VLLM server as a subprocess.

```APIDOC
## VLLMServer.start_server

### Description
Launches the VLLM server as a subprocess.

### Purpose
Launch server subprocess
```

--------------------------------

### Field Extraction Output Example

Source: https://github.com/nanonets/docext/blob/main/_autodocs/03-types.md

An example of the Pandas DataFrame returned after extracting individual fields, showing extracted values and their confidence.

```python
         fields            answer confidence  document_index
0  invoice_number        INV-2025-001      High                0
1    invoice_date       2025-06-24       High                0
2    total_amount          $1,500.00      High                0
```

--------------------------------

### Table Extraction Output Example

Source: https://github.com/nanonets/docext/blob/main/_autodocs/03-types.md

An example of the Pandas DataFrame representing extracted table data, with columns matching the defined table headers.

```python
          Item Description  Quantity  Unit Price
0  Professional Services       2     $500.00
1      Software Licenses       5      $200.00
```

--------------------------------

### Run with Defaults

Source: https://github.com/nanonets/docext/blob/main/_autodocs/README.md

Execute the docext CLI using its default settings. No specific configuration is required.

```bash
docext  # Use defaults
```

--------------------------------

### Initialize NanonetsIDPBenchmark

Source: https://github.com/nanonets/docext/blob/main/_autodocs/05-benchmark-api.md

Instantiate the main benchmark orchestrator by providing the path to the YAML configuration file. This loads datasets, models, and sets up the benchmarking environment.

```python
from docext.benchmark.benchmark import NanonetsIDPBenchmark

benchmark = NanonetsIDPBenchmark(
    benchmark_config_path="configs/benchmark_config.yaml"
)

print(f"Loaded {len(benchmark.datasets)} datasets")
print(f"Models: {list(benchmark.models.keys())}")
```

--------------------------------

### check_ollama_healthcheck

Source: https://github.com/nanonets/docext/blob/main/_autodocs/02-application-ui.md

Checks if the Ollama server is responding by performing an HTTP GET request to the root endpoint.

```APIDOC
## Function: check_ollama_healthcheck

### Description
Checks if Ollama server is responding. Returns True if GET http://{host}:{port} returns status 200.

### Parameters
#### Path Parameters
- **host** (str) - Required - Ollama server host
- **port** (int) - Required - Ollama server port (default 11434)

### Return type
bool
```

--------------------------------

### check_vllm_healthcheck

Source: https://github.com/nanonets/docext/blob/main/_autodocs/02-application-ui.md

Checks if the vLLM server is responding to health requests by performing an HTTP GET request to the health endpoint.

```APIDOC
## Function: check_vllm_healthcheck

### Description
Checks if vLLM server is responding to health requests. Returns True if GET http://{host}:{port}/health returns status 200.

### Parameters
#### Path Parameters
- **host** (str) - Required - vLLM server host
- **port** (int) - Required - vLLM server port

### Return type
bool

### Example
```python
from docext.app.utils import check_vllm_healthcheck

is_ready = check_vllm_healthcheck("127.0.0.1", 8000)
if is_ready:
    print("Server is ready")
else:
    print("Server not responding")
```
```

--------------------------------

### NanonetsIDPBenchmark.__init__

Source: https://github.com/nanonets/docext/blob/main/_autodocs/05-benchmark-api.md

Initializes the NanonetsIDPBenchmark orchestrator. It loads datasets, models, and configurations necessary for running benchmark experiments.

```APIDOC
## NanonetsIDPBenchmark.__init__

### Description
Initializes the main benchmark orchestrator that loads datasets, runs models, and evaluates performance based on a provided configuration file.

### Method
__init__

### Parameters
#### Path Parameters
- **benchmark_config_path** (str) - Required - Absolute path to YAML configuration file

### Request Example
```python
from docext.benchmark.benchmark import NanonetsIDPBenchmark

benchmark = NanonetsIDPBenchmark(
    benchmark_config_path="configs/benchmark_config.yaml"
)

print(f"Loaded {len(benchmark.datasets)} datasets")
print(f"Models: {list(benchmark.models.keys())}")
```
```

--------------------------------

### Run Docext with Custom Configuration Parameters

Source: https://github.com/nanonets/docext/blob/main/_autodocs/04-configuration.md

Execute the docext command with specific configuration flags to customize server ports, model details, memory utilization, and logging levels for local execution.

```bash
docext \
  --vlm_server_port 8000 \
  --vlm_server_host 127.0.0.1 \
  --model_name hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct-AWQ \
  --server_port 7860 \
  --max_model_len 15000 \
  --gpu_memory_utilization 0.9 \
  --max_num_imgs 5 \
  --vllm_start_timeout 300 \
  --log_level debug \
  --max_img_size 2048 \
  --concurrency_limit 1 \
  --dtype bfloat16 \
  --max_gen_tokens 10000
```

--------------------------------

### Default CLI Execution

Source: https://github.com/nanonets/docext/blob/main/_autodocs/QUICK-REFERENCE.md

Run docext with default settings.

```bash
# Default run
docext
```

--------------------------------

### Console Command Entry Point

Source: https://github.com/nanonets/docext/blob/main/_autodocs/07-module-index.md

Use this command to run the docext application from the console.

```bash
docext → docext.app.__main__:main
```

--------------------------------

### Run Docext with OpenRouter API Key for Vendor-Hosted Models

Source: https://github.com/nanonets/docext/blob/main/EXT_README.md

This command is for deploying Docext when using vendor-hosted models via OpenRouter. Set your OpenRouter API key as an environment variable to authenticate.

```bash
docker run --rm \
  --env "OPENROUTER_API_KEY=<secret>" \
  --network host \
  nanonetsopensource/docext:v0.1.10 --model_name "openrouter/meta-llama/llama-4-maverick:free"
```

--------------------------------

### Table Metrics Output

Source: https://github.com/nanonets/docext/blob/main/_autodocs/05-benchmark-api.md

Example JSON output for Table task metrics. Includes structure accuracy, cell accuracy, and header accuracy.

```python
{
    "structure_accuracy": 0.88,
    "cell_accuracy": 0.91,
    "header_accuracy": 0.95
}
```

--------------------------------

### CLI with Custom Ports

Source: https://github.com/nanonets/docext/blob/main/_autodocs/QUICK-REFERENCE.md

Configure custom server ports for docext.

```bash
# Custom ports
docext --server_port 8080 --vlm_server_port 8001
```

--------------------------------

### Host Default vLLM Model Locally

Source: https://github.com/nanonets/docext/blob/main/EXT_README.md

Downloads and hosts the default Qwen/Qwen2.5-VL-7B-Instruct-AWQ model locally using vLLM on port 8000.

```bash
python -m docext.app.app
```

--------------------------------

### KIE Metrics Output

Source: https://github.com/nanonets/docext/blob/main/_autodocs/05-benchmark-api.md

Example JSON output for Key Information Extraction (KIE) task metrics. Includes score and average similarity.

```python
{
    "score": 0.94,  # float (0.0-1.0)
    "avg_similarity": 0.94
}
```

--------------------------------

### Get Dataset Classes

Source: https://github.com/nanonets/docext/blob/main/_autodocs/05-benchmark-api.md

Retrieves uninstantiated dataset classes based on task and dataset names. Use this to load specific datasets for benchmarking.

```python
def get_datasets(
    tasks: list[str],
    datasets: list[str]
) -> list[type]:
```

--------------------------------

### Basic Docext Usage

Source: https://github.com/nanonets/docext/blob/main/_autodocs/04-configuration.md

The primary way to run Docext is via the command line. You can execute it directly or through the python -m module.

```bash
docext [OPTIONS]
python -m docext [OPTIONS]
```

--------------------------------

### docext.__main__ Entry Point

Source: https://github.com/nanonets/docext/blob/main/_autodocs/07-module-index.md

Details the main entry point for the docext console script.

```APIDOC
## Module: docext.__main__

### Description
Provides the main entry point for the console script.

### Function
- `main()` — Entry point for console script. Calls `parse_args()` and invokes `docext.app.app.main()` with the parsed arguments.
```

--------------------------------

### Classification Metrics Output

Source: https://github.com/nanonets/docext/blob/main/_autodocs/05-benchmark-api.md

Example JSON output for Classification task metrics. Includes overall accuracy, precision, recall, F1 score, and per-class metrics.

```python
{
    "accuracy": 0.92,
    "precision": 0.90,
    "recall": 0.91,
    "f1": 0.90,
    "per_class": {
        "invoice": {"precision": 0.95, "recall": 0.92},
        "receipt": {"precision": 0.88, "recall": 0.90},
        ...
    }
}
```

--------------------------------

### Run Docext with Hugging Face Token and Local Model

Source: https://github.com/nanonets/docext/blob/main/EXT_README.md

Use this command to run Docext with a Hugging Face token for authentication and a locally specified model. Ensure your Hugging Face token is set as an environment variable and adjust GPU allocation as needed.

```bash
docker run --rm \
  --env "HUGGING_FACE_HUB_TOKEN=<secret>" \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --network host \
  --shm-size=20.24gb \
  --gpus all \
  nanonetsopensource/docext:v0.1.10 --model_name "hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct-AWQ"
```

--------------------------------

### CLI with GPU and Memory Settings

Source: https://github.com/nanonets/docext/blob/main/_autodocs/QUICK-REFERENCE.md

Control GPU utilization and maximum model length.

```bash
# GPU/Memory
docext --gpu_memory_utilization 0.95 --max_model_len 15000
```

--------------------------------

### Check Ollama Server Health

Source: https://github.com/nanonets/docext/blob/main/_autodocs/02-application-ui.md

Verifies if an Ollama server is operational by sending an HTTP GET request to its host and port. The default port for Ollama is 11434.

```python
def check_ollama_healthcheck(host: str, port: int) -> bool:

    Checks if Ollama server is responding.

    Parameters:

    | Parameter | Type | Required | Default | Description |
    |-----------|------|----------|---------|-------------|
    | host | str | Yes | — | Ollama server host |
    | port | int | Yes | — | Ollama server port (default 11434) |

    Return type: bool

    Returns True if GET http://{host}:{port} returns status 200.
```

--------------------------------

### Low-Memory Device Configuration

Source: https://github.com/nanonets/docext/blob/main/_autodocs/README.md

Configure Docext for devices with limited memory by adjusting image size and model length.

```bash
docext --max_img_size 1024 --max_model_len 8000 --gpu_memory_utilization 0.7
```

--------------------------------

### Initialize VLLMServer

Source: https://github.com/nanonets/docext/blob/main/_autodocs/01-core-api-reference.md

Instantiates a VLLMServer object to manage a local vLLM inference server. Configure model, host, port, and other parameters during initialization.

```python
class VLLMServer:
    def __init__(
        self,
        model_name: str,
        host: str = "0.0.0.0",
        port: int = 8000,
        max_model_len: int = 15000,
        gpu_memory_utilization: float = 0.98,
        max_num_imgs: int = 5,
        vllm_start_timeout: int = 300,
        dtype: str = "bfloat16",
    )
```

--------------------------------

### Get Numeric Confidence Score Messages

Source: https://github.com/nanonets/docext/blob/main/_autodocs/QUICK-REFERENCE.md

Generates messages for numeric confidence scoring (0-100) for specified fields. Requires original messages and an assistant response.

```python
# Numeric confidence (0-100)
msgs = get_fields_confidence_score_messages_numeric(
    messages=original_messages,
    assistant_response=extraction_response,
    fields=["invoice_number"]
)
```

--------------------------------

### Get Binary Confidence Score Messages

Source: https://github.com/nanonets/docext/blob/main/_autodocs/QUICK-REFERENCE.md

Generates messages for binary confidence scoring (High/Low) for specified fields. Requires original messages and an assistant response.

```python
from docext.core.confidence import (
    get_fields_confidence_score_messages_binary,
    get_fields_confidence_score_messages_numeric
)
from docext.core.client import sync_request

# After getting extraction response
extraction_response = '{"invoice_number": "INV-001"}'

# Binary confidence (High/Low)
msgs = get_fields_confidence_score_messages_binary(
    messages=original_messages,
    assistant_response=extraction_response,
    fields=["invoice_number"]
)
conf_response = sync_request(msgs, "hosted_vllm/Qwen/...")
```

--------------------------------

### Host Specific vLLM Model on Custom Port

Source: https://github.com/nanonets/docext/blob/main/EXT_README.md

Downloads and hosts the Qwen/Qwen2.5-VL-32B-Instruct-AWQ model locally using vLLM on port 9000.

```bash
python -m docext.app.app --model_name hosted_vllm/Qwen/Qwen2.5-VL-32B-Instruct-AWQ --vlm_server_port 9000
```

--------------------------------

### VQA Metrics Output

Source: https://github.com/nanonets/docext/blob/main/_autodocs/05-benchmark-api.md

Example JSON output for Visual Question Answering (VQA) task metrics. Includes accuracy, exact match, and partial match scores.

```python
{
    "accuracy": 0.75,
    "exact_match": 0.68,
    "partial_match": 0.92
}
```

--------------------------------

### Set Default Configuration with Environment Variables

Source: https://github.com/nanonets/docext/blob/main/_autodocs/04-configuration.md

Configure Docext for local development by setting environment variables for the VLM model URL and API key. This is equivalent to running docext with no arguments.

```bash
# Equivalent to running 'docext' with no arguments
export VLM_MODEL_URL="http://127.0.0.1:8000/v1"
export API_KEY="EMPTY"
```

--------------------------------

### Check vLLM Server Health

Source: https://github.com/nanonets/docext/blob/main/_autodocs/02-application-ui.md

Use this function to determine if a vLLM server is responding to health requests. It sends an HTTP GET request to the specified host and port.

```python
from docext.app.utils import check_vllm_healthcheck

is_ready = check_vllm_healthcheck("127.0.0.1", 8000)
if is_ready:
    print("Server is ready")
else:
    print("Server not responding")
```

--------------------------------

### Import Benchmarking Modules

Source: https://github.com/nanonets/docext/blob/main/_autodocs/QUICK-REFERENCE.md

Import classes and functions for benchmarking and calculating KIE metrics.

```python
from docext.benchmark.benchmark import NanonetsIDPBenchmark
from docext.benchmark.metrics.kie import get_kie_metrics
```

--------------------------------

### CLI with Image Size Setting

Source: https://github.com/nanonets/docext/blob/main/_autodocs/QUICK-REFERENCE.md

Adjust the maximum image size for processing. Higher values improve quality but decrease speed.

```bash
# Image size (quality vs speed)
docext --max_img_size 2048  # Higher = better quality but slower
```

--------------------------------

### Set API Key for Cloud Providers

Source: https://github.com/nanonets/docext/blob/main/_autodocs/04-configuration.md

Set the API_KEY environment variable for authentication with cloud VLM providers or protected endpoints. Examples are provided for OpenAI and Together AI.

```bash
export API_KEY="sk-..."        # OpenAI
```

```bash
export API_KEY="glhf_..."      # Together AI
```

```bash
export API_KEY="your-token"    # Custom deployment
```

--------------------------------

### CLI with Custom Model

Source: https://github.com/nanonets/docext/blob/main/_autodocs/QUICK-REFERENCE.md

Specify a custom model for docext to use.

```bash
# Custom model
docext --model_name ollama/mistral
```

--------------------------------

### Load and Inspect Benchmark Datasets and Models

Source: https://github.com/nanonets/docext/blob/main/_autodocs/QUICK-REFERENCE.md

Loads a benchmark configuration and iterates through its datasets to print their names and sample counts. Also prints available model keys.

```python
from docext.benchmark.benchmark import NanonetsIDPBenchmark
from docext.benchmark.metrics.kie import get_kie_metrics

# Load benchmark
benchmark = NanonetsIDPBenchmark("configs/benchmark_config.yaml")

# Datasets
for dataset in benchmark.datasets:
    print(f"{dataset.name}: {len(dataset.data)} samples")

# Models
print(benchmark.models.keys())
```

--------------------------------

### Run vLLM Server in Background Thread

Source: https://github.com/nanonets/docext/blob/main/_autodocs/01-core-api-reference.md

Starts the vLLM server in a daemon thread and waits for it to become ready. This is useful for integrating the server into existing applications without blocking the main thread.

```python
server = VLLMServer(model_name="hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct")
thread = server.run_in_background()
# Server is now ready for requests at the configured host/port
```

--------------------------------

### OCR Metrics Output

Source: https://github.com/nanonets/docext/blob/main/_autodocs/05-benchmark-api.md

Example JSON output for Optical Character Recognition (OCR) task metrics. Includes Character Error Rate (CER), Word Error Rate (WER), and accuracy.

```python
{
    "cer": 0.05,     # Character Error Rate
    "wer": 0.12,     # Word Error Rate
    "accuracy": 0.88
}
```

--------------------------------

### Direct Module Invocation Entry Point

Source: https://github.com/nanonets/docext/blob/main/_autodocs/07-module-index.md

Run the docext application directly as a Python module.

```python
python -m docext → docext.__main__.py
```

--------------------------------

### JSON Schema for Field Extraction

Source: https://github.com/nanonets/docext/blob/main/_autodocs/03-types.md

An example JSON schema specifically for extracting fields like invoice number, date, and total amount. Ensure the 'properties' match the fields you need to extract.

```python
format_fields = {
    "type": "object",
    "properties": {
        "invoice_number": {"type": "string"},
        "invoice_date": {"type": "string"},
        "total_amount": {"type": "string"}
    }
}
```

--------------------------------

### Get KIE Metrics

Source: https://github.com/nanonets/docext/blob/main/_autodocs/05-benchmark-api.md

Computes Key Information Extraction metrics using normalized edit distance. Returns an average similarity score between 0.0 and 1.0. Use this for evaluating the accuracy of extracted information.

```python
from docext.benchmark.metrics.kie import get_kie_metrics

score = get_kie_metrics(predictions)
print(f"KIE Score: {score:.2%}")  # e.g., KIE Score: 94.23%
```

--------------------------------

### Connect to Existing vLLM Server

Source: https://github.com/nanonets/docext/blob/main/EXT_README.md

Connects to an already running vLLM server on a specified IP and port. Requires setting the API_KEY environment variable if authentication is used.

```bash
export API_KEY=<your_api_key>
python -m docext.app.app --model_name hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct-AWQ --vlm_server_host <your_ip> --vlm_server_port <your_port>
```

--------------------------------

### Specify Model Name

Source: https://github.com/nanonets/docext/blob/main/_autodocs/04-configuration.md

Choose the vision-language model. Prefixes indicate local (hosted_vllm/, ollama/) or remote (cloud provider names) deployment.

```bash
# Use Qwen model locally
docext --model_name hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct

# Use Ollama
docext --model_name ollama/mistral

# Use OpenAI
docext --model_name gpt-4-vision

# Use specialized OCR model
docext --model_name hosted_vllm/nanonets/Nanonets-OCR-s
```

--------------------------------

### Synchronous VLM Request with Structured Output

Source: https://github.com/nanonets/docext/blob/main/_autodocs/01-core-api-reference.md

Use `sync_request` to make a synchronous call to a VLM endpoint. This example demonstrates extracting an invoice number using a JSON schema for structured output. Ensure the `VLM_MODEL_URL` environment variable is set.

```python
from docext.core.client import sync_request

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Extract the invoice number from this document"},
            {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}
        ]
    }
]

response = sync_request(
    messages=messages,
    model_name="hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct",
    max_tokens=500,
    format={"type": "object", "properties": {"invoice_number": {"type": "string"}}}
)

extracted_text = response["choices"][0]["message"]["content"]
```

--------------------------------

### NanonetsIDPBenchmark._get_datasets

Source: https://github.com/nanonets/docext/blob/main/_autodocs/05-benchmark-api.md

Creates and initializes benchmark datasets based on the loaded configuration. It handles the specific parameter requirements for various dataset types.

```APIDOC
## NanonetsIDPBenchmark._get_datasets

### Description
Creates and initializes benchmark datasets based on configuration. This method ensures that each dataset is properly set up with the parameters required for its specific type.

### Method
_get_datasets

### Return type
list[BenchmarkDataset]

Returns initialized dataset objects with all required parameters from config.

**Dataset Initialization:**
Different datasets require different parameters:

- **docile**: docile_annot_path, annotations_root, pdf_root
- **handwritten_forms**: hf_name, test_split
- **ocr_handwriting**: hf_name, test_split
- **ocr_handwriting_rotated**: hf_name, test_split
- **docvqa**: hf_name, test_split
- **chartqa**: hf_name, test_split
- **checkbox**: hf_name, test_split
- **longdocbench**: hf_name, test_split
- **nanonets_kie**: hf_name
- **nanonets_cls**: hf_name
- **nanonets_tablebench**: hf_name
```

--------------------------------

### Model Name Prefixes for Deployment

Source: https://github.com/nanonets/docext/blob/main/_autodocs/03-types.md

Identifiers for models, indicating the deployment method. Use the appropriate prefix for local servers (vLLM, Ollama) or cloud providers.

```python
# Local vLLM server (self-hosted)
"hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct"
"hosted_vllm/nanonets/Nanonets-OCR-s"

# Local Ollama server (https://ollama.com)
"ollama/mistral"
"ollama/llama2"

# Cloud providers (OpenAI, Google, etc.)
"gpt-4-vision"
"gemini-pro-vision"
"claude-3-sonnet"

# OpenRouter API
"openrouter/...[model-name]"
```

--------------------------------

### Running Docext with Ollama (Local)

Source: https://github.com/nanonets/docext/blob/main/_autodocs/04-configuration.md

Configure Docext to use a local Ollama server. Ensure the Ollama server is running on localhost:11434 before executing this command.

```bash
# Ollama server should be running on localhost:11434
docext \
  --model_name ollama/mistral \
  --vlm_server_host localhost \
  --vlm_server_port 11434 \
  --server_port 7860 \
  --max_img_size 1536
```

--------------------------------

### CLI Performance Settings: High Quality

Source: https://github.com/nanonets/docext/blob/main/_autodocs/QUICK-REFERENCE.md

Achieve high processing quality by using a larger image size and allocating more GPU memory.

```bash
--max_img_size 2048 --gpu_memory_utilization 0.95
```