### Example: Encoding Invoice Bounding Boxes

Source: https://context7.com/dswang2011/docllm/llms.txt

Demonstrates how to use the BoundingBoxEmbedding module with sample invoice data. Boxes are normalized to page dimensions.

```python
# --- Example: encoding bounding boxes from an invoice page ---
# Boxes are normalized: [x0, y0, x1, y1] with page width/height = 1.0
boxes = torch.tensor([[[
    0.05, 0.10, 0.45, 0.15,   # "Invoice Number:" label
    0.50, 0.10, 0.90, 0.15,   # "INV-2024-00123" value
    0.05, 0.20, 0.30, 0.25,   # "Date:" label
    0.35, 0.20, 0.65, 0.25,   # "2024-03-15" value
]]]).view(1, 4, 4)             # batch=1, seq_len=4, 4 coords
box_encoder = BoundingBoxEmbedding(d_model=512)
spatial_emb  = bbox_encoder(boxes)
print(spatial_emb.shape)  # torch.Size([1, 4, 512])
```

--------------------------------

### Example NLI Table Fact Verification

Source: https://context7.com/dswang2011/docllm/llms.txt

This example demonstrates how to use DocLLM for table fact verification. It shows the expected output for a given statement and table data.

```python
table_tokens = [
    "Quarter", "Q1",     "Q2",     "Q3",     "Q4",
    "Revenue", "$2.1M",  "$2.4M",  "$2.8M",  "$3.1M",
    "Growth",  "+5%",    "+14%",   "+17%",   "+11%",
]

statement = "Revenue in Q3 was higher than in Q2."

# verdict, conf = docllm_nli_tabfact(table_tokens, bboxes, statement, model, tokenizer)
# Expected: verdict="entailed", confidence≈0.94  ($2.8M > $2.4M)
print(f"Statement: {statement}")
print("Verdict: entailed")
print("Confidence: 0.94")
```

--------------------------------

### Run DocLLM Inference for Document VQA

Source: https://context7.com/dswang2011/docllm/llms.txt

Use this function to get answers to natural language questions about a document. It requires OCR tokens, bounding boxes, the question, and the DocLLM model and tokenizer.

```python
def docllm_vqa_inference(ocr_tokens, bboxes, question, model,
                          tokenizer, max_new_tokens=64):
    """
    Run DocLLM inference for a document VQA query.

    Args:
        ocr_tokens:     List[str]  — OCR text tokens extracted from the document
        bboxes:         List[List[float]] — Normalized [x0,y0,x1,y1] per token
        question:       str  — Natural language question about the document
        model:          DocLLM model instance
        tokenizer:      Tokenizer compatible with DocLLM
        max_new_tokens: int  — Maximum answer length in tokens

    Returns:
        answer: str — Model-generated answer
    """
    # Build instruction prompt (following DocLLM fine-tuning format)
    prompt = (
        f"### Instruction:\nAnswer the question based on the document.\n\n"
        f"### Question:\n{question}\n\n"
        f"### Document Tokens:\n{' '.join(ocr_tokens)}\n\n"
        f"### Answer:\n"
    )

    # Tokenize prompt + document tokens
    enc = tokenizer(prompt, return_tensors="pt")
    input_ids   = enc["input_ids"]
    bbox_tensor = build_bbox_tensor(ocr_tokens, bboxes, tokenizer)
    # bbox_tensor shape: (1, seq_len, 4)

    # Generate answer autoregressively
    with torch.no_grad():
        output_ids = model.generate(
            input_ids=input_ids,
            bbox=bbox_tensor,
            max_new_tokens=max_new_tokens,
            do_sample=False,          # greedy decoding
            temperature=1.0,
        )

    answer = tokenizer.decode(output_ids[0][input_ids.shape[1]:],
                               skip_special_tokens=True)
    return answer.strip()


# --- Example ---
ocr_tokens = ["Invoice", "Number", ":", "INV-2024-00123",
              "Date", ":", "March", "15", "2024",
              "Total", "Amount", ":", "$1,234.56"]

bboxes = [
    [0.05, 0.10, 0.20, 0.15], [0.21, 0.10, 0.35, 0.15],
    [0.35, 0.10, 0.38, 0.15], [0.39, 0.10, 0.75, 0.15],
    [0.05, 0.20, 0.12, 0.25], [0.13, 0.20, 0.16, 0.25],
    [0.17, 0.20, 0.30, 0.25], [0.31, 0.20, 0.38, 0.25],
    [0.39, 0.20, 0.55, 0.25], [0.05, 0.30, 0.12, 0.35],
    [0.13, 0.30, 0.25, 0.35], [0.26, 0.30, 0.29, 0.35],
    [0.30, 0.30, 0.50, 0.35],
]

question = "What is the total amount on this invoice?"

# answer = docllm_vqa_inference(ocr_tokens, bboxes, question, model, tokenizer)
# Expected output: "$1,234.56"
print("Query:", question)
print("Expected answer: $1,234.56")
```

--------------------------------

### Create Infilling Batch for DocLLM Pre-training

Source: https://context7.com/dswang2011/docllm/llms.txt

Prepares a training batch for DocLLM's text infilling objective by randomly masking contiguous text segments. Ensure token_ids and bbox_coords are properly formatted lists.

```python
import torch
import torch.nn.functional as F

def create_infilling_batch(token_ids: list[list[int]],
                           bbox_coords: list[list[list[float]]],
                           mask_ratio: float = 0.15,
                           pad_id: int = 0,
                           mask_id: int = 1):
    """
    Prepares a training batch for DocLLM's text infilling objective.
    Randomly masks contiguous text segments and returns inputs/labels.

    Args:
        token_ids:   List of token id sequences per document
        bbox_coords: Corresponding bounding boxes per token [x0,y0,x1,y1]
        mask_ratio:  Fraction of tokens to mask
        pad_id:      Padding token id
        mask_id:     [MASK] sentinel token id

    Returns:
        input_ids:   Masked token sequences (batch, max_len)
        bbox_tensor: Bounding box tensor    (batch, max_len, 4)
        labels:      Original ids at masked positions, -100 elsewhere
    """
    import random, torch
    batch_inputs, batch_boxes, batch_labels = [], [], []

    for ids, boxes in zip(token_ids, bbox_coords):
        n = len(ids)
        span_len = max(1, int(n * mask_ratio))
        start = random.randint(0, n - span_len)

        masked_ids = ids[:]
        labels     = [-100] * n
        for i in range(start, start + span_len):
            labels[i]     = ids[i]       # supervise only masked positions
            masked_ids[i] = mask_id

        batch_inputs.append(masked_ids)
        batch_boxes.append(boxes)
        batch_labels.append(labels)

    # Pad to same length
    max_len = max(len(s) for s in batch_inputs)
    def pad(seq, val): return seq + [val] * (max_len - len(seq))

    input_tensor = torch.tensor([pad(s, pad_id) for s in batch_inputs])
    label_tensor = torch.tensor([pad(l, -100)   for l in batch_labels])
    box_tensor   = torch.zeros(len(batch_inputs), max_len, 4)
    for i, boxes in enumerate(batch_boxes):
        for j, b in enumerate(boxes):
            box_tensor[i, j] = torch.tensor(b)

    return input_tensor, box_tensor, label_tensor


# --- Example usage ---
token_ids = [
    [101, 2023, 2003, 1037, 3231, 102],          # "This is an invoice"
    [101, 4769, 1024, 3211, 2358, 2102, 102],     # "Address: 123 Main St"
]
bbox_coords = [
    [[0.1,0.1,0.4,0.15],[0.4,0.1,0.6,0.15],[0.6,0.1,0.7,0.15],
     [0.7,0.1,0.8,0.15],[0.8,0.1,0.9,0.15],[0.9,0.1,1.0,0.15]],
    [[0.1,0.2,0.3,0.25],[0.3,0.2,0.5,0.25],[0.5,0.2,0.6,0.25],
     [0.6,0.2,0.7,0.25],[0.7,0.2,0.8,0.25],[0.8,0.2,0.9,0.25],
     [0.9,0.2,1.0,0.25]],
]

inputs, boxes, labels = create_infilling_batch(token_ids, bbox_coords, mask_ratio=0.3)
print("inputs:", inputs)
print("labels:", labels)   # -100 at unmasked positions, original id at masked positions
print("boxes shape:", boxes.shape)  # torch.Size([2, 7, 4])
```

--------------------------------

### DocLLM Inference for Document VQA (Pseudocode)

Source: https://context7.com/dswang2011/docllm/llms.txt

Illustrates the inference process for Document Visual Question Answering using DocLLM. This is pseudocode; actual implementation requires loading model weights and tokenizer.

```python
# Pseudocode illustrating DocLLM inference for document VQA.
# In practice, the model weights and tokenizer would be loaded from
```

--------------------------------

### Bounding Box Embedding Module

Source: https://context7.com/dswang2011/docllm/llms.txt

Encodes normalized bounding box coordinates into a dense spatial embedding. Requires PyTorch and nn.Module. The input boxes should be normalized to [0, 1].

```python
import torch
import torch.nn as nn

class BoundingBoxEmbedding(nn.Module):
    """
    Encodes normalized bounding box coordinates (x0, y0, x1, y1, width, height)
    into a dense spatial embedding compatible with DocLLM's spatial attention stream.
    """
    def __init__(self, d_model: int, max_position: int = 1024):
        super().__init__()
        # Separate embedding tables for each of the 6 spatial features
        self.x_emb  = nn.Embedding(max_position, d_model // 6)
        self.y_emb  = nn.Embedding(max_position, d_model // 6)
        self.x1_emb = nn.Embedding(max_position, d_model // 6)
        self.y1_emb = nn.Embedding(max_position, d_model // 6)
        self.w_emb  = nn.Embedding(max_position, d_model // 6)
        self.h_emb  = nn.Embedding(max_position, d_model // 6)
        self.proj   = nn.Linear((d_model // 6) * 6, d_model)

    def forward(self, boxes: torch.Tensor) -> torch.Tensor:
        """
        Args:
            boxes: (batch, seq_len, 4) — normalized [x0, y0, x1, y1] in [0, 1]
        Returns:
            spatial_emb: (batch, seq_len, d_model)
        """
        scale = 1023  # map [0,1] → [0, 1023]
        x0 = (boxes[..., 0] * scale).long().clamp(0, 1023)
        y0 = (boxes[..., 1] * scale).long().clamp(0, 1023)
        x1 = (boxes[..., 2] * scale).long().clamp(0, 1023)
        y1 = (boxes[..., 3] * scale).long().clamp(0, 1023)
        w  = ((boxes[..., 2] - boxes[..., 0]) * scale).long().clamp(0, 1023)
        h  = ((boxes[..., 3] - boxes[..., 1]) * scale).long().clamp(0, 1023)

        spatial = torch.cat([
            self.x_emb(x0), self.y_emb(y0),
            self.x1_emb(x1), self.y1_emb(y1),
            self.w_emb(w),   self.h_emb(h)
        ], dim=-1)
        return self.proj(spatial)
```

--------------------------------

### Verify Statement Against Table with DocLLM

Source: https://context7.com/dswang2011/docllm/llms.txt

Use this function for Natural Language Inference on tabular documents. It verifies if a statement is entailed or refuted by the table contents. Requires table tokens, bounding boxes, the statement, a DocLLM model, and a tokenizer.

```python
def docllm_nli_tabfact(table_tokens, bboxes, statement, model, tokenizer):
    """
    Verify whether a natural language statement is entailed or refuted
    by the contents of a document table.

    Args:
        table_tokens: List[str]        — OCR tokens of the table cells
        bboxes:       List[List[float]]— Normalized bounding boxes per token
        statement:    str              — Claim to verify
        model:        DocLLM model
        tokenizer:    Compatible tokenizer

    Returns:
        verdict:      str   — "entailed" or "refuted"
        confidence:   float — Model confidence
    """
    prompt = (
        f"### Instruction:\nDoes the table support or refute the following "
        f'statement? Answer with "entailed" or "refuted".\n\n'
        f"### Statement:\n{statement}\n\n"
        f"### Table Tokens:\n{' '.join(table_tokens)}\n\n"
        f"### Verdict:\n"
    )

    enc = tokenizer(prompt, return_tensors="pt")
    input_ids   = enc["input_ids"]
    bbox_tensor = build_bbox_tensor(table_tokens, bboxes, tokenizer)

    with torch.no_grad():
        outputs = model(input_ids=input_ids, bbox=bbox_tensor)
        logits  = outputs.logits[:, -1, :]

    import torch
    entailed_id = tokenizer.encode("entailed", add_special_tokens=False)[0]
    refuted_id  = tokenizer.encode("refuted",  add_special_tokens=False)[0]

    scores = torch.tensor([logits[0, entailed_id], logits[0, refuted_id]])
    probs  = torch.softmax(scores, dim=0)
    verdict = "entailed" if probs[0] > probs[1] else "refuted"

    return verdict, probs.max().item()


# --- Example: TabFact-style verification ---
```

--------------------------------

### Classify Document Type with DocLLM

Source: https://context7.com/dswang2011/docllm/llms.txt

Use this function to classify a document into one of the provided candidate categories based on OCR tokens and bounding boxes. Ensure you have the DocLLM model and a compatible tokenizer.

```python
def docllm_classify(ocr_tokens, bboxes, candidate_labels, model, tokenizer):
    """
    Classify a document into one of the provided candidate categories.

    Args:
        ocr_tokens:       List[str]   — OCR text tokens
        bboxes:           List[List[float]] — Normalized bounding boxes
        candidate_labels: List[str]   — Possible document types
        model:            DocLLM model
        tokenizer:        Compatible tokenizer

    Returns:
        predicted_class: str  — Most likely document type
        confidence:      float — Softmax probability of top class (approximate)
    """
    labels_str = ", ".join(candidate_labels)
    prompt = (
        f"### Instruction:\nClassify this document into one of the following "
        f"categories: {labels_str}.\n\n"
        f"### Document Tokens:\n{' '.join(ocr_tokens)}\n\n"
        f"### Category:\n"
    )

    enc = tokenizer(prompt, return_tensors="pt")
    input_ids   = enc["input_ids"]
    bbox_tensor = build_bbox_tensor(ocr_tokens, bboxes, tokenizer)

    with torch.no_grad():
        outputs = model(input_ids=input_ids, bbox=bbox_tensor)
        logits  = outputs.logits[:, -1, :]  # next-token logits

    import torch
    label_scores = {}
    for label in candidate_labels:
        label_token_id = tokenizer.encode(label, add_special_tokens=False)[0]
        label_scores[label] = logits[0, label_token_id].item()

    import torch
    scores_tensor = torch.tensor(list(label_scores.values()))
    probs = torch.softmax(scores_tensor, dim=0)

    best_idx        = probs.argmax().item()
    predicted_class = candidate_labels[best_idx]
    confidence      = probs[best_idx].item()

    return predicted_class, confidence
```

```python
# --- Example: classifying an RVL-CDIP document ---
candidate_labels = [
    "letter", "form", "email", "handwritten", "advertisement",
    "scientific report", "scientific publication", "specification",
    "file folder", "news article", "budget", "invoice",
    "presentation", "questionnaire", "resume", "memo"
]

ocr_tokens = ["MEMORANDUM", "TO", ":", "All", "Staff",
              "FROM", ":", "HR", "Department",
              "DATE", ":", "March", "15", "2024",
              "RE", ":", "Updated", "Leave", "Policy"]

# predicted, conf = docllm_classify(ocr_tokens, bboxes, candidate_labels, model, tokenizer)
# Expected: predicted="memo", confidence≈0.91
print("Predicted class: memo")
print("Confidence: 0.91")
```

--------------------------------

### Extract Key-Value Pairs from Document using DocLLM

Source: https://context7.com/dswang2011/docllm/llms.txt

This function extracts specified fields (keys) and their corresponding values from a document, returning them as a JSON object. It requires OCR tokens, bounding boxes, a schema of keys to extract, and the DocLLM model and tokenizer.

```python
import json

def docllm_kie_inference(ocr_tokens, bboxes, schema, model,
                          tokenizer, max_new_tokens=256):
    """
    Extract key-value pairs from a document given a target schema.

    Args:
        ocr_tokens: List[str]        — OCR tokens from document
        bboxes:     List[List[float]]— Normalized bounding boxes per token
        schema:     List[str]        — Keys to extract (e.g. ["vendor", "total"])
        model:      DocLLM model
        tokenizer:  Compatible tokenizer

    Returns:
        extracted: dict — Extracted key-value pairs
    """
    schema_str = ", ".join(schema)
    prompt = (
        f"### Instruction:\nExtract the following fields from the document: "
        f"{schema_str}. Return a JSON object.\n\n"
        f"### Document Tokens:\n{' '.join(ocr_tokens)}\n\n"
        f"### Extracted Fields:\n"
    )

    enc = tokenizer(prompt, return_tensors="pt")
    input_ids   = enc["input_ids"]
    bbox_tensor = build_bbox_tensor(ocr_tokens, bboxes, tokenizer)

    with torch.no_grad():
        output_ids = model.generate(
            input_ids=input_ids,
            bbox=bbox_tensor,
            max_new_tokens=max_new_tokens,
            do_sample=False,
        )

    raw = tokenizer.decode(output_ids[0][input_ids.shape[1]:],
                            skip_special_tokens=True)
    try:
        extracted = json.loads(raw)
    except json.JSONDecodeError:
        extracted = {"raw_output": raw}
    return extracted


# --- Example: KIE on a receipt (CORD-style) ---
ocr_tokens = ["STARBUCKS", "Store", "#12345",
              "Grande", "Latte",    "4.75",
              "Blueberry", "Muffin", "3.25",
              "Subtotal",  "8.00",
              "Tax",       "0.72",
              "Total",     "8.72",
              "Thank", "you", "!"]

schema = ["store_name", "store_id", "line_items", "subtotal", "tax", "total"]

# result = docllm_kie_inference(ocr_tokens, bboxes, schema, model, tokenizer)

```

--------------------------------

### Disentangled Spatial Attention Mechanism in Python

Source: https://context7.com/dswang2011/docllm/llms.txt

Illustrates DocLLM's disentangled attention mechanism where each token has both text and spatial embeddings. This module decomposes attention into four components to capture cross-modal interactions without vision encoders. Requires PyTorch.

```python
import torch
import torch.nn as nn

class DisentangledSpatialAttention(nn.Module):
    """
    Simplified illustration of DocLLM's disentangled attention.
    Each token has a text embedding (t) and a spatial/bbox embedding (s).
    Attention score = t_i·t_j^T + t_i·s_j^T + s_i·t_j^T + s_i·s_j^T
    """
    def __init__(self, d_model: int, n_heads: int):
        super().__init__()
        self.d_head = d_model // n_heads
        self.n_heads = n_heads
        # Projections for text queries/keys
        self.W_qt = nn.Linear(d_model, d_model)
        self.W_kt = nn.Linear(d_model, d_model)
        # Projections for spatial queries/keys
        self.W_qs = nn.Linear(d_model, d_model)
        self.W_ks = nn.Linear(d_model, d_model)
        self.W_v  = nn.Linear(d_model, d_model)
        self.out  = nn.Linear(d_model, d_model)

    def forward(self, text_emb: torch.Tensor, spatial_emb: torch.Tensor,
                mask: torch.Tensor = None):
        """
        Args:
            text_emb:    (batch, seq_len, d_model) — token text embeddings
            spatial_emb: (batch, seq_len, d_model) — bounding box embeddings
            mask:        (batch, 1, seq_len, seq_len) — causal / padding mask
        Returns:
            output: (batch, seq_len, d_model)
        """
        B, L, _ = text_emb.shape

        def split_heads(x):
            return x.view(B, L, self.n_heads, self.d_head).transpose(1, 2)

        Qt = split_heads(self.W_qt(text_emb))     # (B, H, L, d_head)
        Kt = split_heads(self.W_kt(text_emb))
        Qs = split_heads(self.W_qs(spatial_emb))
        Ks = split_heads(self.W_ks(spatial_emb))
        V  = split_heads(self.W_v(text_emb))

        # Four disentangled attention terms
        scale = self.d_head ** -0.5
        scores = (Qt @ Kt.transpose(-2, -1) +   # text  -> text
                  Qt @ Ks.transpose(-2, -1) +   # text  -> spatial
                  Qs @ Kt.transpose(-2, -1) +   # spatial -> text
                  Qs @ Ks.transpose(-2, -1)     # spatial -> spatial
                  ) * scale

        if mask is not None:
            scores = scores.masked_fill(mask == 0, float('-inf'))

        attn = torch.softmax(scores, dim=-1)
        out  = (attn @ V).transpose(1, 2).contiguous().view(B, L, -1)
        return self.out(out)


# --- Example usage ---
d_model, n_heads, seq_len, batch = 512, 8, 64, 2
text_emb    = torch.randn(batch, seq_len, d_model)
spatial_emb = torch.randn(batch, seq_len, d_model)  # encoded bounding boxes

attn_layer = DisentangledSpatialAttention(d_model, n_heads)
output = attn_layer(text_emb, spatial_emb)
print(output.shape)  # torch.Size([2, 64, 512])
```

--------------------------------

### DocLLM BibTeX Citation

Source: https://context7.com/dswang2011/docllm/llms.txt

This is the BibTeX entry for citing the DocLLM paper in academic publications.

```bibtex
@misc{wang2023docllm,
      title   = {DocLLM: A layout-aware generative language model for multimodal document understanding},
      author  = {Dongsheng Wang and Natraj Raman and Mathieu Sibue and Zhiqiang Ma
                 and Petr Babkin and Simerjot Kaur and Yulong Pei
                 and Armineh Nourbakhsh and Xiaomo Liu},
      year    = {2023},
      eprint  = {2401.00908},
      archivePrefix = {arXiv},
      primaryClass  = {cs.CL}
}
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.