Hugging Face Transformers

https://huggingface.co/docs/transformers/main/
Admin
Hugging Face Transformers is an open-source library providing pre-trained models and tools for...
Tokens:42,368
Snippets:122
Trust Score:10
Update:6 hours ago
Show doc for...
Tokens:
Raw
### Install LayoutLMV2 dependencies with pip

Source: https://huggingface.co/docs/transformers/model_doc/layoutlmv2

Installs detectron2, torchvision, and tesseract, which are required for LayoutLMV2. Run these commands in your terminal. Note that passing doctests during development also requires these packages.

```bash
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
python -m pip install torchvision tesseract

```

--------------------------------

### LayoutLMv2ForTokenClassification forward example

Source: https://huggingface.co/docs/transformers/model_doc/layoutlmv2

Example usage of LayoutLMv2ForTokenClassification with AutoProcessor and a dataset. It loads the FUNSD dataset, prepares inputs (image, words, boxes, labels), runs the model, and shows predicted token classes. Note that results are not good without further fine-tuning.

```python
>>> from transformers import AutoProcessor, LayoutLMv2ForTokenClassification, set_seed
>>> from PIL import Image
>>> from datasets import load_dataset

>>> set_seed(0)

>>> datasets = load_dataset("nielsr/funsd", split="test")
>>> labels = datasets.features["ner_tags"].feature.names
>>> id2label = {v: k for v, k in enumerate(labels)}

>>> processor = AutoProcessor.from_pretrained("microsoft/layoutlmv2-base-uncased", revision="no_ocr")
>>> model = LayoutLMv2ForTokenClassification.from_pretrained(
...     "microsoft/layoutlmv2-base-uncased", num_labels=len(labels)
... )

>>> data = datasets[0]
>>> image = Image.open(data["image_path"]).convert("RGB")
>>> words = data["words"]
>>> boxes = data["bboxes"]  # make sure to normalize your bounding boxes
>>> word_labels = data["ner_tags"]
>>> encoding = processor(
...     image,
...     words,
...     boxes=boxes,
...     word_labels=word_labels,
...     padding="max_length",
...     truncation=True,
...     return_tensors="pt",
... )

>>> outputs = model(**encoding)
>>> logits, loss = outputs.logits, outputs.loss

>>> predicted_token_class_ids = logits.argmax(-1)
>>> predicted_tokens_classes = [id2label[t.item()] for t in predicted_token_class_ids[0]]
>>> predicted_tokens_classes[:5]  # results are not good without further fine-tuning
['I-HEADER', 'I-HEADER', 'I-QUESTION', 'I-HEADER', 'I-QUESTION']
```

--------------------------------

### LayoutLMv2Model forward pass example

Source: https://huggingface.co/docs/transformers/model_doc/layoutlmv2

Loads the LayoutLMv2 processor and model, processes an image from a dataset, and runs a forward pass to obtain the last hidden states. The output shape is torch.Size([1, 342, 768]).

```python
>>> from transformers import AutoProcessor, LayoutLMv2Model, set_seed
>>> from PIL import Image
>>> import torch
>>> from datasets import load_dataset

>>> set_seed(0)

>>> processor = AutoProcessor.from_pretrained("microsoft/layoutlmv2-base-uncased")
>>> model = LayoutLMv2Model.from_pretrained("microsoft/layoutlmv2-base-uncased")


>>> dataset = load_dataset("hf-internal-testing/fixtures_docvqa")
>>> image = dataset["test"][0]["image"]

>>> encoding = processor(image, return_tensors="pt")

>>> outputs = model(**encoding)
>>> last_hidden_states = outputs.last_hidden_state

>>> last_hidden_states.shape
torch.Size([1, 342, 768])
```

--------------------------------

### LayoutLMv2ForQuestionAnswering forward with target positions

Source: https://huggingface.co/docs/transformers/model_doc/layoutlmv2

Runs the model with start_positions and end_positions to compute loss, then prints the predicted span indices. The output shows the same indices as the inference example, indicating the model's predictions.

```python
>>> target_start_index = torch.tensor([7])
>>> target_end_index = torch.tensor([14])
>>> outputs = model(**encoding, start_positions=target_start_index, end_positions=target_end_index)
>>> predicted_answer_span_start = outputs.start_logits.argmax(-1).item()
>>> predicted_answer_span_end = outputs.end_logits.argmax(-1).item()
>>> predicted_answer_span_start, predicted_answer_span_end
(30, 191)
```

--------------------------------

### LayoutLMv2ForQuestionAnswering inference with AutoProcessor

Source: https://huggingface.co/docs/transformers/model_doc/layoutlmv2

Loads the LayoutLMv2 model and processor, processes a document image and question, and runs a forward pass to get start and end logits. The predicted answer is decoded from the tokenizer, but the source notes that results are not good without further fine-tuning.

```python
>>> from transformers import AutoProcessor, LayoutLMv2ForQuestionAnswering, set_seed
>>> import torch
>>> from PIL import Image
>>> from datasets import load_dataset

>>> set_seed(0)
>>> processor = AutoProcessor.from_pretrained("microsoft/layoutlmv2-base-uncased")
>>> model = LayoutLMv2ForQuestionAnswering.from_pretrained("microsoft/layoutlmv2-base-uncased")

>>> dataset = load_dataset("hf-internal-testing/fixtures_docvqa")
>>> image = dataset["test"][0]["image"]
>>> question = "When is coffee break?"
>>> encoding = processor(image, question, return_tensors="pt")

>>> outputs = model(**encoding)
>>> predicted_start_idx = outputs.start_logits.argmax(-1).item()
>>> predicted_end_idx = outputs.end_logits.argmax(-1).item()
>>> predicted_start_idx, predicted_end_idx
(30, 191)

>>> predicted_answer_tokens = encoding.input_ids.squeeze()[predicted_start_idx : predicted_end_idx + 1]
>>> predicted_answer = processor.tokenizer.decode(predicted_answer_tokens)
>>> predicted_answer  # results are not good without further fine-tuning
'44 a. m. to 12 : 25 p. m. 12 : 25 to 12 : 58 p. m. 12 : 58 to 4 : 00 p. m. 2 : 00 to 5 : 00 p. m. coffee break coffee will be served for men and women in the lobby adjacent to exhibit area. please move into exhibit area. ( exhibits open ) trrf general session ( part | ) presiding : lee a. waller trrf vice president " introductory remarks " lee a. waller, trrf vice presi - dent individual interviews with trrf public board members and sci - entific advisory council mem - bers conducted by trrf treasurer philip g. kuehn to get answers which the public refrigerated warehousing industry is looking for. plus questions from'
```

--------------------------------

### Initialize LayoutLMv2 model and configuration

Source: https://huggingface.co/docs/transformers/model_doc/layoutlmv2

Shows how to initialize a LayoutLMv2 model and configuration using the microsoft/layoutlmv2-base-uncased style. The model is created with random weights.

```python
>>> from transformers import LayoutLMv2Config, LayoutLMv2Model

>>> # Initializing a LayoutLMv2 microsoft/layoutlmv2-base-uncased style configuration
>>> configuration = LayoutLMv2Config()

>>> # Initializing a model (with random weights) from the microsoft/layoutlmv2-base-uncased style configuration
>>> model = LayoutLMv2Model(configuration)

>>> # Accessing the model configuration
>>> configuration = model.config
```

--------------------------------

### Initialize LayoutLMv2Processor with image processor and tokenizer

Source: https://huggingface.co/docs/transformers/model_doc/layoutlmv2

Shows how to instantiate a LayoutLMv2Processor by combining a LayoutLMv2ImageProcessor and a LayoutLMv2TokenizerFast. The image processor has apply_ocr set to True by default, enabling OCR on images.

```python
from transformers import LayoutLMv2ImageProcessor, LayoutLMv2Processor, LayoutLMv2TokenizerFast


image_processor = LayoutLMv2ImageProcessor()  # apply_ocr is set to True by default
tokenizer = LayoutLMv2TokenizerFast.from_pretrained("microsoft/layoutlmv2-base-uncased")
processor = LayoutLMv2Processor(image_processor, tokenizer)
```

--------------------------------

### LayoutLMv2ForSequenceClassification forward example

Source: https://huggingface.co/docs/transformers/model_doc/layoutlmv2

Example usage of LayoutLMv2ForSequenceClassification with AutoProcessor and a dataset from Hugging Face. It loads a sample image, processes it, and performs a forward pass with a sequence label. The output includes loss and logits, and the predicted index is compared to the dataset label names.

```python
>>> from transformers import AutoProcessor, LayoutLMv2ForSequenceClassification, set_seed
>>> from PIL import Image
>>> import torch
>>> from datasets import load_dataset

>>> set_seed(0)

>>> dataset = load_dataset("aharley/rvl_cdip", split="train", streaming=True)
>>> data = next(iter(dataset))
>>> image = data["image"].convert("RGB")

>>> processor = AutoProcessor.from_pretrained("microsoft/layoutlmv2-base-uncased")
>>> model = LayoutLMv2ForSequenceClassification.from_pretrained(
...     "microsoft/layoutlmv2-base-uncased", num_labels=dataset.info.features["label"].num_classes
... )

>>> encoding = processor(image, return_tensors="pt")
>>> sequence_label = torch.tensor([data["label"]])

>>> outputs = model(**encoding, labels=sequence_label)

>>> loss, logits = outputs.loss, outputs.logits
>>> predicted_idx = logits.argmax(dim=-1).item()
>>> predicted_answer = dataset.info.features["label"].names[4]
>>> predicted_idx, predicted_answer  # results are not good without further fine-tuning
(7, 'advertisement')
```

--------------------------------

### LayoutLMv2ForQuestionAnswering.forward

Source: https://huggingface.co/docs/transformers/model_doc/layoutlmv2

The forward method of LayoutLMv2ForQuestionAnswering performs extractive question answering. It takes input tokens, bounding boxes, document images, and optional masks, and returns span start and end logits.

```APIDOC
## LayoutLMv2ForQuestionAnswering.forward

### Description
Performs extractive question answering on a batch of document images and text tokens. Computes span start and end logits using a linear layer on top of the hidden states.

### Method
forward

### Signature
forward(input_ids, bbox, image, attention_mask=None, token_type_ids=None, position_ids=None, inputs_embeds=None, start_positions=None, end_positions=None, **kwargs)

### Parameters
- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`) - Indices of input sequence tokens in the vocabulary. Can be obtained using AutoTokenizer.
- **bbox** (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, optional) - Bounding boxes of each input token in normalized (x0, y0, x1, y1) format.
- **image** (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)` or `detectron.structures.ImageList`) - Batch of document images.
- **attention_mask** (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, optional) - Mask to avoid attention on padding tokens. Values in [0, 1].
- **token_type_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) - Segment token indices to indicate first and second portions of inputs. Values in [0, 1].
- **position_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) - Indices of positions of each input sequence tokens in the position embeddings.
- **inputs_embeds** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, optional) - Optionally, instead of passing input_ids, you can pass an embedded representation directly.
- **start_positions** (`torch.LongTensor` of shape `(batch_size,)`, optional) - Labels for position of the start of the labeled span for computing the token classification loss.
- **end_positions** (`torch.LongTensor` of shape `(batch_size,)`, optional) - Labels for position of the end of the labeled span for computing the token classification loss.
- **kwargs** (`TransformersKwargs`) - Additional keyword arguments.

### Returns
- **QuestionAnsweringModelOutput** or `tuple(torch.FloatTensor)` - The model output containing the span start and end logits, and optionally the loss if start_positions and end_positions are provided.

### Example
```python
from transformers import LayoutLMv2ForQuestionAnswering, AutoTokenizer
import torch

model = LayoutLMv2ForQuestionAnswering.from_pretrained("microsoft/layoutlmv2-base-uncased")
tokenizer = AutoTokenizer.from_pretrained("microsoft/layoutlmv2-base-uncased")

# Prepare inputs
words = ["Hello", "world"]
normalized_boxes = [[0, 0, 1, 1], [1, 1, 2, 2]]
encoding = tokenizer(words, return_tensors="pt", padding=True, truncation=True)
bbox = torch.tensor([normalized_boxes])
image = torch.randn(1, 3, 224, 224)

outputs = model(input_ids=encoding["input_ids"], bbox=bbox, image=image)
start_logits = outputs.start_logits
end_logits = outputs.end_logits
```
```

--------------------------------

### LayoutLMv2Processor visual question answering with apply_ocr=True

Source: https://huggingface.co/docs/transformers/model_doc/layoutlmv2

For VQA tasks, provide a question. The processor applies OCR by default and creates [CLS] question tokens [SEP] word tokens [SEP].

```python
from PIL import Image

from transformers import LayoutLMv2Processor


processor = LayoutLMv2Processor.from_pretrained("microsoft/layoutlmv2-base-uncased")

image = Image.open(
    "name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images)."
).convert("RGB")
question = "What's his name?"
encoding = processor(image, question, return_tensors="pt").to(model.device)
print(encoding.keys())
# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'image'])
```

--------------------------------

### class transformers.LayoutLMv2ImageProcessorPil

Source: https://huggingface.co/docs/transformers/model_doc/layoutlmv2

Constructor for the LayoutLMv2ImageProcessorPil class. Initializes the image processor with various preprocessing options such as resizing, rescaling, normalization, padding, and center cropping.

```APIDOC
## class transformers.LayoutLMv2ImageProcessorPil

### Description
Initializes a LayoutLMv2ImageProcessorPil instance for image preprocessing. This processor handles conversion to RGB, resizing, rescaling, normalization, padding, and center cropping according to the provided parameters.

### Constructor
`LayoutLMv2ImageProcessorPil(**kwargs)`

### Parameters
- **do_convert_rgb** (`bool`, _optional_) - Whether to convert the image to RGB.
- **do_resize** (`bool`, _optional_, defaults to `True`) - Whether to resize the image.
- **size** (`Annotated[int | list[int] | tuple[int, ...] | dict[str, int] | None]`, _optional_, defaults to `{'height': 224, 'width': 224}`) - Describes the maximum input dimensions to the model.
- **default_to_square** (`bool`, _optional_, defaults to `True`) - Whether to default to a square image when resizing, if size is an int.
- **crop_size** (`Annotated[int | list[int] | tuple[int, ...] | dict[str, int] | None]`, _optional_) - Size of the output image after applying `center_crop`.
- **resample** (`Annotated[Union[int, PILImageResampling, NoneType]]`, _optional_, defaults to `2`) - Resampling filter to use if resizing the image. Only has an effect if `do_resize` is set to `True`.
- **do_rescale** (`bool`, _optional_) - Whether to rescale the image.
- **rescale_factor** (`float`, _optional_) - Rescale factor to rescale the image by if `do_rescale` is set to `True`.
- **do_normalize** (`bool`, _optional_) - Whether to normalize the image.
- **image_mean** (`Union[float, list[float], tuple[float, ...]]`, _optional_) - Image mean to use for normalization. Only has an effect if `do_normalize` is set to `True`.
- **image_std** (`Union[float, list[float], tuple[float, ...]]`, _optional_) - Image standard deviation to use for normalization. Only has an effect if `do_normalize` is set to `True`.
- **do_pad** (`bool`, _optional_) - Whether to pad the image. Padding is done either to the largest size in the batch or to a fixed square size per image.
- **pad_size** (`Annotated[int | list[int] | tuple[int, ...] | dict[str, int] | None]`, _optional_) - The size in `{"height": int, "width": int}` to pad the images to. Must be larger than any image size provided for preprocessing. Applied only when `do_pad=True`.
- **do_center_crop** (`bool`, _optional_) - Whether to center crop the image.
- **data_format** (`Union[str, ChannelDimension]`, _optional_) - Only `ChannelDimension.FIRST` is supported. Added for compatibility with slow processors.
- **input_data_format** (`Union[str, ChannelDimension]`, _optional_) - The channel dimension format for the input image. Can be one of `"channels_first"`, `"channels_last"`, or `"none"`.
```

--------------------------------

### LayoutLMv2Processor inference with apply_ocr=False

Source: https://huggingface.co/docs/transformers/model_doc/layoutlmv2

Loads the processor with revision='no_ocr' and encodes an image with manually provided words and bounding boxes. Returns keys including 'input_ids', 'token_type_ids', 'attention_mask', 'bbox', and 'image'.

```python
from PIL import Image

from transformers import LayoutLMv2Processor


processor = LayoutLMv2Processor.from_pretrained("microsoft/layoutlmv2-base-uncased", revision="no_ocr")

image = Image.open(
    "name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images)."
).convert("RGB")
words = ["hello", "world"]
boxes = [[1, 2, 3, 4], [5, 6, 7, 8]]  # make sure to normalize your bounding boxes
encoding = processor(image, words, boxes=boxes, return_tensors="pt").to(model.device)
print(encoding.keys())
# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'image'])
```

--------------------------------

### LayoutLMv2Processor token classification training with word_labels

Source: https://huggingface.co/docs/transformers/model_doc/layoutlmv2

For token classification tasks, provide word_labels to get token-level labels. By default, only the first wordpiece is labeled and the rest are set to -100. Set only_label_first_subword to False to label all wordpieces.

```python
from PIL import Image

from transformers import LayoutLMv2Processor


processor = LayoutLMv2Processor.from_pretrained("microsoft/layoutlmv2-base-uncased", revision="no_ocr")

image = Image.open(
    "name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images)."
).convert("RGB")
words = ["hello", "world"]
boxes = [[1, 2, 3, 4], [5, 6, 7, 8]]  # make sure to normalize your bounding boxes
word_labels = [1, 2]
encoding = processor(image, words, boxes=boxes, word_labels=word_labels, return_tensors="pt").to(model.device)
print(encoding.keys())
# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'labels', 'image'])
```

--------------------------------

### LayoutLMv2Config constructor

Source: https://huggingface.co/docs/transformers/model_doc/layoutlmv2

Instantiates a LayoutLMv2Config object with various optional parameters to configure the model architecture.

```APIDOC
## LayoutLMv2Config

### Description
Creates a configuration object for LayoutLMv2 models. All parameters are optional and have default values.

### Method
Constructor

### Parameters
All parameters are optional.

- **transformers_version** (`str` or `None`) - Version of transformers library.
- **architectures** (`list[str]` or `None`) - List of model architectures.
- **output_hidden_states** (`bool` or `None`) - Whether to output hidden states.
- **return_dict** (`bool` or `None`) - Whether to return a dict.
- **dtype** (`str` or `torch.dtype` or `None`) - Data type for model.
- **chunk_size_feed_forward** (`int`) - Chunk size for feed forward layers.
- **is_encoder_decoder** (`bool`) - Whether model is encoder-decoder.
- **id2label** (`dict[int, str]` or `dict[str, str]` or `None`) - Mapping from id to label.
- **label2id** (`dict[str, int]` or `dict[str, str]` or `None`) - Mapping from label to id.
- **problem_type** (`Literal['regression', 'single_label_classification', 'multi_label_classification']` or `None`) - Problem type.
- **vocab_size** (`int`) - Vocabulary size.
- **hidden_size** (`int`) - Hidden size.
- **num_hidden_layers** (`int`) - Number of hidden layers.
- **num_attention_heads** (`int`) - Number of attention heads.
- **intermediate_size** (`int`) - Intermediate size.
- **hidden_act** (`str`) - Hidden activation function.
- **hidden_dropout_prob** (`float` or `int`) - Hidden dropout probability.
- **attention_probs_dropout_prob** (`float` or `int`) - Attention dropout probability.
- **max_position_embeddings** (`int`) - Maximum position embeddings.
- **type_vocab_size** (`int`) - Type vocabulary size.
- **initializer_range** (`float`) - Initializer range.
- **layer_norm_eps** (`float`) - Layer norm epsilon.
- **pad_token_id** (`int` or `None`) - Padding token id.
- **max_2d_position_embeddings** (`int`) - Maximum 2D position embeddings.
- **max_rel_pos** (`int`) - Maximum relative position.
- **rel_pos_bins** (`int`) - Relative position bins.
- **fast_qkv** (`bool`) - Whether to use fast QKV.
- **max_rel_2d_pos** (`int`) - Maximum relative 2D position.
- **rel_2d_pos_bins** (`int`) - Relative 2D position bins.
- **convert_sync_batchnorm** (`bool`) - Whether to convert sync batchnorm.
- **image_feature_pool_shape** (`list[int]` or `tuple[int, ...]`) - Shape of image feature pool.
- **coordinate_size** (`int`) - Coordinate size.
- **shape_size** (`int`) - Shape size.
- **has_relative_attention_bias** (`bool`) - Whether to use relative attention bias.
- **has_spatial_attention_bias** (`bool`) - Whether to use spatial attention bias.
- **has_visual_segment_embedding** (`bool`) - Whether to use visual segment embedding.
- **detectron2_config_args** (`dict` or `None`) - Detectron2 configuration arguments.

### Example
```python
from transformers import LayoutLMv2Config

config = LayoutLMv2Config(
    vocab_size=30522,
    hidden_size=768,
    num_hidden_layers=12,
    num_attention_heads=12,
    intermediate_size=3072,
    hidden_act='gelu',
    hidden_dropout_prob=0.1,
    attention_probs_dropout_prob=0.1,
    max_position_embeddings=512,
    type_vocab_size=2,
    initializer_range=0.02,
    layer_norm_eps=1e-12,
    pad_token_id=0,
    max_2d_position_embeddings=1024,
    max_rel_pos=128,
    rel_pos_bins=32,
    fast_qkv=True,
    max_rel_2d_pos=256,
    rel_2d_pos_bins=64,
    convert_sync_batchnorm=True,
    image_feature_pool_shape=(7, 7, 256),
    coordinate_size=128,
    shape_size=128,
    has_relative_attention_bias=True,
    has_spatial_attention_bias=True,
    has_visual_segment_embedding=False,
    detectron2_config_args=None
)
```
```
Tokens:
Raw