IBM Granite Vision 3.2 2B

https://huggingface.co/ibm-granite/granite-vision-3.2-2b
Admin
IBM Granite Vision 3.2-2b is a compact vision-language model designed for visual document...
Tokens:12,222
Snippets:97
Trust Score:10
Update:6 months ago
Show doc for...
Tokens:
Raw
### Install vLLM and Dependencies

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blob/main/README.md

Install the necessary libraries, including PyTorch and vLLM, before using the model with vLLM.

```bash
pip install torch torchvision torchaudio
pip install vllm==0.6.6

```

--------------------------------

### Install vLLM

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blame/main/README.md

Install the necessary libraries for using the model with vLLM. Ensure you have torch, torchvision, and torchaudio installed, along with vLLM version 0.6.6.

```shell
pip install torch torchvision torchaudio
pip install vllm==0.6.6
```

--------------------------------

### Usage with vLLM

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/discussions/1/files

Example of loading and using the granite-vision-3.2-2b model with vLLM. This demonstrates how to initialize the LLM, define sampling parameters, construct a prompt with an image, and generate text.

```python
from vllm import LLM, SamplingParams
from vllm.assets.image import ImageAsset
from huggingface_hub import hf_hub_download
from PIL import Image

model_path = "ibm-granite/granite-vision-3.2-2b"

model = LLM(
model=model_path,
limit_mm_per_prompt={"image": 1},
)

sampling_params = SamplingParams(
temperature=0.2,
max_tokens=64,
)

# Define the question we want to answer and format the prompt
image_token = "<image>"
system_prompt = "<|system|>
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
"

question = "What is the highest scoring model on ChartQA and what is its score?"
prompt = f"{system_prompt}<|user|>
{image_token}
{question}
<|assistant|>
"
img_path = hf_hub_download(repo_id=model_path, filename='example.png')
image = Image.open(img_path).convert("RGB")
print(image)

# Build the inputs to vLLM; the image is passed as `multi_modal_data`.
inputs = {
"prompt": prompt,
"multi_modal_data": {
"image": image,
}
}

outputs = model.generate(inputs, sampling_params=sampling_params)
print(f"Generated text: {outputs[0].outputs[0].text}")
```

--------------------------------

### Install Accelerate Dependency

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/discussions/5

Command to install the accelerate package to resolve missing dependency errors during model initialization.

```bash
 pip install accelerate
```

--------------------------------

### Launch SGLang Server for Granite Vision

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/discussions/7

Command used to start the SGLang server with the granite-vision-3.2-2b model, which triggers the reported architecture support error.

```bash
bash-5.2# python3.12 -m sglang.launch_server --model ibm-granite/granite-vision-3.2-2b --model-path ibm-granite/granite-vision-3.2-2b --port 30000 --host 0.0.0.0 --log-level debug --trust-remote-code --tensor-parallel-size 4 --enable-p2p-check --disable-cuda-graph
```

--------------------------------

### Run inference with vLLM

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/raw/main/README.md

Example of loading the model and generating text from an image input.

```python
from vllm import LLM, SamplingParams
from vllm.assets.image import ImageAsset
from huggingface_hub import hf_hub_download
from PIL import Image

model_path = "ibm-granite/granite-vision-3.2-2b"

model = LLM(
    model=model_path,
    limit_mm_per_prompt={"image": 1},
)

sampling_params = SamplingParams(
    temperature=0.2,
    max_tokens=64,
)

# Define the question we want to answer and format the prompt
image_token = "<image>"
system_prompt = "<|system|>
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
"

question = "What is the highest scoring model on ChartQA and what is its score?"
prompt = f"{system_prompt}<|user|>
{image_token}
{question}
<|assistant|>
"
img_path = hf_hub_download(repo_id=model_path, filename='example.png')
image = Image.open(img_path).convert("RGB")
print(image)

# Build the inputs to vLLM; the image is passed as `multi_modal_data`.
inputs = {
    "prompt": prompt,
    "multi_modal_data": {
        "image": image,
    }
}

outputs = model.generate(inputs, sampling_params=sampling_params)
print(f"Generated text: {outputs[0].outputs[0].text}")
```

--------------------------------

### Install Transformers Library

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blame/main/README.md

Ensure you have the latest version of the transformers library installed. This is required for using the granite-vision-3.2-2b model.

```shell
pip install transformers>=4.49
```

--------------------------------

### Install Transformers Library

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blob/main/README.md

Ensure you have the latest version of the transformers library installed, version 4.49 or higher, to use this model.

```bash
pip install transformers>=4.49

```

--------------------------------

### Load and Use Granite Vision Model with Transformers

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blob/main/README.md?code=true

Example of loading the Granite Vision model and processor using the transformers library. It demonstrates preparing an image and text prompt for a conversational input.

```python
from transformers import AutoProcessor, AutoModelForVision2Seq
from huggingface_hub import hf_hub_download
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

model_path = "ibm-granite/granite-vision-3.2-2b"
processor = AutoProcessor.from_pretrained(model_path)
model = AutoModelForVision2Seq.from_pretrained(model_path).to(device)

# prepare image and text prompt, using the appropriate prompt template

img_path = hf_hub_download(repo_id=model_path, filename='example.png')

conversation = [
 {
 "role": "user",
 "content": [
 {"type": "image", "url": img_path},
 {"type": "text", "text": "What is the highest scoring model on ChartQA and what is its score?"},
 ],
 },
]
inputs = processor.apply_chat_template(
 conversation,
 add_generation_prompt=True,

```

--------------------------------

### Use Granite Vision 3.2-2B with Transformers

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blob/main/README.md

This example demonstrates how to load and use the granite-vision-3.2-2b model with the transformers library for image-text-to-text generation. It prepares an image and text prompt, applies a chat template, and generates output.

```python
from transformers import AutoProcessor, AutoModelForVision2Seq
from huggingface_hub import hf_hub_download
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

model_path = "ibm-granite/granite-vision-3.2-2b"
processor = AutoProcessor.from_pretrained(model_path)
model = AutoModelForVision2Seq.from_pretrained(model_path).to(device)

# prepare image and text prompt, using the appropriate prompt template

img_path = hf_hub_download(repo_id=model_path, filename='example.png')

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": img_path},
            {"type": "text", "text": "What is the highest scoring model on ChartQA and what is its score?"},
        ],
    },
]
inputs = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt"
).to(device)


# autoregressively complete prompt
output = model.generate(**inputs, max_new_tokens=100)
print(processor.decode(output[0], skip_special_tokens=True))

```

--------------------------------

### Usage with Transformers

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/discussions/1/files

Example of how to load and use the granite-vision-3.2-2b model with the Hugging Face transformers library. This includes preparing an image and text prompt, applying a chat template, and generating output.

```python
from transformers import AutoProcessor, AutoModelForVision2Seq
from huggingface_hub import hf_hub_download
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

model_path = "ibm-granite/granite-vision-3.2-2b"
processor = AutoProcessor.from_pretrained(model_path)
model = AutoModelForVision2Seq.from_pretrained(model_path).to(device)

# prepare image and text prompt, using the appropriate prompt template

img_path = hf_hub_download(repo_id=model_path, filename='example.png')

conversation = [
{
"role": "user",
"content": [
{"type": "image", "url": img_path},
{"type": "text", "text": "What is the highest scoring model on ChartQA and what is its score?"},
],
},
]
inputs = processor.apply_chat_template(
conversation,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt"
).to(device)


# autoregressively complete prompt
output = model.generate(**inputs, max_new_tokens=100)
print(processor.decode(output[0], skip_special_tokens=True))
```

--------------------------------

### Raw OCR Output Example

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/discussions/6

Example of OCR output generated from a generic prompt, showing potential character recognition errors.

```text
Niklas Mommigholf1,* 2 Zhong Yang1,* 3 Wiqing Shi1,* 4 Ning Lin1,* 5 Li Fu-Fei1, Hamannch Hajabirai1
Luke Zettlemeyer1* 6 Percy Liang 7 Emmanuel Candola 8 Hamannch Hashimoto1
Abstract
Figure 1: Test-time scaling with α =32K. The benchmark is α =32B
Second, we show that the model's thinking process and
of this approach is to increase the compute time to
can be easily parallelized by recursively terminating the model’s think-
ing better. There has been much work exploring the
multiple layers of the model’s perception when a
128-bit integer test check is answer, often fixing incorrect reasoning
and the impact of this on performance. We have demonstrated that our
to the problem of solving a 32K-instance language model at α =32B
approach can be used in practice for many real-world
problems. For example, we describe their approach using large
scaling (4,32K) with budget forcing allows varying
the number of data points. OpenAI describes their approach by using
2624q, making agreements (Gil et al., 2024), and also
a simple approach (Gil et al., 2024). Our
from [97] to [378] of replicating and testing these approaches. Deeplock [8] (Dropbox.co.uk, et al., 2027)
proves that training only 1,000 samples with arbitrary
parameters is sufficient for a language model at α =32K to
predicting and controlling thinking duration via a simple
language model at α =32B. The creators of
plan et al., 2026, Hotham et al., 2021. The creators of
which consists of 1,000 carefully curated questions paired
W3N-NN and Zy et al. started the project. WS-NN
with a 1K subset and 2B and
Thinking Experiment (Google, 2024). We perform supe-
plexes, LZ and WS proposed using a 1K subset and 2B and
rvisions of this approach to achieve both time scaling and
Washington, Seattle. Also Institute for AI - Conceptual AI. </doc>
```

--------------------------------

### Improved OCR Output via Manual Cropping

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/discussions/6

Example of OCR output after applying manual cropping to the input document, resulting in higher text accuracy.

```text
Test-time scaling is a promising new approach to language modeling that uses extra test-time computation to improve performance. Recently, OpenAI's publicly shared model showed this capability but did not publish its methodology, leading to many replication efforts. We seek the simplest approach to achieve test-time scaling and strong reasoning performance. First, we curate a small dataset sIK of 1,000 questions paired with reasoning traces relating on three criteria: difficulty, diversity, and quality. Second, we develop budget forcing to control test-time computation by lengthening it by appending "Wait" multiple times to the model's generation when it tries to end. This can lead the model to doublecheck its answer, often fixing incorrect reasoning steps. After supervised finetuning the Qwen2.532B-Instruct language model on sIK and equipping it with budget forcing, our model sl-32B excels on competition math questions by up to 27% (MATH and AIME24). Further, scaling sl-32B with budget forcing allows extrapolation beyond its performance without test-time intervention: from 50% to 57% on AIME24. Our model, data, and code are open-source at 
https://github.com/simplescaling/sl
```

--------------------------------

### Load and Use Granite Vision with Transformers

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blob/main/README.md?code=true

Load the model and processor using Hugging Face Transformers. Prepare inputs with image and text, then generate output. Ensure to set the device to 'cuda' if available.

```python
from transformers import AutoProcessor, AutoModelForCausalLM
from PIL import Image
import requests

model_id = "ibm-granite/granite-vision-3.2-2b"

model = AutoModelForCausalLM.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/red-cat.png"
image = Image.open(requests.get(url, stream=True).raw)

question = "What is the highest scoring model on ChartQA and what is its score?"
prompt = f"<image> {question}"

inputs = processor(text=[prompt], images=[image], tokenize=True, return_dict=True, return_tensors="pt").to(device)

# autoregressively complete prompt
output = model.generate(**inputs, max_new_tokens=100)
print(processor.decode(output[0], skip_special_tokens=True))
```

--------------------------------

### Load and Use Granite Vision with vLLM

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blob/main/README.md?code=true

Load the model using vLLM and define sampling parameters. Prepare multimodal inputs with an image and prompt, then generate text. The image is passed via `multi_modal_data`.

```python
from vllm import LLM, SamplingParams
from vllm.assets.image import ImageAsset
from huggingface_hub import hf_hub_download
from PIL import Image

model_path = "ibm-granite/granite-vision-3.2-2b"

model = LLM(
 model=model_path,
 limit_mm_per_prompt={"image": 1},
)

sampling_params = SamplingParams(
 temperature=0.2,
 max_tokens=64,
)

# Define the question we want to answer and format the prompt
image_token = "<image>"
system_prompt = "<|system|>
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
"

question = "What is the highest scoring model on ChartQA and what is its score?"
prompt = f"{system_prompt}<|user|>
{image_token}
{question}
<|assistant|>
"
img_path = hf_hub_download(repo_id=model_path, filename='example.png')
image = Image.open(img_path).convert("RGB")
print(image)

# Build the inputs to vLLM; the image is passed as `multi_modal_data`.
inputs = {
 "prompt": prompt,
 "multi_modal_data": {
 "image": image,
 }
}

outputs = model.generate(inputs, sampling_params=sampling_params)
print(f"Generated text: {outputs[0].outputs[0].text}")
```

--------------------------------

### Model Configuration Parameters

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blame/main/config.json

Configuration settings for the LlavaNextForConditionalGeneration architecture, including image grid pinpoints and vision feature layer selection.

```json
{
    "image_grid_pinpoints": [
        [
            384,
            384
        ],
        [
            384,
            768
        ],
        [
            384,
            1152
        ],
        [
            384,
            1536
        ],
        [
            384,
            1920
        ],
        [
            384,
            2304
        ],
        [
            384,
            2688
        ],
        [
            384,
            3072
        ],
        [
            384,
            3456
        ],
        [
            384,
            3840
        ],
        [
            768,
            384
        ],
        [
            768,
            768
        ],
        [
            768,
            1152
        ],
        [
            768,
            1536
        ],
        [
            768,
            1920
        ],
        [
            1152,
            384
        ],
        [
            1152,
            768
        ],
        [
            1152,
            1152
        ],
        [
            1536,
            384
        ],
        [
            1536,
            768
        ],
        [
            1920,
            384
        ],
        [
            1920,
            768
        ],
        [
            2304,
            384
        ],
        [
            2688,
            384
        ],
        [
            3072,
            384
        ],
        [
            3456,
            384
        ],
        [
            3840,
            384
        ]
    ],
    "tie_word_embeddings": true,
    "transformers_version": "4.45.0.dev0",
    "architectures": [
        "LlavaNextForConditionalGeneration"
    ],
    "model_type": "llava_next",
    "use_image_newline_parameter": true,
    "vision_feature_layer": [
        -24,
        -20,
        -12,
        -1
    ],
    "vision_feature_select_strategy": "full",
    "text_config": {
        "architectures": [
            "GraniteForCausalLM"
        ],
        "attention_bias": false,
        "attention_dropout": 0.1,
        "attention_multiplier": 0.015625,
        "bos_token_id": 0,
        "embedding_multiplier": 12.0,
        "eos_token_id": 0,
        "hidden_act": "silu",
        "hidden_size": 2048,
        "initializer_range": 0.02,
        "intermediate_size": 8192,
        "logits_scaling": 8.0
```

--------------------------------

### Image Question Answering with Granite Vision

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blame/main/README.md

Prepare a prompt with an image token and a question, load an image, and pass it to the model for generation. Ensure the image is opened in RGB format.

```python
image_token = "<image>"
system_prompt = "<|system|>
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
"

question = "What is the highest scoring model on ChartQA and what is its score?"
prompt = f"{system_prompt}<|user|>
{image_token}
{question}
<|assistant|>
"
img_path = hf_hub_download(repo_id=model_path, filename='example.png')
image = Image.open(img_path).convert("RGB")
print(image)

# Build the inputs to vLLM; the image is passed as `multi_modal_data`.
inputs = {
    "prompt": prompt,
    "multi_modal_data": {
        "image": image,
    }
}

outputs = model.generate(inputs, sampling_params=sampling_params)
print(f"Generated text: {outputs[0].outputs[0].text}")
```

--------------------------------

### Load and Use Granite Vision with vLLM

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blob/main/README.md

Load the Granite Vision model using vLLM and perform multi-modal inference by providing an image and a prompt. Ensure the model path and image handling are correctly configured.

```python
from vllm import LLM, SamplingParams
from vllm.assets.image import ImageAsset
from huggingface_hub import hf_hub_download
from PIL import Image

model_path = "ibm-granite/granite-vision-3.2-2b"

model = LLM(
    model=model_path,
    limit_mm_per_prompt={"image": 1},
)

sampling_params = SamplingParams(
    temperature=0.2,
    max_tokens=64,
)

# Define the question we want to answer and format the prompt
image_token = "<image>"
system_prompt = "<|system|>
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
"

question = "What is the highest scoring model on ChartQA and what is its score?"
prompt = f"{system_prompt}<|user|>
{image_token}
{question}
<|assistant|>
"
img_path = hf_hub_download(repo_id=model_path, filename='example.png')
image = Image.open(img_path).convert("RGB")
print(image)

# Build the inputs to vLLM; the image is passed as `multi_modal_data`.
inputs = {
    "prompt": prompt,
    "multi_modal_data": {
        "image": image,
    }
}

outputs = model.generate(inputs, sampling_params=sampling_params)
print(f"Generated text: {outputs[0].outputs[0].text}")

```

--------------------------------

### Model Configuration JSON

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blame/main/generation_config.json

Contains the base configuration settings for the model, including token IDs and the required transformers version.

```json
{
  "_from_model_config": true,
  "bos_token_id": 0,
  "eos_token_id": 0,
  "pad_token_id": 0,
  "transformers_version": "4.45.0.dev0"
}
```

--------------------------------

### Usage with vLLM

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blame/main/README.md

This snippet shows how to load the granite-vision-3.2-2b model using vLLM for inference. It configures the LLM with the model path and sets sampling parameters for generation.

```python
from vllm import LLM, SamplingParams
from vllm.assets.image import ImageAsset
from huggingface_hub import hf_hub_download
from PIL import Image

model_path = "ibm-granite/granite-vision-3.2-2b"

model = LLM(
    model=model_path,
    limit_mm_per_prompt={"image": 1},
)

sampling_params = SamplingParams(
    temperature=0.2,
    max_tokens=64,
)
```

--------------------------------

### Model Configuration Parameters

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blame/main/processor_config.json

Defines the architectural and processing settings for the Granite Vision model.

```json
{
    "patch_size": 14,
    "processor_class": "LlavaNextProcessor",
    "vision_feature_select_strategy": "full"
}
```

--------------------------------

### System Prompt for IBM Granite Vision Model

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/discussions/8

This system prompt is crucial for the model to produce coherent and helpful responses. Ensure this prompt is correctly applied when interacting with the model, especially through compatibility layers like Ollama's OpenAI API.

```json
{
  "messages": [
    {
      "role": "system",
      "content": "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions."
    },
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "\n<img src=\"attachments/eefe8003-1ad5-4ff1-b5cb-ac1120ad6f91\">


  assets:
    [Screenshot 2025-03-05 191900.png](/attachments/eefe8003-1ad5-4ff1-b5cb-ac1120ad6f91)
"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/png;base64,REMOVED/TO/REDUCE/SIZE="
          }
        }
      ]
    },
    {
      "role": "user",
      "content": "Hi, can you see any images?"
    },
    {
      "role": "assistant",
      "content": "\nI'm sorry, but I cannot provide images or links due to the restrictions. You may try opening them through other means."
    }
  ]
}
```

--------------------------------

### Image Processing Configuration for LlavaNext

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blame/main/preprocessor_config.json

This JSON object defines the configuration parameters for the LlavaNext image processor. It specifies image resizing, normalization, and rescaling settings, along with image mean and standard deviation values.

```json
{
    "crop_size": {
        "height": 384,
        "width": 384
    },
    "do_convert_rgb": null,
    "do_normalize": true,
    "do_rescale": true,
    "do_resize": true,
    "image_mean": [
        0.5,
        0.5,
        0.5
    ],
    "image_processor_type": "LlavaNextImageProcessor",
    "image_std": [
        0.5,
        0.5,
        0.5
    ],
    "processor_class": "LlavaNextProcessor",
    "resample": 3,
    "rescale_factor": 0.00392156862745098,
    "size": {
        "height": 384,
        "width": 384
    },
    "image_grid_pinpoints": [
        [384, 384],
        [384, 768],
        [384, 1152],
        [384, 1536],
        [384, 1920],
        [384, 2304],
        [384, 2688],
        [384, 3072],
        [384, 3456],
        [384, 3840],
        [768, 384],
        [768, 768],
        [768, 1152],
        [768, 1536],
        [768, 1920],
        [1152, 384],
        [1152, 768],
        [1152, 1152],
        [1536, 384],
        [1536, 768],
        [1920, 384],
        [1920, 768],
        [2304, 384],
        [2688, 384],
        [3072, 384],
        [3456, 384],
        [3840, 384]
    ]
}
```

--------------------------------

### Tokenizer Configuration JSON

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blob/main/tokenizer_config.json

Configuration object defining the chat template, special tokens, and tokenizer class for the Granite Vision 3.2 2B model.

```json
{
  "chat_template": "{%- if tools %}\n {{- '<|start_of_role|>available_tools<|end_of_role|>\n' }}\n {%- for tool in tools %}\n {{- tool | tojson(indent=4) }}\n {%- if not loop.last %}\n {{- '\n\n' }}\n {%- endif %}\n {%- endfor %}\n {{- '<|end_of_text|>\n' }}\n{%- endif %}\n{%- for message in messages if message['role'] == 'system'%}{% else %}<|system|>\nA chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.\n{% endfor %}{%- for message in messages %}\n {%- if message['role'] == 'system' %}\n {{- '<|system|>\n' + message['content'] + '\n' }}\n {%- elif message['role'] == 'user' %}\n {{- '<|user|>\n' + message['content'] + '\n' }}\n {%- elif message['role'] == 'assistant' %}\n {{- '<|assistant|>\n' + message['content'] + '<|end_of_text|>' }}\n {%- elif message['role'] == 'assistant_tool_call' %}\n {{- '<|start_of_role|>assistant<|end_of_role|><|tool_call|>' + message['content'] + '<|end_of_text|>\n' }}\n {%- elif message['role'] == 'tool_response' %}\n {{- '<|start_of_role|>tool_response<|end_of_role|>' + message['content'] + '<|end_of_text|>\n' }}\n {%- endif %}\n {%- if loop.last and add_generation_prompt %}\n {{- '<|assistant|>\n' }}\n {%- endif %}\n{%- endfor %}",
  "clean_up_tokenization_spaces": true,
  "eos_token": "<|end_of_text|>",
  "errors": "replace",
  "model_max_length": 131072,
  "pad_token": "<|end_of_text|>",
  "padding_side": "right",
  "tokenizer_class": "GPT2Tokenizer",
  "unk_token": "<|end_of_text|>",
  "vocab_size": 49152
}
```

--------------------------------

### Model Loading Error Traceback

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/discussions/5

A traceback showing a NameError when attempting to load the model on an M2 Mac using the mps device.

```python
      7 model_path = "ibm-granite/granite-vision-3.2-2b"
      8 processor = AutoProcessor.from_pretrained(model_path)
----> 9 model = AutoModelForVision2Seq.from_pretrained(model_path).to(device)
     11 # prepare image and text prompt, using the appropriate prompt template
     13 img_path = hf_hub_download(repo_id=model_path, filename='chart.png')

File ~/dev/ai-stuff/experiments/ibm-granite/my-experiments/.venv/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py:571, in _BaseAutoModelClass.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    569     if model_class.config_class == config.sub_configs.get("text_config", None):
    570         config = config.get_text_config()
--> 571     return model_class.from_pretrained(
    572         pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, **kwargs
    573     )
    574 raise ValueError(
    575     f"Unrecognized configuration class {config.__class__} for this kind of AutoModel: {cls.__name__}.\n"
    576     f"Model type should be one of {', '.join(c.__name__ for c in cls._model_mapping.keys())}."
    577 )

File ~/dev/ai-stuff/experiments/ibm-granite/my-experiments/.venv/lib/python3.11/site-packages/transformers/modeling_utils.py:279, in restore_default_torch_dtype.<locals>._wrapper(*args, **kwargs)
    277 old_dtype = torch.get_default_dtype()
    278 try:
--> 279     return func(*args, **kwargs)
    280 finally:
...
   3735 else:
-> 3736     init_contexts = [no_init_weights(), init_empty_weights()]
   3738 return init_contexts

NameError: name 'init_empty_weights' is not defined
```

--------------------------------

### IBM Granite Vision 3 2 2B Tokenizer Configuration

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/resolve/main/tokenizer_config.json?download=true

This JSON object defines the configuration for the tokenizer used with the IBM Granite Vision 3 2 2B model. It specifies special tokens, the chat template for structuring conversations, and various tokenizer settings.

```json
{
            "lstrip": false,
            "normalized": false,
            "rstrip": false,
            "single_word": false,
            "special": true
        }
    },
    "additional_special_tokens": [
        "<|start_of_role|>",
        "<|end_of_role|>",
        "<|tool_call|>"
    ],
    "bos_token": "<|end_of_text|>",
    "chat_template": "{%- if tools %}\n    {{- '<|start_of_role|>available_tools<|end_of_role|>
' }}
    {%- for tool in tools %}
    {{- tool | tojson(indent=4) }}
    {%- if not loop.last %}
        {{- '\n\n' }}
    {%- endif %}
    {%- endfor %}
    {{- '<|end_of_text|>
' }}
{%- endif %}
{%- for message in messages if message['role'] == 'system'%}{% else %}<|system|>
            A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
{% endfor %}{%- for message in messages %}
    {%- if message['role'] == 'system' %}
    {{- '<|system|>
' + message['content'] + '
' }}
    {%- elif message['role'] == 'user' %}
    {{- '<|user|>
' + message['content'] + '
' }}
    {%- elif message['role'] == 'assistant' %}
    {{- '<|assistant|>
'  + message['content'] + '<|end_of_text|>' }}
    {%- elif message['role'] == 'assistant_tool_call' %}
    {{- '<|start_of_role|>assistant<|end_of_role|><|tool_call|>' + message['content'] + '<|end_of_text|>
' }}
    {%- elif message['role'] == 'tool_response' %}
    {{- '<|start_of_role|>tool_response<|end_of_role|>' + message['content'] + '<|end_of_text|>
' }}
    {%- endif %}
    {%- if loop.last and add_generation_prompt %}
    {{- '<|assistant|>
' }}
    {%- endif %}
{%- endfor %}",
    "clean_up_tokenization_spaces": true,
    "eos_token": "<|end_of_text|>",
    "errors": "replace",
    "model_max_length": 131072,
    "pad_token": "<|end_of_text|>",
    "padding_side": "right",
    "tokenizer_class": "GPT2Tokenizer",
    "unk_token": "<|end_of_text|>",
    "vocab_size": 49152
}
```

--------------------------------

### Granite Vision 3 2B Chat Template

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/raw/main/chat_template.json

This chat template is used to format messages and tool information for the Granite Vision 3 2B model. It handles system messages, user messages with images and text, assistant responses, and tool calls/responses.

```json
{
    "chat_template": "{%- if tools %}\n    {{- '<|start_of_role|>available_tools<|end_of_role|>
' }}
    {%- for tool in tools %}
    {{- tool | tojson(indent=4) }}
    {%- if not loop.last %}
        {{- '\n\n' }}
    {%- endif %}
    {%- endfor %}
    {{- '<|end_of_text|>
' }}
{%- endif %}
{%- for message in messages if message['role'] == 'system'%}{% else %}<|system|>
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
{% endfor %}{%- for message in messages %}
    {%- if message['role'] == 'system' %}
    {{- '<|system|>
' + message['content'][0]['text'] + '
' }}
    {%- elif message['role'] == 'user' %}<|user|>
 {# Render all images first #}{% for content in message['content'] | selectattr('type', 'equalto', 'image') %}{{ '<image>
' }}{% endfor %}{# Render all text next #}{% for content in message['content'] | selectattr('type', 'equalto', 'text') %}{{ content['text'] + '
' }}{% endfor %}
{%- elif message['role'] == 'assistant' %}
    {{- '<|assistant|>
'  + message['content'][0]['text']  + '<|end_of_text|>' }}
    {%- elif message['role'] == 'assistant_tool_call' %}
    {{- '<|start_of_role|>assistant<|end_of_role|><|tool_call|>' + message['content'][0]['text']  + '<|end_of_text|>
' }}
    {%- elif message['role'] == 'tool_response' %}
    {{- '<|start_of_role|>tool_response<|end_of_role|>' + message['content'][0]['text'] + '<|end_of_text|>
' }}
    {%- endif %}
    {%- if loop.last and add_generation_prompt %}
    {{- '<|assistant|>
' }}
    {%- endif %}
{%- endfor %}"
```

--------------------------------

### Granite Vision 3.2 2B Token Mapping

Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blame/main/added_tokens.json

Defines the special token IDs used for role management, tool invocation, and image representation in the model.

```json
{
    "<|end_of_role|>": 49153,
    "<|start_of_role|>": 49152,
    "<|tool_call|>": 49154,
    "<image>": "49155"
}
```
Tokens:
Raw