### Install vLLM and Dependencies Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blob/main/README.md Install the necessary libraries, including PyTorch and vLLM, before using the model with vLLM. ```bash pip install torch torchvision torchaudio pip install vllm==0.6.6 ``` -------------------------------- ### Install vLLM Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blame/main/README.md Install the necessary libraries for using the model with vLLM. Ensure you have torch, torchvision, and torchaudio installed, along with vLLM version 0.6.6. ```shell pip install torch torchvision torchaudio pip install vllm==0.6.6 ``` -------------------------------- ### Usage with vLLM Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/discussions/1/files Example of loading and using the granite-vision-3.2-2b model with vLLM. This demonstrates how to initialize the LLM, define sampling parameters, construct a prompt with an image, and generate text. ```python from vllm import LLM, SamplingParams from vllm.assets.image import ImageAsset from huggingface_hub import hf_hub_download from PIL import Image model_path = "ibm-granite/granite-vision-3.2-2b" model = LLM( model=model_path, limit_mm_per_prompt={"image": 1}, ) sampling_params = SamplingParams( temperature=0.2, max_tokens=64, ) # Define the question we want to answer and format the prompt image_token = "" system_prompt = "<|system|> A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. " question = "What is the highest scoring model on ChartQA and what is its score?" prompt = f"{system_prompt}<|user|> {image_token} {question} <|assistant|> " img_path = hf_hub_download(repo_id=model_path, filename='example.png') image = Image.open(img_path).convert("RGB") print(image) # Build the inputs to vLLM; the image is passed as `multi_modal_data`. inputs = { "prompt": prompt, "multi_modal_data": { "image": image, } } outputs = model.generate(inputs, sampling_params=sampling_params) print(f"Generated text: {outputs[0].outputs[0].text}") ``` -------------------------------- ### Install Accelerate Dependency Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/discussions/5 Command to install the accelerate package to resolve missing dependency errors during model initialization. ```bash pip install accelerate ``` -------------------------------- ### Launch SGLang Server for Granite Vision Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/discussions/7 Command used to start the SGLang server with the granite-vision-3.2-2b model, which triggers the reported architecture support error. ```bash bash-5.2# python3.12 -m sglang.launch_server --model ibm-granite/granite-vision-3.2-2b --model-path ibm-granite/granite-vision-3.2-2b --port 30000 --host 0.0.0.0 --log-level debug --trust-remote-code --tensor-parallel-size 4 --enable-p2p-check --disable-cuda-graph ``` -------------------------------- ### Run inference with vLLM Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/raw/main/README.md Example of loading the model and generating text from an image input. ```python from vllm import LLM, SamplingParams from vllm.assets.image import ImageAsset from huggingface_hub import hf_hub_download from PIL import Image model_path = "ibm-granite/granite-vision-3.2-2b" model = LLM( model=model_path, limit_mm_per_prompt={"image": 1}, ) sampling_params = SamplingParams( temperature=0.2, max_tokens=64, ) # Define the question we want to answer and format the prompt image_token = "" system_prompt = "<|system|> A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. " question = "What is the highest scoring model on ChartQA and what is its score?" prompt = f"{system_prompt}<|user|> {image_token} {question} <|assistant|> " img_path = hf_hub_download(repo_id=model_path, filename='example.png') image = Image.open(img_path).convert("RGB") print(image) # Build the inputs to vLLM; the image is passed as `multi_modal_data`. inputs = { "prompt": prompt, "multi_modal_data": { "image": image, } } outputs = model.generate(inputs, sampling_params=sampling_params) print(f"Generated text: {outputs[0].outputs[0].text}") ``` -------------------------------- ### Install Transformers Library Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blame/main/README.md Ensure you have the latest version of the transformers library installed. This is required for using the granite-vision-3.2-2b model. ```shell pip install transformers>=4.49 ``` -------------------------------- ### Install Transformers Library Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blob/main/README.md Ensure you have the latest version of the transformers library installed, version 4.49 or higher, to use this model. ```bash pip install transformers>=4.49 ``` -------------------------------- ### Load and Use Granite Vision Model with Transformers Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blob/main/README.md?code=true Example of loading the Granite Vision model and processor using the transformers library. It demonstrates preparing an image and text prompt for a conversational input. ```python from transformers import AutoProcessor, AutoModelForVision2Seq from huggingface_hub import hf_hub_download import torch device = "cuda" if torch.cuda.is_available() else "cpu" model_path = "ibm-granite/granite-vision-3.2-2b" processor = AutoProcessor.from_pretrained(model_path) model = AutoModelForVision2Seq.from_pretrained(model_path).to(device) # prepare image and text prompt, using the appropriate prompt template img_path = hf_hub_download(repo_id=model_path, filename='example.png') conversation = [ { "role": "user", "content": [ {"type": "image", "url": img_path}, {"type": "text", "text": "What is the highest scoring model on ChartQA and what is its score?"}, ], }, ] inputs = processor.apply_chat_template( conversation, add_generation_prompt=True, ``` -------------------------------- ### Use Granite Vision 3.2-2B with Transformers Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blob/main/README.md This example demonstrates how to load and use the granite-vision-3.2-2b model with the transformers library for image-text-to-text generation. It prepares an image and text prompt, applies a chat template, and generates output. ```python from transformers import AutoProcessor, AutoModelForVision2Seq from huggingface_hub import hf_hub_download import torch device = "cuda" if torch.cuda.is_available() else "cpu" model_path = "ibm-granite/granite-vision-3.2-2b" processor = AutoProcessor.from_pretrained(model_path) model = AutoModelForVision2Seq.from_pretrained(model_path).to(device) # prepare image and text prompt, using the appropriate prompt template img_path = hf_hub_download(repo_id=model_path, filename='example.png') conversation = [ { "role": "user", "content": [ {"type": "image", "url": img_path}, {"type": "text", "text": "What is the highest scoring model on ChartQA and what is its score?"}, ], }, ] inputs = processor.apply_chat_template( conversation, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt" ).to(device) # autoregressively complete prompt output = model.generate(**inputs, max_new_tokens=100) print(processor.decode(output[0], skip_special_tokens=True)) ``` -------------------------------- ### Usage with Transformers Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/discussions/1/files Example of how to load and use the granite-vision-3.2-2b model with the Hugging Face transformers library. This includes preparing an image and text prompt, applying a chat template, and generating output. ```python from transformers import AutoProcessor, AutoModelForVision2Seq from huggingface_hub import hf_hub_download import torch device = "cuda" if torch.cuda.is_available() else "cpu" model_path = "ibm-granite/granite-vision-3.2-2b" processor = AutoProcessor.from_pretrained(model_path) model = AutoModelForVision2Seq.from_pretrained(model_path).to(device) # prepare image and text prompt, using the appropriate prompt template img_path = hf_hub_download(repo_id=model_path, filename='example.png') conversation = [ { "role": "user", "content": [ {"type": "image", "url": img_path}, {"type": "text", "text": "What is the highest scoring model on ChartQA and what is its score?"}, ], }, ] inputs = processor.apply_chat_template( conversation, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt" ).to(device) # autoregressively complete prompt output = model.generate(**inputs, max_new_tokens=100) print(processor.decode(output[0], skip_special_tokens=True)) ``` -------------------------------- ### Raw OCR Output Example Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/discussions/6 Example of OCR output generated from a generic prompt, showing potential character recognition errors. ```text Niklas Mommigholf1,* 2 Zhong Yang1,* 3 Wiqing Shi1,* 4 Ning Lin1,* 5 Li Fu-Fei1, Hamannch Hajabirai1 Luke Zettlemeyer1* 6 Percy Liang 7 Emmanuel Candola 8 Hamannch Hashimoto1 Abstract Figure 1: Test-time scaling with α =32K. The benchmark is α =32B Second, we show that the model's thinking process and of this approach is to increase the compute time to can be easily parallelized by recursively terminating the model’s think- ing better. There has been much work exploring the multiple layers of the model’s perception when a 128-bit integer test check is answer, often fixing incorrect reasoning and the impact of this on performance. We have demonstrated that our to the problem of solving a 32K-instance language model at α =32B approach can be used in practice for many real-world problems. For example, we describe their approach using large scaling (4,32K) with budget forcing allows varying the number of data points. OpenAI describes their approach by using 2624q, making agreements (Gil et al., 2024), and also a simple approach (Gil et al., 2024). Our from [97] to [378] of replicating and testing these approaches. Deeplock [8] (Dropbox.co.uk, et al., 2027) proves that training only 1,000 samples with arbitrary parameters is sufficient for a language model at α =32K to predicting and controlling thinking duration via a simple language model at α =32B. The creators of plan et al., 2026, Hotham et al., 2021. The creators of which consists of 1,000 carefully curated questions paired W3N-NN and Zy et al. started the project. WS-NN with a 1K subset and 2B and Thinking Experiment (Google, 2024). We perform supe- plexes, LZ and WS proposed using a 1K subset and 2B and rvisions of this approach to achieve both time scaling and Washington, Seattle. Also Institute for AI - Conceptual AI. ``` -------------------------------- ### Improved OCR Output via Manual Cropping Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/discussions/6 Example of OCR output after applying manual cropping to the input document, resulting in higher text accuracy. ```text Test-time scaling is a promising new approach to language modeling that uses extra test-time computation to improve performance. Recently, OpenAI's publicly shared model showed this capability but did not publish its methodology, leading to many replication efforts. We seek the simplest approach to achieve test-time scaling and strong reasoning performance. First, we curate a small dataset sIK of 1,000 questions paired with reasoning traces relating on three criteria: difficulty, diversity, and quality. Second, we develop budget forcing to control test-time computation by lengthening it by appending "Wait" multiple times to the model's generation when it tries to end. This can lead the model to doublecheck its answer, often fixing incorrect reasoning steps. After supervised finetuning the Qwen2.532B-Instruct language model on sIK and equipping it with budget forcing, our model sl-32B excels on competition math questions by up to 27% (MATH and AIME24). Further, scaling sl-32B with budget forcing allows extrapolation beyond its performance without test-time intervention: from 50% to 57% on AIME24. Our model, data, and code are open-source at https://github.com/simplescaling/sl ``` -------------------------------- ### Load and Use Granite Vision with Transformers Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blob/main/README.md?code=true Load the model and processor using Hugging Face Transformers. Prepare inputs with image and text, then generate output. Ensure to set the device to 'cuda' if available. ```python from transformers import AutoProcessor, AutoModelForCausalLM from PIL import Image import requests model_id = "ibm-granite/granite-vision-3.2-2b" model = AutoModelForCausalLM.from_pretrained(model_id) processor = AutoProcessor.from_pretrained(model_id) device = "cuda" if torch.cuda.is_available() else "cpu" model.to(device) url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/red-cat.png" image = Image.open(requests.get(url, stream=True).raw) question = "What is the highest scoring model on ChartQA and what is its score?" prompt = f" {question}" inputs = processor(text=[prompt], images=[image], tokenize=True, return_dict=True, return_tensors="pt").to(device) # autoregressively complete prompt output = model.generate(**inputs, max_new_tokens=100) print(processor.decode(output[0], skip_special_tokens=True)) ``` -------------------------------- ### Load and Use Granite Vision with vLLM Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blob/main/README.md?code=true Load the model using vLLM and define sampling parameters. Prepare multimodal inputs with an image and prompt, then generate text. The image is passed via `multi_modal_data`. ```python from vllm import LLM, SamplingParams from vllm.assets.image import ImageAsset from huggingface_hub import hf_hub_download from PIL import Image model_path = "ibm-granite/granite-vision-3.2-2b" model = LLM( model=model_path, limit_mm_per_prompt={"image": 1}, ) sampling_params = SamplingParams( temperature=0.2, max_tokens=64, ) # Define the question we want to answer and format the prompt image_token = "" system_prompt = "<|system|> A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. " question = "What is the highest scoring model on ChartQA and what is its score?" prompt = f"{system_prompt}<|user|> {image_token} {question} <|assistant|> " img_path = hf_hub_download(repo_id=model_path, filename='example.png') image = Image.open(img_path).convert("RGB") print(image) # Build the inputs to vLLM; the image is passed as `multi_modal_data`. inputs = { "prompt": prompt, "multi_modal_data": { "image": image, } } outputs = model.generate(inputs, sampling_params=sampling_params) print(f"Generated text: {outputs[0].outputs[0].text}") ``` -------------------------------- ### Model Configuration Parameters Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blame/main/config.json Configuration settings for the LlavaNextForConditionalGeneration architecture, including image grid pinpoints and vision feature layer selection. ```json { "image_grid_pinpoints": [ [ 384, 384 ], [ 384, 768 ], [ 384, 1152 ], [ 384, 1536 ], [ 384, 1920 ], [ 384, 2304 ], [ 384, 2688 ], [ 384, 3072 ], [ 384, 3456 ], [ 384, 3840 ], [ 768, 384 ], [ 768, 768 ], [ 768, 1152 ], [ 768, 1536 ], [ 768, 1920 ], [ 1152, 384 ], [ 1152, 768 ], [ 1152, 1152 ], [ 1536, 384 ], [ 1536, 768 ], [ 1920, 384 ], [ 1920, 768 ], [ 2304, 384 ], [ 2688, 384 ], [ 3072, 384 ], [ 3456, 384 ], [ 3840, 384 ] ], "tie_word_embeddings": true, "transformers_version": "4.45.0.dev0", "architectures": [ "LlavaNextForConditionalGeneration" ], "model_type": "llava_next", "use_image_newline_parameter": true, "vision_feature_layer": [ -24, -20, -12, -1 ], "vision_feature_select_strategy": "full", "text_config": { "architectures": [ "GraniteForCausalLM" ], "attention_bias": false, "attention_dropout": 0.1, "attention_multiplier": 0.015625, "bos_token_id": 0, "embedding_multiplier": 12.0, "eos_token_id": 0, "hidden_act": "silu", "hidden_size": 2048, "initializer_range": 0.02, "intermediate_size": 8192, "logits_scaling": 8.0 ``` -------------------------------- ### Image Question Answering with Granite Vision Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blame/main/README.md Prepare a prompt with an image token and a question, load an image, and pass it to the model for generation. Ensure the image is opened in RGB format. ```python image_token = "" system_prompt = "<|system|> A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. " question = "What is the highest scoring model on ChartQA and what is its score?" prompt = f"{system_prompt}<|user|> {image_token} {question} <|assistant|> " img_path = hf_hub_download(repo_id=model_path, filename='example.png') image = Image.open(img_path).convert("RGB") print(image) # Build the inputs to vLLM; the image is passed as `multi_modal_data`. inputs = { "prompt": prompt, "multi_modal_data": { "image": image, } } outputs = model.generate(inputs, sampling_params=sampling_params) print(f"Generated text: {outputs[0].outputs[0].text}") ``` -------------------------------- ### Load and Use Granite Vision with vLLM Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blob/main/README.md Load the Granite Vision model using vLLM and perform multi-modal inference by providing an image and a prompt. Ensure the model path and image handling are correctly configured. ```python from vllm import LLM, SamplingParams from vllm.assets.image import ImageAsset from huggingface_hub import hf_hub_download from PIL import Image model_path = "ibm-granite/granite-vision-3.2-2b" model = LLM( model=model_path, limit_mm_per_prompt={"image": 1}, ) sampling_params = SamplingParams( temperature=0.2, max_tokens=64, ) # Define the question we want to answer and format the prompt image_token = "" system_prompt = "<|system|> A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. " question = "What is the highest scoring model on ChartQA and what is its score?" prompt = f"{system_prompt}<|user|> {image_token} {question} <|assistant|> " img_path = hf_hub_download(repo_id=model_path, filename='example.png') image = Image.open(img_path).convert("RGB") print(image) # Build the inputs to vLLM; the image is passed as `multi_modal_data`. inputs = { "prompt": prompt, "multi_modal_data": { "image": image, } } outputs = model.generate(inputs, sampling_params=sampling_params) print(f"Generated text: {outputs[0].outputs[0].text}") ``` -------------------------------- ### Model Configuration JSON Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blame/main/generation_config.json Contains the base configuration settings for the model, including token IDs and the required transformers version. ```json { "_from_model_config": true, "bos_token_id": 0, "eos_token_id": 0, "pad_token_id": 0, "transformers_version": "4.45.0.dev0" } ``` -------------------------------- ### Usage with vLLM Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blame/main/README.md This snippet shows how to load the granite-vision-3.2-2b model using vLLM for inference. It configures the LLM with the model path and sets sampling parameters for generation. ```python from vllm import LLM, SamplingParams from vllm.assets.image import ImageAsset from huggingface_hub import hf_hub_download from PIL import Image model_path = "ibm-granite/granite-vision-3.2-2b" model = LLM( model=model_path, limit_mm_per_prompt={"image": 1}, ) sampling_params = SamplingParams( temperature=0.2, max_tokens=64, ) ``` -------------------------------- ### Model Configuration Parameters Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blame/main/processor_config.json Defines the architectural and processing settings for the Granite Vision model. ```json { "patch_size": 14, "processor_class": "LlavaNextProcessor", "vision_feature_select_strategy": "full" } ``` -------------------------------- ### System Prompt for IBM Granite Vision Model Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/discussions/8 This system prompt is crucial for the model to produce coherent and helpful responses. Ensure this prompt is correctly applied when interacting with the model, especially through compatibility layers like Ollama's OpenAI API. ```json { "messages": [ { "role": "system", "content": "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions." }, { "role": "user", "content": [ { "type": "text", "text": "\n assets: [Screenshot 2025-03-05 191900.png](/attachments/eefe8003-1ad5-4ff1-b5cb-ac1120ad6f91) " }, { "type": "image_url", "image_url": { "url": "data:image/png;base64,REMOVED/TO/REDUCE/SIZE=" } } ] }, { "role": "user", "content": "Hi, can you see any images?" }, { "role": "assistant", "content": "\nI'm sorry, but I cannot provide images or links due to the restrictions. You may try opening them through other means." } ] } ``` -------------------------------- ### Image Processing Configuration for LlavaNext Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blame/main/preprocessor_config.json This JSON object defines the configuration parameters for the LlavaNext image processor. It specifies image resizing, normalization, and rescaling settings, along with image mean and standard deviation values. ```json { "crop_size": { "height": 384, "width": 384 }, "do_convert_rgb": null, "do_normalize": true, "do_rescale": true, "do_resize": true, "image_mean": [ 0.5, 0.5, 0.5 ], "image_processor_type": "LlavaNextImageProcessor", "image_std": [ 0.5, 0.5, 0.5 ], "processor_class": "LlavaNextProcessor", "resample": 3, "rescale_factor": 0.00392156862745098, "size": { "height": 384, "width": 384 }, "image_grid_pinpoints": [ [384, 384], [384, 768], [384, 1152], [384, 1536], [384, 1920], [384, 2304], [384, 2688], [384, 3072], [384, 3456], [384, 3840], [768, 384], [768, 768], [768, 1152], [768, 1536], [768, 1920], [1152, 384], [1152, 768], [1152, 1152], [1536, 384], [1536, 768], [1920, 384], [1920, 768], [2304, 384], [2688, 384], [3072, 384], [3456, 384], [3840, 384] ] } ``` -------------------------------- ### Tokenizer Configuration JSON Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blob/main/tokenizer_config.json Configuration object defining the chat template, special tokens, and tokenizer class for the Granite Vision 3.2 2B model. ```json { "chat_template": "{%- if tools %}\n {{- '<|start_of_role|>available_tools<|end_of_role|>\n' }}\n {%- for tool in tools %}\n {{- tool | tojson(indent=4) }}\n {%- if not loop.last %}\n {{- '\n\n' }}\n {%- endif %}\n {%- endfor %}\n {{- '<|end_of_text|>\n' }}\n{%- endif %}\n{%- for message in messages if message['role'] == 'system'%}{% else %}<|system|>\nA chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.\n{% endfor %}{%- for message in messages %}\n {%- if message['role'] == 'system' %}\n {{- '<|system|>\n' + message['content'] + '\n' }}\n {%- elif message['role'] == 'user' %}\n {{- '<|user|>\n' + message['content'] + '\n' }}\n {%- elif message['role'] == 'assistant' %}\n {{- '<|assistant|>\n' + message['content'] + '<|end_of_text|>' }}\n {%- elif message['role'] == 'assistant_tool_call' %}\n {{- '<|start_of_role|>assistant<|end_of_role|><|tool_call|>' + message['content'] + '<|end_of_text|>\n' }}\n {%- elif message['role'] == 'tool_response' %}\n {{- '<|start_of_role|>tool_response<|end_of_role|>' + message['content'] + '<|end_of_text|>\n' }}\n {%- endif %}\n {%- if loop.last and add_generation_prompt %}\n {{- '<|assistant|>\n' }}\n {%- endif %}\n{%- endfor %}", "clean_up_tokenization_spaces": true, "eos_token": "<|end_of_text|>", "errors": "replace", "model_max_length": 131072, "pad_token": "<|end_of_text|>", "padding_side": "right", "tokenizer_class": "GPT2Tokenizer", "unk_token": "<|end_of_text|>", "vocab_size": 49152 } ``` -------------------------------- ### Model Loading Error Traceback Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/discussions/5 A traceback showing a NameError when attempting to load the model on an M2 Mac using the mps device. ```python 7 model_path = "ibm-granite/granite-vision-3.2-2b" 8 processor = AutoProcessor.from_pretrained(model_path) ----> 9 model = AutoModelForVision2Seq.from_pretrained(model_path).to(device) 11 # prepare image and text prompt, using the appropriate prompt template 13 img_path = hf_hub_download(repo_id=model_path, filename='chart.png') File ~/dev/ai-stuff/experiments/ibm-granite/my-experiments/.venv/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py:571, in _BaseAutoModelClass.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs) 569 if model_class.config_class == config.sub_configs.get("text_config", None): 570 config = config.get_text_config() --> 571 return model_class.from_pretrained( 572 pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, **kwargs 573 ) 574 raise ValueError( 575 f"Unrecognized configuration class {config.__class__} for this kind of AutoModel: {cls.__name__}.\n" 576 f"Model type should be one of {', '.join(c.__name__ for c in cls._model_mapping.keys())}." 577 ) File ~/dev/ai-stuff/experiments/ibm-granite/my-experiments/.venv/lib/python3.11/site-packages/transformers/modeling_utils.py:279, in restore_default_torch_dtype.._wrapper(*args, **kwargs) 277 old_dtype = torch.get_default_dtype() 278 try: --> 279 return func(*args, **kwargs) 280 finally: ... 3735 else: -> 3736 init_contexts = [no_init_weights(), init_empty_weights()] 3738 return init_contexts NameError: name 'init_empty_weights' is not defined ``` -------------------------------- ### IBM Granite Vision 3 2 2B Tokenizer Configuration Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/resolve/main/tokenizer_config.json?download=true This JSON object defines the configuration for the tokenizer used with the IBM Granite Vision 3 2 2B model. It specifies special tokens, the chat template for structuring conversations, and various tokenizer settings. ```json { "lstrip": false, "normalized": false, "rstrip": false, "single_word": false, "special": true } }, "additional_special_tokens": [ "<|start_of_role|>", "<|end_of_role|>", "<|tool_call|>" ], "bos_token": "<|end_of_text|>", "chat_template": "{%- if tools %}\n {{- '<|start_of_role|>available_tools<|end_of_role|> ' }} {%- for tool in tools %} {{- tool | tojson(indent=4) }} {%- if not loop.last %} {{- '\n\n' }} {%- endif %} {%- endfor %} {{- '<|end_of_text|> ' }} {%- endif %} {%- for message in messages if message['role'] == 'system'%}{% else %}<|system|> A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. {% endfor %}{%- for message in messages %} {%- if message['role'] == 'system' %} {{- '<|system|> ' + message['content'] + ' ' }} {%- elif message['role'] == 'user' %} {{- '<|user|> ' + message['content'] + ' ' }} {%- elif message['role'] == 'assistant' %} {{- '<|assistant|> ' + message['content'] + '<|end_of_text|>' }} {%- elif message['role'] == 'assistant_tool_call' %} {{- '<|start_of_role|>assistant<|end_of_role|><|tool_call|>' + message['content'] + '<|end_of_text|> ' }} {%- elif message['role'] == 'tool_response' %} {{- '<|start_of_role|>tool_response<|end_of_role|>' + message['content'] + '<|end_of_text|> ' }} {%- endif %} {%- if loop.last and add_generation_prompt %} {{- '<|assistant|> ' }} {%- endif %} {%- endfor %}", "clean_up_tokenization_spaces": true, "eos_token": "<|end_of_text|>", "errors": "replace", "model_max_length": 131072, "pad_token": "<|end_of_text|>", "padding_side": "right", "tokenizer_class": "GPT2Tokenizer", "unk_token": "<|end_of_text|>", "vocab_size": 49152 } ``` -------------------------------- ### Granite Vision 3 2B Chat Template Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/raw/main/chat_template.json This chat template is used to format messages and tool information for the Granite Vision 3 2B model. It handles system messages, user messages with images and text, assistant responses, and tool calls/responses. ```json { "chat_template": "{%- if tools %}\n {{- '<|start_of_role|>available_tools<|end_of_role|> ' }} {%- for tool in tools %} {{- tool | tojson(indent=4) }} {%- if not loop.last %} {{- '\n\n' }} {%- endif %} {%- endfor %} {{- '<|end_of_text|> ' }} {%- endif %} {%- for message in messages if message['role'] == 'system'%}{% else %}<|system|> A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. {% endfor %}{%- for message in messages %} {%- if message['role'] == 'system' %} {{- '<|system|> ' + message['content'][0]['text'] + ' ' }} {%- elif message['role'] == 'user' %}<|user|> {# Render all images first #}{% for content in message['content'] | selectattr('type', 'equalto', 'image') %}{{ ' ' }}{% endfor %}{# Render all text next #}{% for content in message['content'] | selectattr('type', 'equalto', 'text') %}{{ content['text'] + ' ' }}{% endfor %} {%- elif message['role'] == 'assistant' %} {{- '<|assistant|> ' + message['content'][0]['text'] + '<|end_of_text|>' }} {%- elif message['role'] == 'assistant_tool_call' %} {{- '<|start_of_role|>assistant<|end_of_role|><|tool_call|>' + message['content'][0]['text'] + '<|end_of_text|> ' }} {%- elif message['role'] == 'tool_response' %} {{- '<|start_of_role|>tool_response<|end_of_role|>' + message['content'][0]['text'] + '<|end_of_text|> ' }} {%- endif %} {%- if loop.last and add_generation_prompt %} {{- '<|assistant|> ' }} {%- endif %} {%- endfor %}" ``` -------------------------------- ### Granite Vision 3.2 2B Token Mapping Source: https://huggingface.co/ibm-granite/granite-vision-3.2-2b/blame/main/added_tokens.json Defines the special token IDs used for role management, tool invocation, and image representation in the model. ```json { "<|end_of_role|>": 49153, "<|start_of_role|>": 49152, "<|tool_call|>": 49154, "": "49155" } ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.