### Set Up Granite Docling in Docling SDK
Source: https://www.ibm.com/granite/docs/models/docling
Integrate the Granite Docling model within the Docling SDK for programmatic document conversion. This example shows basic setup and usage with default values, and an alternative using the macOS MPS accelerator.
```python
from docling.datamodel import vlm_model_specs
from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import (
VlmPipelineOptions,
)
from docling.document_converter import DocumentConverter, PdfFormatOption
from docling.pipeline.vlm_pipeline import VlmPipeline
source = "https://arxiv.org/pdf/2501.17887"
###### USING SIMPLE DEFAULT VALUES
# - GraniteDocling model
# - Using the transformers framework
converter = DocumentConverter(
format_options={
InputFormat.PDF: PdfFormatOption(
pipeline_cls=VlmPipeline,
),
}
)
doc = converter.convert(source=source).document
print(doc.export_to_markdown())
###### USING MACOS MPS ACCELERATOR
# For more options see the compare_vlm_models.py example.
pipeline_options = VlmPipelineOptions(
vlm_options=vlm_model_specs.GRANITEDOCLING_MLX,
)
converter = DocumentConverter(
format_options={
InputFormat.PDF: PdfFormatOption(
pipeline_cls=VlmPipeline,
pipeline_options=pipeline_options,
),
}
)
doc = converter.convert(source=source).document
print(doc.export_to_markdown())
```
--------------------------------
### Install vLLM
Source: https://www.ibm.com/granite/docs/models/speech
Install the vLLM library, a high-throughput and memory-efficient inference engine for large language models.
```bash
pip install vllm
```
--------------------------------
### Install Required Libraries
Source: https://www.ibm.com/granite/docs/models/granite4-1
Installs PyTorch, Accelerate, and Transformers libraries necessary for using Granite models.
```bash
pip install torch torchvision torchaudio
pip install accelerate
pip install transformers
```
--------------------------------
### Download Ollama Installer
Source: https://www.ibm.com/granite/docs/run/granite-with-ollama-windows
Use PowerShell to download the Ollama installer executable.
```powershell
$url = 'https://ollama.com/download/OllamaSetup.exe'
$outputPath = 'C:\Downloads\OllamaSetup.exe'
Invoke-WebRequest -Uri $url -OutFile $outputPath
& $outputPath
```
--------------------------------
### Install Flash Attention 2
Source: https://www.ibm.com/granite/docs/models/embedding
Optional installation for faster encoding with Granite Embedding models. This can improve inference speed.
```bash
pip install flash_attn
```
--------------------------------
### Install Sentence Transformers Library
Source: https://www.ibm.com/granite/docs/models/embedding
Install the Sentence Transformers library to easily use the Granite Embedding models for encoding text.
```bash
pip install sentence_transformers
```
--------------------------------
### RAG with watsonx.ai
Source: https://www.ibm.com/granite/docs/models/granite
This snippet demonstrates RAG using watsonx.ai. Ensure you have a watsonx.ai API key and project ID, and have followed the setup guide.
```python
import requests
import json
WATSONX_APIKEY = "WATSONX_APIKEY" # Replace with your watsonx.ai API key
WATSONX_PROJECT_ID = "WATSONX_PROJECT_ID" # Replace with your watsonx.ai project id
# STEP 1: Retrieve bearer token using WATSONX_APIKEY
token_url = "https://iam.cloud.ibm.com/oidc/token"
token_data = {
"grant_type": "urn:ibm:params:oauth:grant-type:apikey",
"apikey": WATSONX_APIKEY,
}
token_headers = {"Content-Type": "application/x-www-form-urlencoded"}
token_response = requests.post(token_url, data=token_data, headers=token_headers)
if token_response.status_code != 200:
raise Exception("Failed to retrieve token: " + token_response.text)
access_token = token_response.json()["access_token"]
# STEP 2: Setup call to the chat endpoint in watsonx.ai
url = "https://us-south.ml.cloud.ibm.com/ml/v1/text/chat?version=2025-10-25"
headers = {
"Accept": "application/json",
"Content-Type": "application/json",
"Authorization": f"Bearer {access_token}",
}
chat = [
{
"role": "user",
"content": "Could you please tell me what the first Bridget Jones's movie is about?, please be brief in your response.",
}
]
documents = [
{
"doc_id": "1",
"title": "Bridget Jones: The Edge of Reason (2004)",
```
--------------------------------
### Install vLLM from Source
Source: https://www.ibm.com/granite/docs/models/vision
Install vLLM from its GitHub repository to use Granite Vision 4.1. This is necessary until an official release includes the support.
```Bash
git clone https://github.com/vllm-project/vllm.git
cd vllm
pip install -e " .[cuda]"
```
--------------------------------
### Install Hugging Face Transformers and PyTorch
Source: https://www.ibm.com/granite/docs/models/embedding
Installs the necessary libraries for using Hugging Face Transformers and PyTorch.
```bash
pip install transformers torch
```
--------------------------------
### Install Transformers and Dependencies
Source: https://www.ibm.com/granite/docs/models/speech
Install the necessary libraries for using Granite Speech models with the Transformers library. Ensure you have a recent version of transformers installed.
```Python
pip install transformers torchaudio peft soundfile
```
--------------------------------
### Install Model Signing Library
Source: https://www.ibm.com/granite/docs/model-standards/signature-verification
Install the model-signing library (v1.0.1) using pip. Ensure you use the specified version for compatibility.
```bash
pip install 'model-signing==v1.1.1'
```
--------------------------------
### Example JSON Output
Source: https://www.ibm.com/granite/docs/models/granite
This is an example of JSON output generated by the system, showing activity details and participant information.
```json
{
"activityID": "KAYAK-0423",
"participantInfo": {
"participantName": "Jamie Patterson",
"age": 27
},
"activityDate": "2023-04-15",
"equipmentNeeded": ["kayak", "paddle", "safety vest"]
}
```
--------------------------------
### Fast Batch Inference with vLLM
Source: https://www.ibm.com/granite/docs/models/docling
Example for fast batch inference using vLLM. This snippet requires vLLM and docling_core to be installed. It assumes page images are placed in an 'img/' directory.
```python
# Prerequisites:
# pip install vllm
# pip install docling_core
# place page images you want to convert into "img/" dir
import time
import os
from vllm import LLM, SamplingParams
from transformers import AutoProcessor
from PIL import Image
```
--------------------------------
### Install LangChain and related libraries
Source: https://www.ibm.com/granite/docs/models/embedding
Installs various LangChain components and utility libraries required for building a retrieval workflow with a vector database.
```Python
pip install langchain_huggingface sentence_transformers \
langchain_milvus 'pymilvus[milvus_lite]' \
langchain_community \
langchain_text_splitters \
wget
```
--------------------------------
### Send Sample Request to vLLM API
Source: https://www.ibm.com/granite/docs/run/granite-with-vllm-containerized
Sends a POST request to the vLLM OpenAI-compatible API endpoint to get a chat completion. This example queries the 'ibm-granite/granite-4.0-h-small' model.
```sh
curl -X POST http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "ibm-granite/granite-4.0-h-small",
"messages": [
{"role": "user", "content": "How are you today?"}
]
}'
```
--------------------------------
### Text Extraction Prompt Example
Source: https://www.ibm.com/granite/docs/use-cases/prompt-engineering
This example demonstrates a prompt designed to extract specific financial information, 'Line Of Credit Facility Maximum Borrowing Capacity', from a given 10K sentence. It includes examples of expected input and output.
```text
<|start_of_role|>user<|end_of_role|>Extract the Line Of Credit Facility Maximum Borrowing Capacity from the 10K sentences.
Your response should only include the answer. Do not provide any further explanation.
Here are some examples, complete the last one:
10K Sentence:
The credit agreement also provides that up to $500 million in commitments may be used for letters of credit.
Line Of Credit Facility Maximum Borrowing Capacity:
$500M
10K Sentence:
In March 2020, we upsized the Credit Agreement by $100 million, which matures July 2023, to $2.525 billion.
Line Of Credit Facility Maximum Borrowing Capacity:
$2.525B
10K Sentence:
We prepared our impairment test as of October 1, 2022 and determined that the fair values of each of our reporting units exceeded net book value by more than 50%. Among our reporting units, the narrowest difference between the calculated fair value and net book value was in our Principal Markets segment's Canada reporting unit, whose calculated fair value exceeded its net book value by 53%. Future developments related to macroeconomic factors, including increases to the discount rate used, or changes to other inputs and assumptions, including revenue growth, could reduce the fair value of this and/or other reporting units and lead to impairment. There were no goodwill impairment losses recorded for the nine months ended December 31, 2022. Cumulatively, the Company has recorded $469 million in goodwill impairment charges within its former EMEA ($293 million) and current United States ($176 million) reporting units. Revolving Credit Agreement In October 2021, we entered into a $3.15 billion multi-currency revolving credit agreement (the "Revolving Credit Agreement") for our future liquidity needs. The Revolving Credit Agreement expires, unless extended, in October 2026. Interest rates on borrowings under the Revolving Credit Agreement are based on prevailing market interest rates, plus a margin, as further described in the Revolving Credit Agreement. The total expense recorded by the Company for the Revolving Credit Agreement was not material in any of the periods presented. We may voluntarily prepay borrowings under the Revolving Credit Agreement without premium or penalty, subject to customary "breakage" costs. The Revolving Credit Agreement includes certain customary mandatory prepayment provisions. Interest on Debt Interest expense for the three and nine months ended December 31, 2022 was $27 million and $65 million, compared to $18 million and $50 million for the three and nine months ended December 31, 2021. Most of the interest for the pre-Separation period presented in the historical Consolidated Income Statement reflects the allocation of interest expense associated with debt issued by IBM from which a portion of the proceeds benefited Kyndryl.
Line Of Credit Facility Maximum Borrowing Capacity:<|end_of_text|>
<|start_of_role|>assistant<|end_of_role|>
$3.15B<|end_of_text|>
```
--------------------------------
### System Prompt with Tool Signatures
Source: https://www.ibm.com/granite/docs/use-cases/prompt-engineering
Example of a system prompt that automatically includes tool signatures for function calls.
```plaintext
<|start_of_role|>system<|end_of_role|>You are a helpful assistant with access to the following tools. You may call one or more tools to assist with the user query.
You are provided with function signatures within XML tags:
{"type": "function", "function": {"name": "get_current_weather", "description": "Get the current weather for a specified city.", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "Name of the city"}}, "required": ["city"]}}}
{"type": "function", "function": {"name": "get_time", "description": "Get the current time for a specified location.", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "Coordinates of the location"}}, "required": ["location"]}}}
For each tool call, return a json object with function name and arguments within XML tags:
{"name": , "arguments": }
. If a tool does not exist in the provided list of tools, notify the user that you do not have the ability to fulfill the request.<|end_of_text|>
```
--------------------------------
### Start Ollama Service
Source: https://www.ibm.com/granite/docs/run/granite-with-ollama-mac
Run this command to start the Ollama background service that manages AI models.
```shell
ollama serve
```
--------------------------------
### Python Setup for Text Classification with Hugging Face
Source: https://www.ibm.com/granite/docs/use-cases/prompt-engineering
This Python code initializes the tokenizer and model from the 'ibm-granite/granite-4.0-h-tiny' path. It prepares the prompt and tokenizes it for model inference.
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda"
model_path = "ibm-granite/granite-4.0-h-tiny"
tokenizer = AutoTokenizer.from_pretrained(model_path)
# drop device_map if running on CPU
model = AutoModelForCausalLM.from_pretrained(model_path, device_map=device)
model.eval()
prompt = """Classify the sentiment of the movie reviews as positive or negative.
Your response should only include the answer. Do not provide any further explanation.
Here are some examples, complete the last one:
Review:
Oh, where do I even begin? Barbie 2023 is a tour de force that has left me utterly captivated, enchanted, and spellbound. Every moment of this cinematic marvel was nothing short of pure excellence, deserving nothing less than a perfect 10 out of 10 rating!
Sentiment:
positive
Review:
It's a shame because I love Little Women and Lady Bird. Barbie did not serve. It's a long ramble about how matriarchy is perfect and patriarchy is stupid. I agree with the latter, but the execution is just awful. Mattel's CEO and corporate people started as a mockery of men having all the top positions in a company; as the movie goes, they're just there at the back of your mind, and it's an unpleasant experience because they did nothing after. There were so many unnecessary scenes that could have been meaningful. For example, what's the scene where Allan fights other Kens for? I'm also surprised that there were no gay Barbies and Kens. And I thought this was supposed to \"break\" gender norms. So many seeds were planted at once, but none reached their full potential.
Sentiment:
negative
Review:
Russell Crowe's new action movie Land of Bad can't break the star's recent streak of poor Rotten Tomatoes ratings. Co-starring Liam and Luke Hemsworth, Crowe's latest action effort concerns a special ops mission in the Philippines that goes wrong, leading to a harrowing fight for survival. Crowe plays Reaper, a drone pilot who must guide a group of outnumbered and stranded soldiers back to safety.
Despite the Land of Bad cast including big names like Crowe and two of the Hemsworths, the action film has failed to impress critics, currently sitting at 52% fresh on Rotten Tomatoes on 24 reviews, becoming the fifth straight Crowe-starring film to be certified rotten by the review aggregator.
Sentiment:"""
chat=[
{"role": "user", "content": prompt}
]
chat = tokenizer.apply_chat_template(chat,tokenize=False, add_generation_prompt=True)
# tokenize the text
input_tokens = tokenizer(chat, return_tensors="pt").to(device)
```
--------------------------------
### Install Ollama on Linux
Source: https://www.ibm.com/granite/docs/run/granite-with-ollama-linux
Use this command to install Ollama on your Linux system. It sets up a systemd service for background operation.
```bash
curl -fsSL https://ollama.com/install.sh | sh
```
--------------------------------
### Granite Chat Template Example
Source: https://www.ibm.com/granite/docs/models/granite
Illustrates the chat template format used by Granite models for structuring conversational input and output.
```text
<|start_of_role|>user<|end_of_role|>What is the largest ocean on Earth?<|end_of_text|>
<|start_of_role|>assistant<|end_of_role|>The largest ocean on Earth is the Pacific Ocean. It covers an area of about 63.8 million square miles (165.25 million square kilometers), which is more than twice the size of the second-largest ocean, the Atlantic Ocean. The Pacific Ocean lies between the Americas to the east and Asia and Australia to the west.<|end_of_text|>
```
--------------------------------
### Example Tool Call for Bengaluru Weather
Source: https://www.ibm.com/granite/docs/models/granite
Another example of an assistant's tool call, this time requesting the weather for Bengaluru. It uses the 'get_current_weather' function with the 'city' argument set to 'Bengaluru'.
```xml
{"name": "get_current_weather", "arguments": {"city": "Bengaluru"}}
```
--------------------------------
### RAG with Transformers
Source: https://www.ibm.com/granite/docs/models/granite
Demonstrates how to perform RAG using the Transformers library. Ensure you have the necessary libraries installed and a compatible device (e.g., CUDA).
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda"
model_path = "ibm-granite/granite-4.0-h-tiny"
tokenizer = AutoTokenizer.from_pretrained(model_path)
# drop device_map if running on CPU
model = AutoModelForCausalLM.from_pretrained(model_path, device_map=device)
model.eval()
chat=[
{"role": "user", "content": "Could you please tell me what the first Bridget Jones's movie is about?, please be brief in your response."}
]
documents=[
{
"doc_id": 1,
"title": "Bridget Jones: The Edge of Reason (2004)",
"text": "Bridget Jones: The Edge of Reason (2004) - Bridget is currently living a happy life with her lawyer boyfriend Mark Darcy, however not only does she start to become threatened and jealous of Mark's new young intern, she is angered by the fact Mark is a Conservative voter. With so many issues already at hand, things get worse for Bridget as her ex-lover, Daniel Cleaver, re-enters her life; the only help she has are her friends and her reliable diary.",
"source": ""
},
{
"doc_id": 2,
"title": "Bridget Jones's Baby (2016)",
"text": "Bridget Jones's Baby (2016) - Bridget Jones is struggling with her current state of life, including her break up with her love Mark Darcy. As she pushes forward and works hard to find fulfillment in her life seems to do wonders until she meets a dashing and handsome American named Jack Quant. Things from then on go great, until she discovers that she is pregnant but the biggest twist of all, she does not know if Mark or Jack is the father of her child.",
"source": ""
},
{
"doc_id": 3,
"title": "Bridget Jones's Diary (2001)",
"text": "Bridget Jones's Diary (2001) - Bridget Jones is a binge drinking and chain smoking thirty-something British woman trying to keep her love life in order while also dealing with her job as a publisher. When she attends a Christmas party with her parents, they try to set her up with their neighbours' son, Mark. After being snubbed by Mark, she starts to fall for her boss Daniel, a handsome man who begins to send her suggestive e-mails that leads to a dinner date. Daniel reveals that he and Mark attended college together, in that time Mark had an affair with his fiancée. Bridget decides to get a new job as a TV presenter after finding Daniel being frisky with a colleague. At a dinner party, she runs into Mark who expresses his affection for her, Daniel claims he wants Bridget back, the two fight over her and Bridget must make a decision who she wants to be with.",
"source": ""
},
]
chat = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True, documents=documents)
# tokenize the text
input_tokens = tokenizer(chat, return_tensors="pt").to(device)
# generate output tokens
output = model.generate(**input_tokens,
max_new_tokens=800, temperature=0)
# decode output tokens into text
output = tokenizer.batch_decode(output)
# print output
print(output[0])
```
--------------------------------
### System Prompt with Document Integration
Source: https://www.ibm.com/granite/docs/use-cases/prompt-engineering
Example of a system prompt that automatically includes document information for the assistant to use.
```plaintext
<|start_of_role|>system<|end_of_role|>You are a helpful assistant with access to the following documents. You may use one or more documents to assist with the user query.
You are given a list of documents within XML tags:
{"doc_id": 1, "title": "History Document Title", "text": "From the early 12th century, French builders developed the Gothic style, marked by the use of rib vaults, pointed arches, flying buttresses, and large stained glass windows. It was used mainly in churches and cathedrals, and continued in use until the 16th century in much of Europe. Classic examples of Gothic architecture include Chartres Cathedral and Reims Cathedral in France as well as Salisbury Cathedral in England. Stained glass became a crucial element in the design of churches, which continued to use extensive wall-paintings, now almost all lost.", "source": ""}
```
--------------------------------
### Chat Template Example
Source: https://www.ibm.com/granite/docs/models/granite
Illustrates a chat template format used for instructing the model on its role and available tools. This format is often used in conjunction with tool-calling capabilities.
```text
<|start_of_role|>system<|end_of_role|>You are a helpful assistant with access to the following tools. You may call one or more tools to assist with the user query.
You are provided with function signatures within XML tags:
```
--------------------------------
### Basic Chat Template Example
Source: https://www.ibm.com/granite/docs/use-cases/prompt-engineering
This snippet shows the format for a basic chat template used with Granite 4.0 models. It defines user and assistant roles and their respective content.
```text
<|start_of_role|>user<|end_of_role|>What is the largest ocean on Earth?<|end_of_text|>
<|start_of_role|>assistant<|end_of_role|>The largest ocean on Earth is the Pacific Ocean. It covers an area of about 63.8 million square miles (165.25 million square kilometers), which is more than twice the size of the second-largest ocean, the Atlantic Ocean. The Pacific Ocean lies between the Americas to the east and Asia and Australia to the west.<|end_of_text|>
```
--------------------------------
### Text Classification Prompt with Examples
Source: https://www.ibm.com/granite/docs/use-cases/prompt-engineering
This prompt guides the model to classify movie review sentiment by providing examples. It's designed for few-shot learning to improve accuracy.
```text
<|start_of_role|>user<|end_of_role|>Classify the sentiment of the movie reviews as positive or negative.
Your response should only include the answer. Do not provide any further explanation.
Here are some examples, complete the last one:
Review:
Oh, where do I even begin? Barbie 2023 is a tour de force that has left me utterly captivated, enchanted, and spellbound. Every moment of this cinematic marvel was nothing short of pure excellence, deserving nothing less than a perfect 10 out of 10 rating!
Sentiment:
positive
Review:
It's a shame because I love Little Women and Lady Bird. Barbie did not serve. It's a long ramble about how matriarchy is perfect and patriarchy is stupid. I agree with the latter, but the execution is just awful. Mattel's CEO and corporate people started as a mockery of men having all the top positions in a company; as the movie goes, they're just there at the back of your mind, and it's an unpleasant experience because they did nothing after. There were so many unnecessary scenes that could have been meaningful. For example, what's the scene where Allan fights other Kens for? I'm also surprised that there were no gay Barbies and Kens. And I thought this was supposed to "break" gender norms. So many seeds were planted at once, but none reached their full potential.
Sentiment:
negative
Review:
Russell Crowe's new action movie Land of Bad can't break the star's recent streak of poor Rotten Tomatoes ratings. Co-starring Liam and Luke Hemsworth, Crowe's latest action effort concerns a special ops mission in the Philippines that goes wrong, leading to a harrowing fight for survival. Crowe plays Reaper, a drone pilot who must guide a group of outnumbered and stranded soldiers back to safety.
Despite the Land of Bad cast including big names like Crowe and two of the Hemsworths, the action film has failed to impress critics, currently sitting at 52% fresh on Rotten Tomatoes on 24 reviews, becoming the fifth straight Crowe-starring film to be certified rotten by the review aggregator.
Sentiment:<|end_of_text|>
<|start_of_role|>assistant<|end_of_role|>positive<|end_of_text|>
```
--------------------------------
### Example Tool Call for Current Weather
Source: https://www.ibm.com/granite/docs/models/granite
An example of an assistant's response making a tool call to get the current weather in New York. The 'get_current_weather' function is invoked with the 'city' argument set to 'New York'.
```xml
{"name": "get_current_weather", "arguments": {"city": "New York"}}
```
--------------------------------
### Build a Retrieval Workflow with LangChain and Milvus
Source: https://www.ibm.com/granite/docs/models/embedding
Demonstrates setting up a vector database using Milvus and LangChain, loading documents, splitting them into chunks, adding them to the database, and performing a similarity search.
```Python
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_milvus import Milvus
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import CharacterTextSplitter
import os, tempfile, wget
# load the embedding model
embeddings_model = HuggingFaceEmbeddings(model_name="ibm-granite/granite-embedding-30m-english")
# setup the vectordb
db_file = tempfile.NamedTemporaryFile(prefix="milvus_", suffix=".db", delete=False).name
print(f"The vector database will be saved to {db_file}")
vector_db = Milvus(
embedding_function=embeddings_model,
connection_args={"uri": db_file},
auto_id=True,
index_params={"index_type": "AUTOINDEX"},
)
# load example corpus file
filename = 'state_of_the_union.txt'
url = 'https://raw.github.com/IBM/watson-machine-learning-samples/master/cloud/data/foundation_models/state_of_the_union.txt'
if not os.path.isfile(filename):
wget.download(url, out=filename)
loader = TextLoader(filename)
documents = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=500, chunk_overlap=0)
texts = text_splitter.split_documents(documents)
# add processed documents to the vectordb
vector_db.add_documents(texts)
# search the vectordb with the query
query = "What did the president say about Ketanji Brown Jackson"
docs = vector_db.similarity_search(query)
print(docs[0].page_content)
```
--------------------------------
### Single-Page Inference with Transformers
Source: https://www.ibm.com/granite/docs/models/docling
Perform single-page image inference using plain transformers. Ensure you have torch, docling_core, and transformers installed. This example processes an image and outputs DocTags and Markdown.
```python
# Prerequisites:
# pip install torch
# pip install docling_core
# pip install transformers
import torch
from docling_core.types.doc import DoclingDocument
from docling_core.types.doc.document import DocTagsDocument
from transformers import AutoProcessor, AutoModelForVision2Seq
from transformers.image_utils import load_image
from pathlib import Path
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
# Load images
image = load_image("https://huggingface.co/ibm-granite/granite-docling-258M/resolve/main/assets/new_arxiv.png")
# Initialize processor and model
processor = AutoProcessor.from_pretrained("ibm-granite/granite-docling-258M")
model = AutoModelForVision2Seq.from_pretrained(
"ibm-granite/granite-docling-258M",
torch_dtype=torch.bfloat16,
_attn_implementation="flash_attention_2" if DEVICE == "cuda" else "sdpa",
).to(DEVICE)
# Create input messages
messages = [
{
"role": "user",
"content": [
{"type": "image"},
{"type": "text", "text": "Convert this page to docling."}
]
},
]
# Prepare inputs
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=[image], return_tensors="pt")
inputs = inputs.to(DEVICE)
# Generate outputs
generated_ids = model.generate(**inputs, max_new_tokens=8192)
prompt_length = inputs.input_ids.shape[1]
trimmed_generated_ids = generated_ids[:, prompt_length:]
doctags = processor.batch_decode(
trimmed_generated_ids,
skip_special_tokens=False,
)[0].lstrip()
print(f"DocTags: \n{doctags}\n")
# Populate document
doctags_doc = DocTagsDocument.from_doctags_and_image_pairs([doctags], [image])
# create a docling document
doc = DoclingDocument.load_from_doctags(doctags_doc, document_name="Document")
print(f"Markdown:\n{doc.export_to_markdown()}\n")
## export as any format.
# Path("out/").mkdir(parents=True, exist_ok=True)
# HTML:
# output_path_html = Path("out/") / "example.html"
# doc.save_as_html(output_path_html)
# Markdown:
# output_path_md = Path("out/") / "example.md"
# doc.save_as_markdown(output_path_md)
```
--------------------------------
### Setup and Utility Functions for Granite Guardian
Source: https://www.ibm.com/granite/docs/models/guardian
This snippet sets up the necessary imports, loads the Granite Guardian model and tokenizer, and defines utility functions for building judge prompts and parsing model outputs. It includes instructions for both 'think' and 'no-think' modes.
```Python
import json
import re
from vllm import LLM, SamplingParams
from transformers import AutoTokenizer
model_path = "ibm-granite/granite-guardian-4.1-8b"
# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_path)
llm = LLM(model=model_path, max_model_len=8192)
sampling_params = SamplingParams(temperature=0.0, max_tokens=2048)
# Guardian judge instructions for think / no-think modes
GUARDIAN_JUDGE_THINK = (
"As a judge agent, carefully analyze whether the provided text meets the "
"given judging criteria. Use all available information, including conversations, "
"documents, and tools. Reason through the evidence step by step before providing "
"your score. Wrap your reasoning in tags, then provide your "
"score in tags."
)
GUARDIAN_JUDGE_NOTHINK = (
"As a judge agent, assess whether the provided text meets the given "
"judging criteria using all available information, including conversations, "
"documents, and tools. Provide your score immediately without explanation. "
"Output empty \n tags followed by your score in tags."
)
def build_guardian_block(criteria, think=False):
judge_instruction = GUARDIAN_JUDGE_THINK if think else GUARDIAN_JUDGE_NOTHINK
return (
f"{judge_instruction}\n\n"
f"### Criteria: {criteria}\n\n"
f"### Scoring Schema: If the last assistant's text meets the criteria, "
f"return 'yes'; otherwise, return 'no'."
)
def parse_output(text):
text_clean = re.sub(r".*?", "", text, flags=re.DOTALL).strip()
match = re.findall(r"\s*(.*?)\s*", text_clean, re.DOTALL)
if match:
return match[0].strip().lower()
return None
```
--------------------------------
### Granite Chat Template with Tool-Use and RAG
Source: https://www.ibm.com/granite/docs/use-cases/prompt-engineering
This example demonstrates how a Granite chat template can combine tool-use and RAG capabilities. The order of elements in the template is important: tools are listed first, followed by documents. A user-defined system prompt, if provided, will appear at the very beginning.
```text
<|start_of_role|>system<|end_of_role|>You are a weather assistant that responds with relevant function calls instead of natural language.
You are a helpful assistant with access to the following tools. You may call one or more tools to assist with the user query.
You are provided with function signatures within XML tags:
{"type": "function", "function": {"name": "get_current_weather", "description": "Get the current weather for a specified city.", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "Name of the city"}}, "required": ["city"]}}}
{"type": "function", "function": {"name": "get_time", "description": "Get the current time for a specified location.", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "Coordinates of the location"}}, "required": ["location"]}}}
For each tool call, return a json object with function name and arguments within XML tags:
{"name": , "arguments": }
. If a tool does not exist in the provided list of tools, notify the user that you do not have the ability to fulfill the request.
You are a helpful assistant with access to the following documents. You may use one or more documents to assist with the user query.
You are given a list of documents within XML tags:
{"doc_id": 1, "title": "", "text": "From the early 12th century, French builders developed the Gothic style, marked by the use of rib vaults, pointed arches, flying buttresses, and large stained glass windows. It was used mainly in churches and cathedrals, and continued in use until the 16th century in much of Europe. Classic examples of Gothic architecture include Chartres Cathedral and Reims Cathedral in France as well as Salisbury Cathedral in England. Stained glass became a crucial element in the design of churches, which continued to use extensive wall-paintings, now almost all lost.", "source": ""}
{"doc_id": 2, "title": "", "text": "From long time ago, French builders developed the Gothic style, marked by the use of rib vaults, pointed arches, flying buttresses, and large stained glass windows. It was used mainly in churches and cathedrals, and continued in use until the 16th century in much of Europe. Classic examples of Gothic architecture include Chartres Cathedral and Reims Cathedral in France as well as Salisbury Cathedral in England. Stained glass became a crucial element in the design of churches, which continued to use extensive wall-paintings, now almost all lost.", "source": ""}
```
--------------------------------
### Setup and Call OpenAI Compatible Endpoint
Source: https://www.ibm.com/granite/docs/models/granite
Configures and executes a call to an OpenAI compatible endpoint for chat completions. This snippet assumes the endpoint is set up, potentially via vLLM.
```python
import requests
import json
# STEP 1: Setup call to the OpenAI compatible endpoint
url = "/chat/completions"
headers = {
"Accept": "application/json",
"Content-Type": "application/json",
}
chat = [
{"role": "user", "content": "What's the current weather in New York?"},
{
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "chatcmpl-tool-a5ecf628c4fd4be3995cadb1ba13c8ad",
"type": "function",
"function": {
"name": "get_current_weather",
"arguments": '{"city": "New York"}',
},
}
],
},
{
"role": "tool",
"content": "New York is sunny with a temperature of 30°C.",
"tool_call_id": "chatcmpl-tool-a5ecf628c4fd4be3995cadb1ba13c8ad",
},
{
"role": "user",
"content": "OK, Now tell me what's the weather like in Bengaluru at this moment?",
},
]
tools = [
{
"type": "function",
"function": {
"name": "get_current_weather",
"description": "Get the current weather",
"parameters": {
"type": "object",
"properties": {
"location": {
"description": "The city and state, e.g. San Francisco, CA",
"type": "string",
},
},
"required": ["location"],
},
},
},
{
"type": "function",
"function": {
"name": "get_stock_price",
"description": "Retrieves the current stock price for a given ticker symbol. The ticker symbol must be a valid symbol for a publicly traded company on a major US stock exchange like NYSE or NASDAQ. The tool will return the latest trade price in USD. It should be used when the user asks about the current or most recent price of a specific stock. It will not provide any other information about the stock or company.",
"parameters": {
"type": "object",
"properties": {
"ticker": {
"description": "The stock ticker symbol, e.g. AAPL for Apple Inc.",
"type": "string",
},
},
},
},
},
]
body = {
"messages": chat,
"tools": tools,
"model_id": "ibm-granite/granite-4.0-h-small",
"max_completion_tokens": 2000,
"temperature": 0,
}
# STEP 2: Execute call to the chat endpoint
response = requests.post(url, headers=headers, json=body)
if response.status_code != 200:
raise Exception("Non-200 response: " + str(response.text))
data = response.json()
print(json.dumps(data, indent=4))
```
--------------------------------
### Install Ollama via Homebrew
Source: https://www.ibm.com/granite/docs/run/granite-with-ollama-mac
Use this command to install Ollama using the Homebrew package manager on macOS.
```shell
brew install ollama
```
--------------------------------
### Define Training Arguments for Fine-tuning
Source: https://www.ibm.com/granite/docs/fine-tune/time-series
Configure training parameters such as output directory, learning rate, epochs, batch size, and evaluation strategy. Adjust these arguments based on your specific dataset and use case.
```python
finetune_forecast_args = TrainingArguments(
output_dir="train_output",
overwrite_output_dir=True,
learning_rate=0.001,
num_train_epochs=100,
do_eval=True,
evaluation_strategy="epoch",
per_device_train_batch_size=64,
per_device_eval_batch_size=64,
dataloader_num_workers=8,
report_to=None,
save_strategy="epoch",
logging_strategy="epoch",
save_total_limit=1,
logging_dir="train_logs",
load_best_model_at_end=True,
metric_for_best_model="eval_loss",
greater_is_better=False,
)
```
--------------------------------
### Prepare Time Series Data for Fine-tuning
Source: https://www.ibm.com/granite/docs/fine-tune/time-series
Initializes the TimeSeriesPreprocessor to handle target and control columns, sets prediction and context lengths, and enables standard scaling. It then extracts training, validation, and testing datasets.
```Python
from granite_tsfm.preprocessing import TimeSeriesPreprocessor
from granite_tsfm.datasets import get_datasets
tsp = TimeSeriesPreprocessor(
id_columns=[],
timestamp_column="date"
target_columns=["value1", "value2"],
control_columns=["value3"],
prediction_length=96,
context_length=512,
scaling=True,
scaling_type="standard",
)
dset_train, dset_valid, dset_test = get_datasets(tsp, data, split_config={"train": 0.6, "test": 0.2})
```
--------------------------------
### Set Up Early Stopping and Tracking Callbacks
Source: https://www.ibm.com/granite/docs/fine-tune/time-series
Implement callbacks for early stopping to prevent overfitting and a tracking callback for monitoring training statistics. Early stopping halts training when performance plateaus.
```python
early_stopping_callback = EarlyStoppingCallback(
early_stopping_patience=10, # Number of epochs with no improvement after which to stop
early_stopping_threshold=0.0, # Minimum improvement required to consider as improvement
)
tracking_callback = TrackingCallback()
```
--------------------------------
### Install Dependencies for Granite Vision
Source: https://www.ibm.com/granite/docs/models/vision
Installs PyTorch with CUDA support and necessary Hugging Face Transformers libraries. Ensure your environment matches the tested Python version (3.11).
```bash
pip install torch==2.10.0 --index-url https://download.pytorch.org/whl/cu128
pip install transformers==4.57.6 peft==0.18.1 tokenizers==0.22.2 pillow==12.1.1
```
--------------------------------
### Initialize Tokenizer and Model
Source: https://www.ibm.com/granite/docs/use-cases/prompt-engineering
This Python snippet demonstrates how to load a pre-trained tokenizer and model from the Hugging Face Hub for use with IBM Granite models. It specifies the device to be used (e.g., 'cuda').
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda"
model_path = "ibm-granite/granite-4.0-h-tiny"
tokenizer = AutoTokenizer.from_pretrained(model_path)
```
--------------------------------
### Query vLLM Server with OpenAI API
Source: https://www.ibm.com/granite/docs/models/vision
Query a running vLLM server using the OpenAI-compatible API. This example demonstrates how to send image and text prompts for chart and table tasks.
```Python
import base64
from openai import OpenAI
from huggingface_hub import hf_hub_download
from PIL import Image
model_id = "ibm-granite/granite-vision-4.1-4b"
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
def run_inference(client, model_id, image_path, tag):
with open(image_path, "rb") as f:
image_b64 = base64.b64encode(f.read()).decode("utf-8")
messages = [
{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}},
{"type": "text", "text": tag},
]}
]
response = client.chat.completions.create(
model=model_id, messages=messages, max_tokens=4096, temperature=0,
)
return response.choices[0].message.content
chart_path = hf_hub_download(repo_id=model_id, filename="chart.jpg")
table_path = hf_hub_download(repo_id=model_id, filename="table.png")
# Chart tasks
for tag in ["", "", ""]:
result = run_inference(client, model_id, chart_path, tag)
print(f"{tag}:\n{result}\n")
# Table tasks
for tag in ["", "", ""]:
result = run_inference(client, model_id, table_path, tag)
print(f"{tag}:\n{result}\n")
```
--------------------------------
### Run Granite Model Interactively
Source: https://www.ibm.com/granite/docs/run/granite-with-ollama-linux
Starts an interactive chat session with the specified Granite model in the terminal.
```bash
ollama run ibm/granite4
```
--------------------------------
### User-Defined System Prompt
Source: https://www.ibm.com/granite/docs/use-cases/prompt-engineering
Example of a user-defined system prompt for a weather assistant that uses function calls.
```plaintext
<|start_of_role|>system<|end_of_role|>You are a weather assistant that responds with relevant function calls instead of natural language.<|end_of_text|>
```
--------------------------------
### Define Tools and Generate Tool Calls with Granite-4.1-30B
Source: https://www.ibm.com/granite/docs/models/granite4-1
This Python snippet demonstrates how to set up the Granite-4.1-30B model, define tools using OpenAI's function definition schema, and generate a tool call based on a user's query. Ensure you have the transformers library installed and the model downloaded.
```Python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda"
model_path = "ibm-granite/granite-4.1-30b"
tokenizer = AutoTokenizer.from_pretrained(model_path)
# drop device_map if running on CPU
model = AutoModelForCausalLM.from_pretrained(model_path, device_map=device)
model.eval()
tools = [
{
"type": "function",
"function": {
"name": "get_current_weather",
"description": "Get the current weather for a specified city.",
"parameters": {
"type": "object",
"properties": {
"city": {
"type": "string",
"description": "Name of the city"
}
},
"required": ["city"]
}
}
}
]
# change input text as desired
chat = [
{ "role": "user", "content": "What's the weather like in Boston right now?" },
]
chat = tokenizer.apply_chat_template(chat,
tokenize=False,
tools=tools,
add_generation_prompt=True)
# tokenize the text
input_tokens = tokenizer(chat, return_tensors="pt").to(device)
# generate output tokens
output = model.generate(**input_tokens,
max_new_tokens=100)
# decode output tokens into text
output = tokenizer.batch_decode(output)
# print output
print(output[0])
```