IBM Granite

https://www.ibm.com/granite/docs/
Admin
IBM Granite is a family of open-source AI models including language models, embedding models, speech...
Tokens:63,868
Snippets:389
Trust Score:10
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### Install Sentence Transformers

Source: https://www.ibm.com/granite/docs/models/embedding

Installs the SentenceTransformer library required for the usage examples.

```bash
pip install sentence_transformers  
```

--------------------------------

### Install Transformers and PyTorch

Source: https://www.ibm.com/granite/docs/models/embedding

Installs the Transformers library and PyTorch required for the Hugging Face Transformers usage example.

```bash
pip install transformers torch  
```

--------------------------------

### Start Ollama service

Source: https://www.ibm.com/granite/docs/run/granite-with-ollama-mac

Starts the Ollama background service that manages models. Run this after installation.

```bash
ollama serve  
```

--------------------------------

### Install vLLM

Source: https://www.ibm.com/granite/docs/models/speech

Install the vLLM library using pip.

```bash
pip install vllm  
```

--------------------------------

### Install Flash Attention 2

Source: https://www.ibm.com/granite/docs/models/embedding

Optional installation for faster encoding with Flash Attention 2 support.

```bash
pip install flash_attn  
```

--------------------------------

### Download and run Ollama installer with PowerShell

Source: https://www.ibm.com/granite/docs/run/granite-with-ollama-windows

Downloads the Ollama installer from the official URL and runs it. Use this in PowerShell on Windows to install Ollama without manually downloading.

```powershell
$url = 'https://ollama.com/download/OllamaSetup.exe'  

$outputPath = 'C:\Downloads\OllamaSetup.exe'  

Invoke-WebRequest -Uri $url -OutFile $outputPath  

& $outputPath  

```

--------------------------------

### Install vLLM from source for Granite Vision 4.1

Source: https://www.ibm.com/granite/docs/models/vision

Install vLLM from source to get native support for Granite Vision 4.1, which is available as of commit d249a9e. Until an official release ships, this is required before serving the model.

```bash
git clone https://github.com/vllm-project/vllm.git  

cd vllm  

pip install -e ".[cuda]"  

```

--------------------------------

### Install model_signing library

Source: https://www.ibm.com/granite/docs/model-standards/signature-verification

Install the model_signing (v1.0.1) library using pip.

```bash
pip install 'model-signing==v1.1.1'  
```

--------------------------------

### Install required libraries for Granite 4.2

Source: https://www.ibm.com/granite/docs/models/granite4-2

Install torch, torchvision, torchaudio, accelerate, and transformers via pip before using the model.

```bash
pip install torch torchvision torchaudio  

pip install accelerate  

pip install transformers  
```

--------------------------------

### Install dependencies for Granite-4.1-30B

Source: https://www.ibm.com/granite/docs/models/granite4-1

Install the required libraries for using Granite-4.1-30B: PyTorch, Accelerate, and Transformers.

```bash
pip install torch torchvision torchaudio  

pip install accelerate  

pip install transformers  

```

--------------------------------

### Logical reasoning example with chat format

Source: https://www.ibm.com/granite/docs/use-cases/prompt-engineering

Example of a basic logical reasoning question and its expected assistant response, showing the chat format with role markers and end-of-text tokens.

```text
<|start_of_role|>user<|end_of_role|>If both statements below are true:  

  

If a person is a teacher, then they work at a school.  

  

Maria does not work at a school.  

  

What can we conclude about Maria?<|end_of_text|>  

<|start_of_role|>assistant<|end_of_role|>We can conclude that Maria is not a teacher.  

  

Here's the reasoning:  

  

1. The first statement says that if a person is a teacher, then they work at a school. This can be written as: Teacher → Works at a school.  

2. The second statement tells us that Maria does not work at a school. This can be written as: Not Works at a school.  

3. Since the first statement establishes a conditional relationship between being a teacher and working at a school, we can use the contrapositive of this statement to draw a conclusion about Maria.  

4. The contrapositive of "Teacher → Works at a school" is "Not Works at a school → Not a teacher."  

5. Since Maria does not work at a school (Not Works at a school), we can conclude that Maria is not a teacher (Not a teacher).  

  

Therefore, based on the given statements, we can conclude that Maria is not a teacher.<|end_of_text|>  


```

--------------------------------

### Example Chat Template Output

Source: https://www.ibm.com/granite/docs/models/granite4-0

Example chat template output showing the user and assistant roles with special tokens. Demonstrates the expected format for Granite 4 models.

```text
<|start_of_role|>user<|end_of_role|>What is the largest ocean on Earth?<|end_of_text|>  

<|start_of_role|>assistant<|end_of_role|>The largest ocean on Earth is the Pacific Ocean. It covers an area of about 63.8 million square miles (165.25 million square kilometers), which is more than twice the size of the second-largest ocean, the Atlantic Ocean. The Pacific Ocean lies between the Americas to the east and Asia and Australia to the west.<|end_of_text|>  

```

--------------------------------

### Install Ollama on Linux

Source: https://www.ibm.com/granite/docs/run/granite-with-ollama-linux

Installs Ollama via the automatic script, which requires root access and sets up a systemd service named ollama.service.

```bash
curl -fsSL https://ollama.com/install.sh | sh  
```

--------------------------------

### Construct and run the Trainer

Source: https://www.ibm.com/granite/docs/fine-tune/time-series

Builds the Hugging Face Trainer with the model, training arguments, datasets, callbacks, and optimizer/scheduler, then calls train() to start fine-tuning.

```python
finetune_forecast_trainer = Trainer(  

  model=model,  

  args=finetune_forecast_args,  

  train_dataset=dset_train,  

  eval_dataset=dset_val,  

  callbacks=[early_stopping_callback, tracking_callback],  

  optimizers=(optimizer, scheduler),  

)  

finetune_forecast_trainer.train()  
```

--------------------------------

### FIM chat template example with assistant response

Source: https://www.ibm.com/granite/docs/models/granite4-0

Example of a FIM chat template with the user prompt containing the FIM tokens and the assistant response completing the fibonacci function. Shows the expected output format with role markers and end-of-text tokens.

```text
<|start_of_role|>user<|end_of_role|><|fim_prefix|>  

def fibonacci(n):  

result =  

<|fim_suffix|>  

return result  

<|fim_middle|><|end_of_text|>  

<|start_of_role|>assistant<|end_of_role|>    if n == 0:  

    return 0  

elif n == 1:  

    return 1  

else:  

    return fibonacci(n-1) + fibonacci(n-2)  

<|end_of_text|>  
```

--------------------------------

### Online mode: Launch vLLM server

Source: https://www.ibm.com/granite/docs/models/speech

Start the vLLM server for the Granite Speech model with an API key and a maximum model length of 2048.

```bash
vllm serve ibm-granite/granite-speech-4.1-2b \  

    --api-key token-abc123 \  

    --max-model-len 2048  
```

--------------------------------

### Requirement checking with Granite Guardian

Source: https://www.ibm.com/granite/docs/models/guardian

Shows how to use Granite Guardian to judge whether a response meets a user-defined requirement, such as a formatting rule. The example defines a criteria string and uses build_guardian_block with think=True to construct the guardrail message. The printed score indicates whether the requirement is met; here it is 'no' because the fourth line starts with a lowercase letter.

```python
user_text = "Write a short poem about the ocean. Use exactly 4 lines. Each line must start with a capital letter."  

response_text = "Waves crash upon the sandy shore,\nBeneath the moonlit sky so bright,\nThe ocean sings forevermore,\na lullaby into the night."  

  

think = True  

criteria = "Each line of the response starts with a capital letter."  

  

messages = [  

    {"role": "user", "content": user_text},  

    {"role": "assistant", "content": response_text},  

    {"role": "user", "content": build_guardian_block(criteria, think=think)},  

]  

  

prompt = tokenizer.apply_chat_template(  

    messages, tokenize=False, add_generation_prompt=True  

)  

  

output = llm.generate([prompt], sampling_params)  

response = output[0].outputs[0].text.strip()  

score = parse_output(response)  

print(f"Requirement met: {score}")  # no (4th line starts with lowercase "a")  
```

--------------------------------

### Install Ollama with Homebrew

Source: https://www.ibm.com/granite/docs/run/granite-with-ollama-mac

Installs Ollama on macOS via Homebrew. The desktop app is an alternative installation method.

```bash
brew install ollama  
```

--------------------------------

### Install dependencies for Granite Vision 4.1 4B

Source: https://www.ibm.com/granite/docs/models/vision

Install the required packages for Granite Vision 4.1 4B. Tested with Python 3.11. The first command installs PyTorch 2.10.0 with CUDA 12.8 support; the second installs the transformers, peft, tokenizers, and pillow versions.

```bash
pip install torch==2.10.0 --index-url https://download.pytorch.org/whl/cu128  

pip install transformers==4.57.6 peft==0.18.1 tokenizers==0.22.2 pillow==12.1.1  

```

--------------------------------

### Install dependencies for Granite Speech

Source: https://www.ibm.com/granite/docs/models/speech

Install the required packages for using Granite Speech with Transformers. Requires transformers>=4.52.1.

```bash
pip install transformers torchaudio peft soundfile  
```

--------------------------------

### Setup imports, load model, and define utility functions for Granite Guardian

Source: https://www.ibm.com/granite/docs/models/guardian

This snippet sets up the environment for using Granite Guardian with vLLM and Hugging Face tokenizer. It loads the model 'ibm-granite/granite-guardian-4.1-8b', defines sampling parameters, and includes judge instructions for think and no-think modes. It also provides functions to build a guardian block and parse the model output, extracting the score from <score> tags.

```python
import json  

import re  

from vllm import LLM, SamplingParams  

from transformers import AutoTokenizer  

  

model_path = "ibm-granite/granite-guardian-4.1-8b"  

  

# Load tokenizer and model  

tokenizer = AutoTokenizer.from_pretrained(model_path)  

llm = LLM(model=model_path, max_model_len=8192)  

sampling_params = SamplingParams(temperature=0.0, max_tokens=2048)  

  

# Guardian judge instructions for think / no-think modes  

GUARDIAN_JUDGE_THINK = (  

    " thinkingAs a judge agent, carefully analyze whether the provided text meets the "  

    "given judging criteria. Use all available information, including conversations, "  

    "documents, and tools. Reason through the evidence step by step before providing "  

    "your score. Wrap your reasoning in  thinking response tags, then provide your "  

    "score in <score></score> tags."  

)  

GUARDIAN_JUDGE_NOTHINK = (  

    "<no-think>As a judge agent, assess whether the provided text meets the given "  

    "judging criteria using all available information, including conversations, "  

    "documents, and tools. Provide your score immediately without explanation. "  

    "Output empty  thinking\n response tags followed by your score in <score></score> tags."  

)  

  

def build_guardian_block(criteria, think=False):  

    judge_instruction = GUARDIAN_JUDGE_THINK if think else GUARDIAN_JUDGE_NOTHINK  

    return (  

        f"<guardian>{judge_instruction}\n\n"  

        f"### Criteria: {criteria}\n\n"  

        f"### Scoring Schema: If the last assistant's text meets the criteria, "  

        f"return 'yes'; otherwise, return 'no'."  

    )  

  

def parse_output(text):  

    text_clean = re.sub(r" thinking.*? response", "", text, flags=re.DOTALL).strip()  

    match = re.findall(r"<score>\s*(.*?)\s*</score>", text_clean, re.DOTALL)  

    if match:  

        return match[0].strip().lower()  

    return None  

  

```

--------------------------------

### Client example for Granite Vision 4.1 with OpenAI API

Source: https://www.ibm.com/granite/docs/models/vision

Query the vLLM server using the OpenAI-compatible API. It downloads sample chart and table images from the model repo and runs chart tasks (<chart2csv>, <chart2summary>, <chart2code>) and table tasks (<tables_json>, <tables_html>, <tables_otsl>).

```python
import base64  
from openai import OpenAI  
from huggingface_hub import hf_hub_download  
from PIL import Image  

model_id = "ibm-granite/granite-vision-4.1-4b"  
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")  

def run_inference(client, model_id, image_path, tag):  
    with open(image_path, "rb") as f:  
        image_b64 = base64.b64encode(f.read()).decode("utf-8")  
    messages = [  
        {"role": "user", "content": [  
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}},  
            {"type": "text", "text": tag},  
        ]}  
    ]  
    response = client.chat.completions.create(  
        model=model_id, messages=messages, max_tokens=4096, temperature=0,  
    )  
    return response.choices[0].message.content  

chart_path = hf_hub_download(repo_id=model_id, filename="chart.jpg")  
table_path = hf_hub_download(repo_id=model_id, filename="table.png")  

# Chart tasks  
for tag in ["<chart2csv>", "<chart2summary>", "<chart2code>"]:  
    result = run_inference(client, model_id, chart_path, tag)  
    print(f"{tag}:\n{result}\n")  

# Table tasks  
for tag in ["<tables_json>", "<tables_html>", "<tables_otsl>"]:  
    result = run_inference(client, model_id, table_path, tag)  
    print(f"{tag}:\n{result}\n")  

```

--------------------------------

### Use Granite embedding model with LangChain and Milvus for similarity search

Source: https://www.ibm.com/granite/docs/models/embedding

Loads the Granite embedding model, creates a Milvus vector database, ingests a sample text file, and performs a similarity search. The example downloads the state_of_the_union.txt file if not present and prints the top result's content.

```python
from langchain_huggingface import HuggingFaceEmbeddings  
from langchain_milvus import Milvus  
from langchain_community.document_loaders import TextLoader  
from langchain_text_splitters import CharacterTextSplitter  
import os, tempfile, wget  

# load the embedding model  
embeddings_model = HuggingFaceEmbeddings(model_name="ibm-granite/granite-embedding-30m-english")  

# setup the vectordb  
db_file = tempfile.NamedTemporaryFile(prefix="milvus_", suffix=".db", delete=False).name  
print(f"The vector database will be saved to {db_file}")  
vector_db = Milvus(  
  embedding_function=embeddings_model,  
  connection_args={"uri": db_file},  
  auto_id=True,  
  index_params={"index_type": "AUTOINDEX"},  
  )  

# load example corpus file  
filename = 'state_of_the_union.txt'  
url = 'https://raw.github.com/IBM/watson-machine-learning-samples/master/cloud/data/foundation_models/state_of_the_union.txt'  

if not os.path.isfile(filename):  
  wget.download(url, out=filename)  

loader = TextLoader(filename)  
documents = loader.load()  
text_splitter = CharacterTextSplitter(chunk_size=500, chunk_overlap=0)  
texts = text_splitter.split_documents(documents)  

# add processed documents to the vectordb  
vector_db.add_documents(texts)  

# search the vectordb with the query  
query = "What did the president say about Ketanji Brown Jackson"  
docs = vector_db.similarity_search(query)  
print(docs[0].page_content)  

```

--------------------------------

### Convert PDF with Docling SDK using VlmPipeline and GraniteDocling

Source: https://www.ibm.com/granite/docs/models/docling

Use the Docling SDK to convert a PDF with the GraniteDocling VLM pipeline. The first part uses default values (transformers framework); the second part shows how to use the macOS MPS accelerator by setting vlm_options to vlm_model_specs.GRANITEDOCLING_MLX.

```python
from docling.datamodel import vlm_model_specs  
from docling.datamodel.base_models import InputFormat  
from docling.datamodel.pipeline_options import (  
    VlmPipelineOptions,  
)  
from docling.document_converter import DocumentConverter, PdfFormatOption  
from docling.pipeline.vlm_pipeline import VlmPipeline  

source = "https://arxiv.org/pdf/2501.17887"  

###### USING SIMPLE DEFAULT VALUES  
# - GraniteDocling model  
# - Using the transformers framework  

converter = DocumentConverter(  
    format_options={  
        InputFormat.PDF: PdfFormatOption(  
            pipeline_cls=VlmPipeline,  
        ),  
    }  
)  

doc = converter.convert(source=source).document  

print(doc.export_to_markdown())  


###### USING MACOS MPS ACCELERATOR  
# For more options see the compare_vlm_models.py example.  

pipeline_options = VlmPipelineOptions(  
    vlm_options=vlm_model_specs.GRANITEDOCLING_MLX,  
)  

converter = DocumentConverter(  
    format_options={  
        InputFormat.PDF: PdfFormatOption(  
            pipeline_cls=VlmPipeline,  
            pipeline_options=pipeline_options,  
        ),  
    }  
)  

doc = converter.convert(source=source).document  

print(doc.export_to_markdown())
```

--------------------------------

### Install dependencies for embedding model with LangChain and Milvus

Source: https://www.ibm.com/granite/docs/models/embedding

Install the required Python packages for using the embedding model with LangChain, Milvus, and document loaders. The command uses line continuations and includes the optional pymilvus extra for Milvus Lite.

```bash
pip install langchain_huggingface sentence_transformers \

    langchain_milvus 'pymilvus[milvus_lite]' \

    langchain_community \

    langchain_text_splitters \

    wget  

```

--------------------------------

### Expected output for tool calling

Source: https://www.ibm.com/granite/docs/models/granite

Expected output from the tool-calling example. Shows the model's reasoning and the tool_call XML-like structure with the function name and parameters.

```text
 thinking  

The user is asking for the weather in Boston right now. There's a function called  

get_current_weather that takes a city parameter. I need to call that with the city set to Boston.  

 response  

<tool_call>  

<function=get_current_weather>  

<parameter=city>  

Boston  

</parameter>  

</function>  

</tool_call>  

<|im_end|>  

```
Tokens:
Raw