### Install Dependencies

Source: https://github.com/google-health/medgemma/blob/main/notebooks/evaluate_on_medqa.ipynb

Installs the necessary libraries: transformers, vllm, and datasets. Use this to set up your environment for the evaluation.

```python
# Install dependencies
!pip install -q transformers==4.57.3 vllm==0.17.1 datasets
```

--------------------------------

### Install and Upgrade Libraries

Source: https://github.com/google-health/medgemma/blob/main/notebooks/cxr_longitudinal_comparison_with_hugging_face.ipynb

Installs or upgrades the 'accelerate' and 'transformers' libraries quietly. Ensure these libraries are up-to-date for optimal performance and compatibility.

```bash
! pip install --upgrade --quiet accelerate transformers
```

--------------------------------

### Install Dependencies

Source: https://github.com/google-health/medgemma/blob/main/notebooks/reinforcement_learning_with_hugging_face.ipynb

Installs necessary libraries for fine-tuning MedGemma models. Ensure you are using the specified versions for compatibility.

```bash
!pip install -q -U transformers==4.57.3 trl[vllm]==0.23.1 torch==2.8.0 datasets==4.6.1 tensorflow==2.20.0 fastai==2.8.7 gensim==4.4.0 wandb==0.25.0 huggingface_hub==0.36.2
# Tested with python 3.12 and !pip install transformers==4.57.3 trl[vllm]==0.23.1 torch==2.8.0 on an A100 40GB GPU.
```

--------------------------------

### Visualize Training Curves with TensorBoard

Source: https://github.com/google-health/medgemma/blob/main/notebooks/reinforcement_learning_with_hugging_face.ipynb

Installs TensorBoard and launches it to visualize training curves. Ensure the log directory is correctly specified.

```python
# Visualize training curves
! pip install -q tensorboard
%load_ext tensorboard
%tensorboard --logdir /content/tuned_medgemma4b/ --port 6007
```

--------------------------------

### Install Required Libraries

Source: https://github.com/google-health/medgemma/blob/main/notebooks/fine_tune_with_hugging_face.ipynb

Installs essential Python libraries for fine-tuning, including bitsandbytes, datasets, evaluate, peft, tensorboard, transformers, and trl.

```bash
! pip install --upgrade --quiet bitsandbytes datasets evaluate peft tensorboard transformers trl
```

--------------------------------

### Install Required Libraries

Source: https://github.com/google-health/medgemma/blob/main/notebooks/quick_start_with_hugging_face.ipynb

Installs or upgrades the accelerate, bitsandbytes, and transformers libraries, which are essential for running transformer-based models efficiently.

```bash
! pip install --upgrade --quiet accelerate bitsandbytes transformers
```

--------------------------------

### Print Example Model Answer

Source: https://github.com/google-health/medgemma/blob/main/notebooks/reinforcement_learning_with_hugging_face.ipynb

Prints the first model answer from the results DataFrame to inspect an example output.

```python
print(results_df['model_answer'].values[0])
```

--------------------------------

### Install pydicom Library

Source: https://github.com/google-health/medgemma/blob/main/notebooks/high_dimensional_ct_model_garden.ipynb

Installs the pydicom library, which is required for working with DICOM files.

```python
# @title Install pydicom Python library

%%capture
! pip install pydicom
```

--------------------------------

### PredictionApplication Setup for Gunicorn

Source: https://context7.com/google-health/medgemma/llms.txt

Configures and runs a `PredictionApplication` using Gunicorn for production deployment. Sets environment variables for port, routes, and predictor instance.

```python
import os
from serving.serving_framework import server_gunicorn, inline_prediction_executor
from serving.serving_framework.triton import (
    triton_server_model_runner,
    triton_streaming_server_model_runner,
    server_health_check,
)
from serving import predictor
import transformers, jsonschema, yaml

# Required environment variables
os.environ["AIP_HTTP_PORT"] = "8080"
os.environ["AIP_HEALTH_ROUTE"] = "/health"
os.environ["AIP_PREDICT_ROUTE"] = "/predict"
os.environ["MODEL_REST_PORT"] = "8501"

processor = transformers.AutoProcessor.from_pretrained("google/medgemma-4b-it")
to_prompt = lambda conv, p: processor.apply_chat_template(conv, tokenize=False, **p)

with open("python/serving/vertex_schemata/request.yaml") as f:
    validator = jsonschema.Draft202012Validator(yaml.safe_load(f))

predictor_instance = predictor.MedGemmaPredictor(
    prompt_converter=to_prompt,
    instance_validator=validator,
)
executor = inline_prediction_executor.InlinePredictionExecutor(
    predictor_instance.predict,
    triton_streaming_server_model_runner.TritonStreamingServerModelRunner,
)
health_checker = server_health_check.TritonServerHealthCheck(port=8501)

app = server_gunicorn.PredictionApplication(
    executor,
    health_check=health_checker,
    options={"bind": "0.0.0.0:8080", "workers": 3, "timeout": 120},
    instance_input=False,
    additional_routes={"/v1/chat/completions": executor},
)
app.run()
# Server now handles:
#   GET  /health              -> "ok" / 503
#   POST /predict             -> Vertex AI batch predict format
#   POST /v1/chat/completions -> OpenAI-compatible format
```

--------------------------------

### Install ez-wsi-dicomweb Python library

Source: https://github.com/google-health/medgemma/blob/main/notebooks/high_dimensional_pathology_model_garden.ipynb

Installs the ez-wsi-dicomweb library version 6.1.5. This is required for accessing and processing DICOM imaging data.

```python
# @title Install [ez-wsi-dicomweb](https://colab.sandbox.google.com/github/GoogleCloudPlatform/EZ-WSI-DICOMweb/blob/main/ez_wsi_demo.ipynb) Python library

%%capture
! pip install ez-wsi-dicomweb==6.1.5
```

--------------------------------

### Prompt Med-Gemma with Image URL

Source: https://github.com/google-health/medgemma/blob/main/notebooks/quick_start_with_dicom.ipynb

This example demonstrates how to prompt Med-Gemma with an image directly from a HTTPS URL. The image format can be specified, and it will be converted to DICOM if needed.

```python
# @title Step 5.3 - Prompt MedGemma with image URL

prompt = "Describe this X-ray."

file_type = 'JPEG' # @param ["JPEG", "PNG", "DICOM"]
file_type = file_type.lower()
if file_type == "dicom":
  file_type = "dcm"
# Image attribution: Stillwaterising, CC0, via Wikimedia Commons
url = f"https://storage.googleapis.com/hai-cd3-foundations-public-vault-entry/med_gemma/colab_example/cxr/chest_xray_pa_3-8-2010.{file_type}"

# Generate chat completion formatted prompt.
content = []
content.append({"type": "text", "text": prompt})
content.append({
                "type": "image_url",
                "image_url": {"url": url}
               })

messages = [
    get_system_instruction(endpoint_name, is_thinking, "You are an expert radiologist."),
    {
        "role": "user",
        "content": content
    }
]

instance = {
    "@requestFormat": "chatCompletions",
    "messages": messages,
    "max_tokens": get_max_tokens(endpoint_name, is_thinking),
    "temperature": 0
}

display_med_gemma_prompt("Describe chest x-ray read from a HTTPS server.", instance)
call_med_gemma_inference(endpoint_name, use_dedicated_endpoint, is_thinking, instance)
```

--------------------------------

### Format Test Data for Model Input

Source: https://github.com/google-health/medgemma/blob/main/notebooks/fine_tune_with_hugging_face.ipynb

Loads the test dataset, shuffles it, selects a subset of 1000 samples, and formats each example to include image and text prompts for the model.

```python
from typing import Any

from datasets import load_dataset


def format_test_data(example: dict[str, Any]) -> dict[str, Any]:
    example["messages"] = [
        {
            "role": "user",
            "content": [
                {
                    "type": "image",
                },
                {
                    "type": "text",
                    "text": PROMPT,
                },
            ],
        },
    ]
    return example


test_data = load_dataset("./CRC-VAL-HE-7K", split="train")
test_data = test_data.shuffle(seed=42).select(range(1000))
test_data = test_data.map(format_test_data)
```

--------------------------------

### Install Dependencies for EHR Navigator Agent

Source: https://github.com/google-health/medgemma/blob/main/notebooks/ehr_navigator_agent.ipynb

Installs necessary Python packages for the EHR Navigator Agent, including langchain, langchain-google-vertexai, and langgraph. Ensure your environment meets the specified version constraints.

```python
#@title Step 1 - Install dependencies

%pip install --upgrade --quiet langchain-google-vertexai langchain langchain-core google-api-python-client requests google-cloud-aiplatform langgraph "requests>=3.2.5,<3.0.0" "pyarrow>=14.0.0,<20.0.0a0"
```

--------------------------------

### Visualize EHR Navigator Agent Workflow

Source: https://github.com/google-health/medgemma/blob/main/notebooks/ehr_navigator_agent.ipynb

Installs graphviz and generates a visual diagram of the EHR Navigator Agent's workflow, illustrating the sequence of operations and LLM interactions.

```python
# @title Agentic Workflow
# @markdown The agent's workflow is as follows:

# @markdown 1. **Discover**: Uses the `get_patient_data_manifest` tool to identify all available FHIR resource types and associated codes for the specified patient. This provides an overview of the data landscape.
# @markdown 2. **Identify Relevant Resource Types**: Based on the manifest and the user's question, the LLM (MedGemma) identifies the FHIR resource types most likely to contain the necessary information.
# @markdown 3. **Select Data to Retrieve**: Based on the relevant resource types and the manifest, the LLM (MedGemma) plans the specific `get_patient_fhir_resource` tool calls needed. This planning happens once at the beginning for the selected resource types.
# @markdown 4. **Execute & Collect Facts Sequentially**:
# @markdown     * Executes each planned tool call to retrieve a FHIR resource.
# @markdown     * Prompts the LLM (MedGemma) to act as a "fact filter agent", extracting **relevant facts** based *only* on the result of that specific call and the original user question for context. The reason for this apporach is that FHIR resources across an entire patient record may not fit in the LLM context window.
# @markdown     * All these individually collected sets of facts are gathered into a list.
# @markdown 5. **Final Answer**: After all planned tool calls are executed and relevant facts are collected from each, the agent generates the final, comprehensive answer by synthesizing information from the **complete list of collected facts**.

!apt -qqq install graphviz
import graphviz

graphviz_diagram = """digraph G {
  rankdir=LR; // Left to Right
  node [shape=box, style="rounded,filled"];

  // Define nodes
  start [label="User Input", fillcolor="lightblue"];
  manifest_tool [label="call_manifest_tool\n(Discover)", fillcolor="lightblue"];
  identify_relevant [label="Identify Relevant\nResource Types", fillcolor="lightgreen"];
  plan [label="Select Data to Retrieve\n(Based on Manifest)", fillcolor="lightgreen"];
  execute_tool [label="Execute Data Retrieval", fillcolor="lightgreen"];
  final_answer_node [label="final_answer\n(Synthesize Facts)", fillcolor="lightyellow"];
  end [label="End", fillcolor="lightgrey"];

  llm [label="MedGemma", shape=oval, fillcolor="white"];
  fhir [label="FHIR store", shape=oval, fillcolor="pink"];

  // Define main flow edges
  start -> manifest_tool;
  manifest_tool -> identify_relevant;
  identify_relevant -> plan;
  plan -> execute_tool;

  // Add a loop for the execute node
  plan -> plan [label="Loop over resources"];
  execute_tool -> execute_tool [label="Loop over resources"];

  execute_tool -> final_answer_node;
  final_answer_node -> end;

  // Add dotted edges to represent LLM interactions
  manifest_tool -> fhir [style=dotted];
  identify_relevant -> llm [style=dashed]; // LLM identifies relevant resource types
  plan -> llm [style=dashed]; // LLM selects data to retrieve
  execute_tool -> fhir [style=dotted];
  execute_tool -> llm [style=dashed]; // LLM summarizes facts
  final_answer_node -> llm [style=dashed]; // LLM synthesizes final answer
}"""

dot = graphviz.Source(graphviz_diagram)
display(dot)
```

--------------------------------

### Agent Tool Output Example

Source: https://github.com/google-health/medgemma/blob/main/notebooks/ehr_navigator_agent.ipynb

This snippet shows the detailed output from the agent's tools, including resource discovery and FHIR API requests made for a specific patient.

```text
Output:

Invoking agent with question: 'What specific medications were administered to the patient during their sepsis encounter?. Patient ID e4350e97-bb8c-70b7-9997-9e098cfacef8.'


Result:
<IPython.core.display.Markdown object>


Result:
<IPython.core.display.Markdown object>


Output:
...[Tool] Discovering all available Encounter resources for patient: e4350e97-bb8c-70b7-9997-9e098cfacef8
...[Tool] Making request to: /fhir/Encounter?patient=Patient/e4350e97-bb8c-70b7-9997-9e098cfacef8
...[Tool] Discovering all available Practitioner resources for patient: e4350e97-bb8c-70b7-9997-9e098cfacef8
...[Tool] Making request to: /fhir/Practitioner?patient=Patient/e4350e97-bb8c-70b7-9997-9e098cfacef8
...[Tool] Making request to: /fhir/Practitioner/?&_page_token=AXzQLg5Cho6STwtPN6K63Huoz_6q6oDhg3yo4KblgMJsZ6iqwaYiE-xcPG1a3P1qzdWiVGGqB4Oko6bVO3ROh5Lipwx8_4AhzDWUy22NkjhwBsGf_rWK61kII6MM7gZv4R_gGO1k4pzYB50XmJHT3meUfagGiP_P-Zytmee3zYpZvI7DzsG1vL721POWaSJr4YQn1MmEiaU0B7TLERycTMdZ4G4uVv9pYMzrkmGo-WQ0FaD4f6P0Di-0ZKSL40OeuDoW9DUGn1x7Dklq84MzjqmCoHw%3D
...[Tool] Making request to: /fhir/Practitioner/?&_page_token=AXzQLg7lSlDPqMhmMVg7GRM7Fm54Z8zBYmsa9kvJGX1MTbqk4GIuSadYvnZQW0bZ8PplMIcBBwcL-D5--n3eB92Nqu51m-O3TXPT2H9nvgDdpefJASdZ_NWHae4eWKjMt8lWilvABavbOQdkQeMAs6T3Q33wpOyhkFrefVz7tQhFfTetEJTV399W3-EVv9QfXzb98Y6i5tXUiDgVDPjSemilqj4BPrjj0oKrWs7OuKVqbsSeduYHqrOyVN0mBDjVg56j8cdPpcQxNfZEg-lz9HbPBr7EczAc
...[Tool] Discovering all available Condition resources for patient: e4350e97-bb8c-70b7-9997-9e098cfacef8
...[Tool] Making request to: /fhir/Condition?patient=Patient/e4350e97-bb8c-70b7-9997-9e098cfacef8
...[Tool] Discovering all available Observation resources for patient: e4350e97-bb8c-70b7-9997-9e098cfacef8
...[Tool] Making request to: /fhir/Observation?patient=Patient/e4350e97-bb8c-70b7-9997-9e098cfacef8
...[Tool] Making request to: /fhir/Observation/?patient=Patient%2Fe4350e97-bb8c-70b7-9997-9e098cfacef8&_page_token=AXzQLg7XndAHNWFKd5TV8_fKcYvxXaHh2JxRp1TX3o39zq8kw72Lo56sb54mjw0qATunGomzxfOfypWIUOk2LiWUsAxMqQeYMCdfIrMQvraRdUeFZneZNEsMeDgZOTdjbNKXg1Sm10g7H_EyZT3bZGgc-XlCfrc4cBbOO58Q2nrI9gyjI5rnx6pZKqyo6GfqnoSDrtiEfJurJH4dwtQlVJVwu6AQpTwa8PDGJNoptO3c7juPAeUXaI4rOKJ9c_i2D-irm2iZbL9NmNWh3wDHkhKB
...[Tool] Discovering all available AllergyIntolerance resources for patient: e4350e97-bb8c-70b7-9997-9e098cfacef8
...[Tool] Making request to: /fhir/AllergyIntolerance?patient=Patient/e4350e97-bb8c-70b7-9997-9e098cfacef8
...[Tool] Discovering all available FamilyMemberHistory resources for patient: e4350e97-bb8c-70b7-9997-9e098cfacef8
...[Tool] Making request to: /fhir/FamilyMemberHistory?patient=Patient/e4350e97-bb8c-70b7-9997-9e098cfacef8
...[Tool] Discovering all available MedicationRequest resources for patient: e4350e97-bb8c-70b7-9997-9e098cfacef8
...[Tool] Making request to: /fhir/MedicationRequest?patient=Patient/e4350e97-bb8c-70b7-9997-9e098cfacef8
...[Tool] Discovering all available MedicationStatement resources for patient: e4350e97-bb8c-70b7-9997-9e098cfacef8
...[Tool] Making request to: /fhir/MedicationStatement?patient=Patient/e4350e97-bb8c-70b7-9997-9e098cfacef8
...[Tool] Discovering all available MedicationAdministration resources for patient: e4350e97-bb8c-70b7-9997-9e098cfacef8
...[Tool] Making request to: /fhir/MedicationAdministration?patient=Patient/e4350e97-bb8c-70b7-9997-9e098cfacef8
...[Tool] Discovering all available DiagnosticReport resources for patient: e4350e97-bb8c-70b7-9997-9e098cfacef8
...[Tool] Making request to: /fhir/DiagnosticReport?patient=Patient/e4350e97-bb8c-70b7-9997-9e098cfacef8
...[Tool] Discovering all available Procedure resources for patient: e4350e97-bb8c-70b7-9997-9e098cfacef8
...[Tool] Making request to: /fhir/Procedure?patient=Patient/e4350e97-bb8c-70b7-9997-9e098cfacef8
...[Tool] Making request to: /fhir/Procedure/?patient=Patient%2Fe4350e97-bb8c-70b7-9997-9e098cfacef8&_page_token=AXzQLg4xi_OzlGOaVbNZFyeEoINZboef0dPpCXBid5zBhllFx6sSeIkgRHJlvYRvedzSfFk4xA62AHbyhVVVokhrhiWn7dUhAPx28hoPVunpJ64kjKCXLf-5CVWsxFpYYoNkC3BHF9mLqkjQa80seAR2MEoM_O9CvLfQY--mkNfbprS_N__EHaDaUvkGAXOVomwp-ZxO4sQebqmKwp1ViOR3kR5LVxpa8sYXMdr49Ip-V8ikPA7b0nLrEzVAwr0vdXPshxF4LZnaeUlswHAq7pk%3D
```

--------------------------------

### Hugging Face Authentication Setup

Source: https://github.com/google-health/medgemma/blob/main/notebooks/reinforcement_learning_with_hugging_face.ipynb

Sets up Hugging Face authentication, using Colab secrets if available or prompting for login otherwise. It also configures the Hugging Face home directory for Colab Enterprise.

```python
import os
import sys

if "google.colab" in sys.modules and not os.environ.get("VERTEX_PRODUCT"):
    # Use secret if running in Google Colab
    from google.colab import userdata
    os.environ["HF_TOKEN"] = userdata.get("HF_TOKEN")
else:
    # Store Hugging Face data under `/content` if running in Colab Enterprise
    if os.environ.get("VERTEX_PRODUCT") == "COLAB_ENTERPRISE":
        os.environ["HF_HOME"] = "/content/hf"
    # Authenticate with Hugging Face
    from huggingface_hub import get_token
    if get_token() is None:
        from huggingface_hub import notebook_login
        notebook_login()

```

--------------------------------

### Specify Image and Text Inputs for CXR Comparison

Source: https://github.com/google-health/medgemma/blob/main/notebooks/cxr_longitudinal_comparison_with_hugging_face.ipynb

Defines the prompt for comparing two CXR images and downloads the image files from provided URLs. This step requires the PIL library for image handling and os for path manipulation. The prompt guides the model to focus on longitudinal changes and student-relevant details.

```python
import os
from PIL import Image

prompt = f"""Provide a comparison of these two images and include details from
the image which students should take note of when reading longitudinal CXR
"""

# Image attribution: Stillwaterising, CC0, via Wikimedia Commons
image1_url = "https://storage.googleapis.com/hai-cd3-foundations-public-vault-entry/med_gemma/colab_example/cxr/longitudinal_cxr_before.png"  # @param {type: "string"}
# Image generated by Gemini and reviewed by in-house board certified radiologist
image2_url = "https://storage.googleapis.com/hai-cd3-foundations-public-vault-entry/med_gemma/colab_example/cxr/longitudinal_cxr_after.png"  # @param {type: "string"}
! wget -nc -q {image1_url} {image2_url}
image_filename1 = os.path.basename(image1_url)
image_filename2 = os.path.basename(image2_url)
image1 = Image.open(image_filename1)
image2 = Image.open(image_filename2)
```

--------------------------------

### Set up Google Cloud Environment

Source: https://github.com/google-health/medgemma/blob/main/notebooks/quick_start_with_model_garden.ipynb

Provides instructions for setting up the Google Cloud environment, including enabling the Vertex AI API and ensuring billing is enabled for the project.

```python
# @title Set up Google Cloud environment

# @markdown #### Prerequisites
# @markdown 1. To get started with Vertex AI, you will need an existing Google Cloud project with the Vertex AI API enabled. Refer to the guide to [set up a project and development environment](https://docs.cloud.google.com/vertex-ai/docs/start/cloud-environment).
# @markdown 2. Confirm that [billing is enabled](https://cloud.google.com/billing/docs/how-to/modify-project) for your project.

```

--------------------------------

### Initialize Quantization and Load Model

Source: https://github.com/google-health/medgemma/blob/main/notebooks/fine_tune_with_hugging_face.ipynb

Sets up the quantization configuration and loads the specified MedGemma model from the Hugging Face Hub. This is a prerequisite for QLoRA fine-tuning.

```python
import torch
from transformers import AutoProcessor, AutoModelForImageTextToText, BitsAndBytesConfig

model_id = "google/medgemma-4b-it"
```

--------------------------------

### Launch Fine-tuning

Source: https://github.com/google-health/medgemma/blob/main/notebooks/fine_tune_with_hugging_face.ipynb

Initiate the fine-tuning process by calling the train() method on the configured SFTTrainer instance. This step may take a significant amount of time to complete.

```python
trainer.train()
```

--------------------------------

### Set up Cloud Storage Bucket

Source: https://github.com/google-health/medgemma/blob/main/notebooks/quick_start_with_model_garden.ipynb

Configures a Cloud Storage bucket for batch inference inputs and outputs. It either creates a new bucket or uses an existing one specified by BUCKET_URI.

```python
# Cloud Storage bucket for storing batch inference artifacts.
# A unique bucket will be created for the purpose of this notebook. If you
# prefer using your own GCS bucket, change the value of BUCKET_URI above.
if BUCKET_URI is None or BUCKET_URI.strip() == "":
    now = datetime.datetime.now().strftime("%Y%m%d%H%M%S")
    BUCKET_URI = f"gs://{PROJECT_ID}-tmp-{now}-{str(uuid.uuid4())[:4]}"
    BUCKET_NAME = "/".join(BUCKET_URI.split("/")[:3])
    ! gcloud storage buckets create --location {REGION} {BUCKET_URI}
else:
    assert BUCKET_URI.startswith("gs://"), "BUCKET_URI must start with `gs://`."
    BUCKET_NAME = "/".join(BUCKET_URI.split("/")[:3])
    shell_output = ! gcloud storage buckets describe {BUCKET_NAME} | grep "location:" | sed "s/location://"
    bucket_region = shell_output[0].strip().lower()
    if bucket_region != REGION:
        raise ValueError(
            f"Bucket region {bucket_region} is different from notebook region {REGION}"
        )
print(f"Using this Cloud Storage Bucket: {BUCKET_URI}")
```

--------------------------------

### Set up EHR Agent and Imports

Source: https://github.com/google-health/medgemma/blob/main/notebooks/ehr_navigator_agent.ipynb

Imports necessary libraries for LangChain, LangGraph, and Google authentication. Authenticates the user for Google Cloud access.

```python
import os
import json
import requests
from typing import Optional, TypedDict, Annotated, List, Union
import operator

# LangChain & LangGraph Imports
from langchain_google_vertexai import VertexAIModelGarden
from langchain_core.messages import HumanMessage, SystemMessage, AIMessage, ToolMessage, ToolCall
from langchain_core.tools import render_text_description
from langchain_core.tools import tool
from langgraph.graph import StateGraph, END
from langgraph.prebuilt import ToolNode

# Google Auth Imports
from google.colab import auth
from google.auth import default as get_auth_default
from google.auth.transport import requests as google_auth_requests

print("Authenticating user...")
auth.authenticate_user()
print("Authentication successful.")
```

--------------------------------

### Helper to Get FHIR Resource

Source: https://github.com/google-health/medgemma/blob/main/notebooks/ehr_navigator_agent.ipynb

Internal helper function to make authenticated GET requests to the FHIR store. It handles pagination and can optionally clean technical metadata from the response.

```python
def _get_fhir_resource(resource_path: str) -> dict:
    """Helper function to make an authenticated GET request to the FHIR store and compact."""
    try:
        credentials, _ = get_auth_default()
        request = google_auth_requests.Request()
        credentials.refresh(request)
        headers = {"Authorization": f"Bearer {credentials.token}"}

        all_entries = []
        next_url = f"{FHIR_STORE_URL}/{resource_path}"

        while next_url:
            print(f"...[Tool] Making request to: {next_url[next_url.find('/fhir/'):]}")
            response = requests.get(next_url, headers=headers)
            response.raise_for_status()
            current_page = response.json()

            if "entry" in current_page:
                all_entries.extend(current_page["entry"])

            next_url = None # Reset next_url for each iteration
            for link in current_page.get("link", []):
                if link.get("relation") == "next":
                    next_url = link.get("url")
                    break

        # Reconstruct the bundle with all entries
        full_bundle = {"resourceType": "Bundle", "type": "searchset", "total": len(all_entries), "entry": all_entries}

        def clean(obj):
            # Remove .resource.meta (timestamps/versions) from all objects
            if isinstance(obj, list): return [clean(i) for i in obj]
            if isinstance(obj, dict): return {k: clean(v) for k, v in obj.items() if k != "meta"}
            return obj.split("/fhir/")[-1] if isinstance(obj, str) and "/fhir/" in obj else obj

        # [OPTIONAL] Strip technical metadata and shorten URLs
        for e in all_entries:
            e.pop("fullUrl", None);
            e.pop("search", None)
            if "resource" in e: e["resource"] = clean(e["resource"])

        return full_bundle
    except Exception as e:
        return {"error": f"An error occurred: {str(e)}"}
```

--------------------------------

### Initialize Vertex AI and Deploy MedGemma 1.5

Source: https://github.com/google-health/medgemma/blob/main/notebooks/high_dimensional_pathology_model_garden.ipynb

Sets up environment variables, enables the Compute Engine API, and initializes the Vertex AI API. It then creates a Vertex AI endpoint object for MedGemma 1.5 and validates the endpoint name.

```python
ENDPOINT_ID = ""  # @param {type: "string", placeholder:"e.g. 123456789"}
ENDPOINT_REGION = ""  # @param {type: "string", placeholder:"e.g. us-central1"}

os.environ["CLOUDSDK_CORE_PROJECT"] = Google_Cloud_Project
os.environ["GOOGLE_CLOUD_PROJECT"] = Google_Cloud_Project
os.environ["GOOGLE_CLOUD_REGION"] = ENDPOINT_REGION

# Enable the Compute Engine API, if not already.
print("Enabling Compute Engine API.")
! gcloud services enable compute.googleapis.com

# Initialize Vertex AI API.
print("Initializing Vertex AI API.")
aiplatform.init(project=os.environ["GOOGLE_CLOUD_PROJECT"],
                location=os.environ["GOOGLE_CLOUD_REGION"],
                api_transport="rest")

endpoint = aiplatform.Endpoint(
    endpoint_name=ENDPOINT_ID,
    project=Google_Cloud_Project,
    location=ENDPOINT_REGION,
)

# Use the endpoint name to check that you are using an appropriate model variant.
# These checks are based on the default endpoint name from the Model Garden
# deployment settings.
ENDPOINT_NAME = endpoint.display_name
# Check that "1.5" (formatted as "1_5") is in the endpoint name
if "1_5" not in ENDPOINT_NAME:
    raise ValueError(
        "The examples in this notebook are intended to be used with MedGemma "
        "1.5. Please deploy and use an endpoint with the MedGemma 1.5 model."
    )
```

--------------------------------

### Step 3 - Set Parameters & Initialize Vertex AI

Source: https://github.com/google-health/medgemma/blob/main/notebooks/quick_start_with_dicom.ipynb

Configures project-specific parameters like Google Cloud Project ID, Endpoint ID, and region, then initializes the Vertex AI API. It also enables the Compute Engine API if it's not already active.

```python
# @title Step 3 - Set the parameters & initialize
from google.cloud import aiplatform

# @markdown #### Prerequisites

# @markdown 1. Make sure that [billing is enabled](https://cloud.google.com/billing/docs/how-to/modify-project) for your project.

# @markdown 2. Make sure that either the Compute Engine API is enabled or that you have the [Service Usage Admin](https://cloud.google.com/iam/docs/understanding-roles#serviceusage.serviceUsageAdmin) (`roles/serviceusage.serviceUsageAdmin`) role to enable the API.

# @markdown This section enables the Compute Engine API (if not already enabled), and initializes the Vertex AI API.

Google_Cloud_Project = ""  # @param {type: "string", placeholder:"e.g. MyProject"}
ENDPOINT_ID = ""  # @param {type: "string", placeholder:"e.g. 123456789"}
ENDPOINT_REGION = ""  # @param {type: "string", placeholder:"e.g. us-central1"}

# @markdown Set `use_dedicated_endpoint` if you are using a [dedicated endpoint](https://cloud.google.com/vertex-ai/docs/predictions/choose-endpoint-type) (`True` by default for Model Garden deployments). Uncheck this option for all other endpoint types.
use_dedicated_endpoint = True  # @param {type: "boolean"}

# @markdown Set `is_thinking` to `True` to turn on thinking mode. **Note:** Thinking is supported for the 27B variants only.
is_thinking = False  # @param {type: "boolean"}

# Enable the Compute Engine API, if not already.
print("Enabling Compute Engine API.")
! gcloud services enable compute.googleapis.com --project {Google_Cloud_Project}

# Initialize Vertex AI API.
print("Initializing Vertex AI API.")
aiplatform.init(project=Google_Cloud_Project, location=ENDPOINT_REGION, api_transport="rest")

endpoint = aiplatform.Endpoint(
    endpoint_name=ENDPOINT_ID,
    project=Google_Cloud_Project,
    location=ENDPOINT_REGION,
)

```

--------------------------------

### Configure Model Arguments for Fine-tuning

Source: https://github.com/google-health/medgemma/blob/main/notebooks/fine_tune_with_hugging_face.ipynb

Sets up model arguments including attention implementation, data type, and device mapping. It also configures quantization settings for efficient loading.

```python
if torch.cuda.get_device_capability()[0] < 8:
    raise ValueError("GPU does not support bfloat16, please use a GPU that supports bfloat16.")

model_kwargs = dict(
    attn_implementation="eager",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

model_kwargs["quantization_config"] = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=model_kwargs["torch_dtype"],
    bnb_4bit_quant_storage=model_kwargs["torch_dtype"],
)

model = AutoModelForImageTextToText.from_pretrained(model_id, **model_kwargs)
processor = AutoProcessor.from_pretrained(model_id)

# Use right padding to avoid issues during training
processor.tokenizer.padding_side = "right"
```

--------------------------------

### Initialize and Train GRPO Trainer

Source: https://github.com/google-health/medgemma/blob/main/notebooks/reinforcement_learning_with_hugging_face.ipynb

Initializes the GRPOTrainer with the specified model, reward functions, training arguments, and dataset. This step prepares the model for reinforcement learning fine-tuning.

```python
trainer = GRPOTrainer(
    model=ckpt,
    reward_funcs=[correctness_reward_func],
    args=training_args,
    train_dataset=train_dataset,
)
```

--------------------------------

### Specify image and prompt for analysis

Source: https://github.com/google-health/medgemma/blob/main/notebooks/cxr_anatomy_localization_with_hugging_face.ipynb

Sets up the object to be localized and constructs a detailed prompt for the model. The prompt includes instructions for output format, bounding box normalization, and anatomical reference points. It also downloads the specified image.

```python
import os
from PIL import Image

object_name = "right clavicle"  # @param {type: "string"}

prompt = f"""Instructions:
The following user query will require outputting bounding boxes. The format of bounding boxes coordinates is [y0, x0, y1, x1] where (y0, x0) must be top-left corner and (y1, x1) the bottom-right corner. This implies that x0 < x1 and y0 < y1. Always normalize the x and y coordinates the range [0, 1000], meaning that a bounding box starting at 15% of the image width would be associated with an x coordinate of 150. You MUST output a single parseable json list of objects enclosed into ```json...``` brackets, for instance ```json[{{"box_2d": [800, 3, 840, 471], "label": "car"}}, {{"box_2d": [400, 22, 600, 73], "label": "dog"}}]``` is a valid output. Now answer to the user query.

Remember "left" refers to the patient's left side where the heart is and sometimes underneath an L in the upper right corner of the image.

Query:
Where is the {object_name}? Don't give a final answer without reasoning. Output the final answer in the format "Final Answer: X" where X is a JSON list of objects. The object needs a "box_2d" and "label" key. Answer:"""

# Image attribution: Stillwaterising, CC0, via Wikimedia Commons
image_url = "https://upload.wikimedia.org/wikipedia/commons/c/c8/Chest_Xray_PA_3-8-2010.png"  # @param {type: "string"}
! wget -nc -q {image_url}
image_filename = os.path.basename(image_url)
image = Image.open(image_filename)
```

--------------------------------

### Set up Google Cloud Environment for Vertex AI

Source: https://github.com/google-health/medgemma/blob/main/notebooks/high_dimensional_pathology_model_garden.ipynb

Configures the Google Cloud project and enables the Compute Engine API. It also initializes the Vertex AI API, preparing the environment for deploying and using MedGemma 1.5 models.

```python
import os
from google.cloud import aiplatform

Google_Cloud_Project = ""  # @param {type: "string", placeholder:"e.g. MyProject"}
```

--------------------------------

### Run Inference with Pipeline API (Image-Text)

Source: https://github.com/google-health/medgemma/blob/main/notebooks/quick_start_with_hugging_face.ipynb

Uses the `pipeline` API to get a response from the model with image and text input. Displays the user prompt, image, and model response.

```python
output = pipe(text=messages, max_new_tokens=max_new_tokens, do_sample=False)
response = output[0]["generated_text"][-1]["content"]

display(Markdown(f"---\n\n**[ User ]**\n\n{prompt}"))
display(IPImage(filename=image_filename, height=300))
if is_thinking:
    thought, response = response.split("<unused95>")
    thought = thought.replace("<unused94>thought\n", "")
    display(Markdown(f"---\n\n**[ MedGemma thinking trace ]**\n\n{thought}"))
display(Markdown(f"---\n\n**[ MedGemma ]**\n\n{response}\n\n---"))
```

--------------------------------

### Initiate Batch Prediction Job

Source: https://github.com/google-health/medgemma/blob/main/notebooks/quick_start_with_model_garden.ipynb

Starts a batch prediction job using the specified model and configuration. This includes setting up input/output locations, machine types, and service accounts.

```python
batch_predict_job_name = common_util.get_job_name_with_datetime(
    prefix=f"batch-predict-{MODEL_ID}"
)

text_batch_predict_job = models["model"].batch_predict(
    job_display_name=batch_predict_job_name,
    gcs_source=os.path.join(
        BUCKET_URI, batch_predict_prefix, f"input/{instances_filename}"
    ),
    gcs_destination_prefix=os.path.join(
        BUCKET_URI, batch_predict_prefix, "output"
    ),
    machine_type=machine_type,
    accelerator_type=accelerator_type,
    accelerator_count=accelerator_count,
    service_account=SERVICE_ACCOUNT,
)

text_batch_predict_job.wait()

print(text_batch_predict_job.display_name)
print(text_batch_predict_job.resource_name)
print(text_batch_predict_job.state)
```

--------------------------------

### Configure Training Parameters with SFTConfig

Source: https://github.com/google-health/medgemma/blob/main/notebooks/fine_tune_with_hugging_face.ipynb

Set up training parameters using SFTConfig for the SFTTrainer. This includes defining output directories, epochs, batch sizes, learning rate, optimization, logging, saving strategies, and precision settings.

```python
from trl import SFTConfig

num_train_epochs = 1  # @param {type: "number"}
learning_rate = 2e-4  # @param {type: "number"}

args = SFTConfig(
    output_dir="medgemma-4b-it-sft-lora-crc100k",            # Directory and Hub repository id to save the model to
    num_train_epochs=num_train_epochs,                       # Number of training epochs
    per_device_train_batch_size=4,                           # Batch size per device during training
    per_device_eval_batch_size=4,                            # Batch size per device during evaluation
    gradient_accumulation_steps=4,                           # Number of steps before performing a backward/update pass
    gradient_checkpointing=True,                             # Enable gradient checkpointing to reduce memory usage
    optim="adamw_torch_fused",                               # Use fused AdamW optimizer for better performance
    logging_steps=50,                                        # Number of steps between logs
    save_strategy="epoch",                                   # Save checkpoint every epoch
    eval_strategy="steps",                                   # Evaluate every `eval_steps`
    eval_steps=50,                                           # Number of steps between evaluations
    learning_rate=learning_rate,                             # Learning rate based on QLoRA paper
    bf16=True,                                               # Use bfloat16 precision
    max_grad_norm=0.3,                                       # Max gradient norm based on QLoRA paper
    warmup_ratio=0.03,                                       # Warmup ratio based on QLoRA paper
    lr_scheduler_type="linear",                              # Use linear learning rate scheduler
    push_to_hub=True,                                        # Push model to Hub
    report_to="tensorboard",                                 # Report metrics to tensorboard
    gradient_checkpointing_kwargs={"use_reentrant": False},  # Set gradient checkpointing to non-reentrant to avoid issues
    dataset_kwargs={"skip_prepare_dataset": True},           # Skip default dataset preparation to preprocess manually
    remove_unused_columns = False,                           # Columns are unused for training but needed for data collator
    label_names=["labels"],                                  # Input keys that correspond to the labels
)
```

--------------------------------

### DICOM Image Analysis with Chat Completion API

Source: https://context7.com/google-health/medgemma/llms.txt

Examples of using the Chat Completion API with DICOM image sources, including single instances, series URIs, and ROI extraction.

```python
DICOM_BASE = "https://healthcare.googleapis.com/v1/projects/my-proj/locations/us/datasets/radiology/dicomStores/store"

# Single DICOM instance (e.g., CXR)
payload_dicom_instance = {
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Report on this chest radiograph."},
                {
                    "type": "image_dicom",
                    "image_dicom": {
                        "dicom_source": f"{DICOM_BASE}/dicomWeb/studies/1.2.3/series/1.2.3.4/instances/1.2.3.4.5",
                        "access_credential": "application_default"
                    }
                }
            ]
        }
    ],
    "max_tokens": 400
}
```

```python
# DICOM Series URI — server automatically fetches and sequences all slices
payload_dicom_series = {
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this CT volume."},
                {
                    "type": "image_dicom",
                    "image_dicom": {
                        "dicom_source": f"{DICOM_BASE}/dicomWeb/studies/1.2.3/series/1.2.3.4",
                        "access_credential": "application_default"
                    }
                }
            ]
        }
    ],
    "max_tokens": 600
}
```

```python
# DICOM with patch coordinates (ROI extraction)
payload_dicom_patch = {
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Classify this tissue patch."},
                {
                    "type": "image_dicom",
                    "image_dicom": {
                        "dicom_source": f"{DICOM_BASE}/dicomWeb/studies/1.2.3/series/1.2.3.5/instances/1.2.3.5.1",
                        "access_credential": "application_default",
                        "patch_coordinates_list": [
                            {"x_origin": 512, "y_origin": 256, "width": 224, "height": 224}
                        ]
                    }
                }
            ]
        }
    ],
    "max_tokens": 200
}
```

--------------------------------

### Run Batch Inference on Pretrained Model

Source: https://github.com/google-health/medgemma/blob/main/notebooks/fine_tune_with_hugging_face.ipynb

Performs batch inference on the test dataset using the pretrained model pipeline. Processes the outputs to get predictions and computes baseline metrics.

```python
pt_outputs = pt_pipe(
    text=test_data["messages"],
    images=test_data["image"],
    max_new_tokens=40,
    batch_size=64,
    return_full_text=False,
)

pt_predictions = [postprocess(out) for out in pt_outputs]
```

```python
pt_metrics = compute_metrics(pt_predictions)
print(f"Baseline metrics: {pt_metrics}")
```

--------------------------------

### Generate Responses using OpenAI SDK

Source: https://github.com/google-health/medgemma/blob/main/notebooks/quick_start_with_model_garden.ipynb

Send chat completions requests to the endpoint using the OpenAI SDK. This method is not compatible with DICOM endpoints and requires specific authentication setup.

```python
# @markdown This section shows how to send [chat completions](https://platform.openai.com/docs/api-reference/chat) requests to the endpoint using the OpenAI SDK.

# @markdown Click "Show Code" to see more details.

if "dicom" in ENDPOINT_NAME:
    raise NotImplementedError(
        "MedGemma DICOM endpoints are currently not OpenAI-compatible. Use the "
        "Vertex AI SDK to run inference."
    )

display(Markdown(f"---\n\n**[ User ]**\n\n{prompt}"))
display(IPImage(filename=image_filename, height=300))

creds, project = google.auth.default()
auth_req = google.auth.transport.requests.Request()
creds.refresh(auth_req)

ENDPOINT_RESOURCE_NAME = endpoints["endpoint"].resource_name

if use_dedicated_endpoint:
    DEDICATED_ENDPOINT_DNS = endpoints["endpoint"].gca_resource.dedicated_endpoint_dns
    BASE_URL = f"https://{DEDICATED_ENDPOINT_DNS}/v1beta1/{ENDPOINT_RESOURCE_NAME}"
else:
    BASE_URL = f"https://{ENDPOINT_REGION}-aiplatform.googleapis.com/v1beta1/{ENDPOINT_RESOURCE_NAME}"

client = openai.OpenAI(base_url=BASE_URL, api_key=creds.token)

model_response = client.chat.completions.create(
    model="",
    messages=messages,
    max_completion_tokens=max_tokens,
    temperature=0,
)
response = model_response.choices[0].message.content

if is_thinking:
    thought, response = response.split("<unused95>")
    thought = thought.replace("<unused94>thought\n", "")
    display(Markdown(f"---\n\n**[ MedGemma thinking trace ]**\n\n{thought}"))
display(Markdown(f"---\n\n**[ MedGemma ]**\n\n{response}\n\n---"))
```