### Initialize and Run Benchmark Script

Source: https://github.com/bowang-lab/medrax/blob/main/experiments/benchmark_medrax.ipynb

Initializes the necessary tools by calling `get_tools()` and then starts the main execution flow of the benchmark script by calling `main()`.

```python
tools = get_tools()
main(tools)
```

--------------------------------

### MedRAX Project Setup and Logging

Source: https://github.com/bowang-lab/medrax/blob/main/experiments/benchmark_medrax.ipynb

Configures essential project paths, logging, and model parameters for the MedRAX application. It sets up directory structures for prompts, benchmarks, models, figures, and experiment logs, ensuring proper data handling and output recording.

```python
import os
import logging
from datetime import datetime

# Setup directory paths
ROOT = "set this directory to where MedRAX is, .e.g /home/MedRAX"
PROMPT_FILE = f"{ROOT}/medrax/docs/system_prompts.txt"
BENCHMARK_FILE = f"{ROOT}/benchmark/questions"
MODEL_DIR = "set this to where the tool models are, e.g /home/models"
FIGURES_DIR = f"{ROOT}/benchmark/figures"

model_name = "medrax"
temperature = 0.2
medrax_logs = f"{ROOT}/experiments/medrax_logs"
log_filename = f"{medrax_logs}/{model_name}_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json"
logging.basicConfig(filename=log_filename, level=logging.INFO, format="%(message)s", force=True)
device = "cuda"
```

--------------------------------

### Project Dependencies and Setup

Source: https://github.com/bowang-lab/medrax/blob/main/experiments/benchmark_medrax.ipynb

This snippet details the essential Python libraries and modules imported for the medrax project. It covers AI/LLM frameworks like LangChain and OpenAI, machine learning tools like PyTorch, data handling utilities, and environment configuration.

```python
import operator
import warnings
from typing import *
import traceback

import os
import torch
from dotenv import load_dotenv
from IPython.display import Image
from langgraph.checkpoint.memory import MemorySaver
from langgraph.graph import END, StateGraph
from langchain_core.messages import AnyMessage, HumanMessage, SystemMessage, ToolMessage
from langchain_openai import ChatOpenAI
from transformers import logging
import matplotlib.pyplot as plt
import numpy as np
import re

from medrax.agent import *
from medrax.tools import *
from medrax.utils import *

import json
import openai
import os
import glob
import time
import logging
from datetime import datetime
from tenacity import retry, wait_exponential, stop_after_attempt

warnings.filterwarnings("ignore")
_ = load_dotenv()
```

--------------------------------

### Create MedRAX Agent

Source: https://github.com/bowang-lab/medrax/blob/main/experiments/benchmark_medrax.ipynb

Initializes and configures the MedRAX agent. It loads system prompts, sets up a memory checkpointer, and instantiates the language model (ChatOpenAI) with specified parameters. The agent is then configured with tools, logging, and a system prompt.

```python
from langchain_openai import ChatOpenAI
from medrax.agent import Agent
from medrax.memory import MemorySaver
from langchain_core.messages import HumanMessage

def get_agent(tools):
    prompts = load_prompts_from_file(PROMPT_FILE)
    prompt = prompts["MEDICAL_ASSISTANT"]

    checkpointer = MemorySaver()
    model = ChatOpenAI(model="gpt-4o", temperature=temperature, top_p=0.95)
    agent = Agent(
        model,
        tools=tools,
        log_tools=True,
        log_dir="logs",
        system_prompt=prompt,
        checkpointer=checkpointer,
    )
    thread = {"configurable": {"thread_id": "1"}}
    return agent, thread
```

--------------------------------

### Initialize MedRAX Tools

Source: https://github.com/bowang-lab/medrax/blob/main/experiments/benchmark_medrax.ipynb

Creates and returns a list of specialized tools for medical image analysis, including report generation, classification, segmentation, phrase grounding, VQA, and multimodal LLM processing. These tools are configured with model directories and device settings.

```python
from medrax.tools import ChestXRayReportGeneratorTool, ChestXRayClassifierTool, ChestXRaySegmentationTool, XRayPhraseGroundingTool, XRayVQATool, LlavaMedTool

def get_tools():
    report_tool = ChestXRayReportGeneratorTool(cache_dir=MODEL_DIR, device=device)
    xray_classification_tool = ChestXRayClassifierTool(device=device)
    segmentation_tool = ChestXRaySegmentationTool(device=device)
    grounding_tool = XRayPhraseGroundingTool(
        cache_dir=MODEL_DIR, temp_dir="temp", device=device, load_in_8bit=True
    )
    xray_vqa_tool = XRayVQATool(cache_dir=MODEL_DIR, device=device)
    llava_med_tool = LlavaMedTool(cache_dir=MODEL_DIR, device=device, load_in_8bit=True)

    return [
        report_tool,
        xray_classification_tool,
        segmentation_tool,
        grounding_tool,
        xray_vqa_tool,
        llava_med_tool,
    ]
```

--------------------------------

### Clone MedRAX Repository

Source: https://github.com/bowang-lab/medrax/blob/main/README.md

Standard Git command to clone the MedRAX project repository from GitHub. This is the first step for local setup.

```Bash
git clone https://github.com/bowang-lab/MedRAX.git
cd MedRAX
```

--------------------------------

### Display Benchmark Progress and Results

Source: https://github.com/bowang-lab/medrax/blob/main/experiments/benchmark_medrax.ipynb

Prints detailed progress updates during benchmark execution, including case and question IDs, model answers, and correct answers. It also outputs a final summary of processed and skipped questions.

```python
                print(f"Skipped question: Case ID {case_id}, Question ID {question_id}")
                continue

            print(
                f"Progress: Case {cases_processed}/{total_cases}, Question {questions_processed}/{total_questions}"
            )
            print(f"Case ID: {case_id}")
            print(f"Question ID: {question_id}")
            print(f"Final Response: {final_response}")
            print(f"Model Answer: {model_answer}")
            print(f"Correct Answer: {question_data['answer']}")
            print(f"----------------------------------------------------------------\n")

    print(f"\nBenchmark Summary:")
    print(f"Total Cases Processed: {cases_processed}")
    print(f"Total Questions Processed: {questions_processed}")
    print(f"Total Questions Skipped: {skipped_questions}")
```

--------------------------------

### Load Benchmark Questions by Case ID

Source: https://github.com/bowang-lab/medrax/blob/main/experiments/benchmark_medrax.ipynb

Finds all JSON question files within a specified case directory in the benchmark data. It uses glob to search for files matching a pattern.

```python
def load_benchmark_questions(case_id):
    benchmark_dir = "../benchmark/questions"
    return glob.glob(f"{benchmark_dir}/{case_id}/{case_id}_*.json")
```

--------------------------------

### Create Multimodal Request for MedRAX

Source: https://github.com/bowang-lab/medrax/blob/main/experiments/benchmark_medrax.ipynb

Constructs a multimodal request by parsing question data, case details, and figures. It identifies and locates relevant subfigures, formats them into a prompt, and prepares image URLs for the agent, enabling context-aware analysis of medical cases.

```python
import json
import os

def create_multimodal_request(question_data, case_details, case_id, question_id, agent, thread):
    # Parse required figures
    try:
        # Try multiple ways of parsing figures
        if isinstance(question_data["figures"], str):
            try:
                required_figures = json.loads(question_data["figures"])
            except json.JSONDecodeError:
                required_figures = [question_data["figures"]]
        elif isinstance(question_data["figures"], list):
            required_figures = question_data["figures"]
        else:
            required_figures = [str(question_data["figures"])]
    except Exception as e:
        print(f"Error parsing figures: {e}")
        required_figures = []

    # Ensure each figure starts with "Figure "
    required_figures = [
        fig if fig.startswith("Figure ") else f"Figure {fig}" for fig in required_figures
    ]

    subfigures = []
    for figure in required_figures:
        # Handle both regular figures and those with letter suffixes
        base_figure_num = "".join(filter(str.isdigit, figure))
        figure_letter = "".join(filter(str.isalpha, figure.split()[-1])) or None

        # Find matching figures in case details
        matching_figures = [
            case_figure
            for case_figure in case_details.get("figures", [])
            if case_figure["number"] == f"Figure {base_figure_num}"
        ]

        if not matching_figures:
            print(f"No matching figure found for {figure} in case {case_id}")
            continue

        for case_figure in matching_figures:
            # If a specific letter is specified, filter subfigures
            if figure_letter:
                matching_subfigures = [
                    subfig
                    for subfig in case_figure.get("subfigures", [])
                    if subfig.get("number", "").lower().endswith(figure_letter.lower())
                    or subfig.get("label", "").lower() == figure_letter.lower()
                ]
                subfigures.extend(matching_subfigures)
            else:
                # If no letter specified, add all subfigures
                subfigures.extend(case_figure.get("subfigures", []))

    # Add images to content
    figure_prompt = ""
    image_urls = []

    for subfig in subfigures:
        if "number" in subfig:
            subfig_number = subfig["number"].lower().strip().replace(" ", "_") + ".jpg"
            subfig_path = os.path.join(FIGURES_DIR, case_id, subfig_number)
            figure_prompt += f"{subfig_number} located at {subfig_path}\n"

    # The function is incomplete as it only prepares figure_prompt and image_urls, 
    # but doesn't use them to construct the final message content for the agent.
    # Assuming the intention is to pass these to the agent.
    # For demonstration, we'll return the prepared parts.
    return figure_prompt, image_urls
```

--------------------------------

### Initialize Chest X-Ray Generator Tool (Manual Setup)

Source: https://github.com/bowang-lab/medrax/blob/main/README.md

Initializes the ChestXRayGeneratorTool, which requires manual setup of RoentGen weights. Weights must be placed in the specified model directory.

```python
ChestXRayGeneratorTool(
    model_path=f"{model_dir}/roentgen", 
    temp_dir=temp_dir, 
    device=device
)
```

--------------------------------

### Run MedRAX Agent Workflow

Source: https://github.com/bowang-lab/medrax/blob/main/experiments/benchmark_medrax.ipynb

Executes the MedRAX agent's workflow with a given prompt and optional image URLs. It streams messages through the agent, captures the final response, and retrieves the agent's state, facilitating interactive medical analysis.

```python
def run_medrax(agent, thread, prompt, image_urls=[]):
    messages = [
        HumanMessage(
            content=[
                {"type": "text", "text": prompt},
            ]
            + [{"type": "image_url", "image_url": {"url": image_url}} for image_url in image_urls]
        )
    ]

    final_response = None
    for event in agent.workflow.stream({"messages": messages}, thread):
        for v in event.values():
            final_response = v

    final_response = final_response["messages"][-1].content.strip()
    agent_state = agent.workflow.get_state(thread)

    return final_response, str(agent_state)
```

--------------------------------

### Install MedRAX Package

Source: https://github.com/bowang-lab/medrax/blob/main/README.md

Installs the MedRAX package in editable mode using pip. This allows for development and modification of the package directly from the cloned repository.

```Bash
pip install -e .
```

--------------------------------

### Main Benchmark Evaluation Loop

Source: https://github.com/bowang-lab/medrax/blob/main/experiments/benchmark_medrax.ipynb

Orchestrates the benchmark evaluation process. It loads metadata, iterates through cases, initializes agents, loads questions, and calls the request processing function. It includes logic to skip processed cases and count skipped questions.

```python
def main(tools):
    with open("../data/eurorad_metadata.json", "r") as file:
        data = json.load(file)

    total_cases, total_questions = count_total_questions()
    cases_processed = 0
    questions_processed = 0
    skipped_questions = 0

    print(f"Beginning benchmark evaluation for model {model_name} with temperature {temperature}\n")

    for case_id, case_details in data.items():
        if int(case_details["case_id"]) <= 17158:
            continue

        print(f"----------------------------------------------------------------")
        agent, thread = get_agent(tools)

        question_files = load_benchmark_questions(case_id)
        if not question_files:
            continue

        cases_processed += 1
        for question_file in question_files:
            with open(question_file, "r") as file:
                question_data = json.load(file)
                question_id = os.path.basename(question_file).split(".")[0]

            # agent, thread = get_agent(tools)
            questions_processed += 1
            final_response, model_answer = create_multimodal_request(
                question_data, case_details, case_id, question_id, agent, thread
            )

            # Handle cases where response is None
            if final_response is None:
                skipped_questions += 1
```

--------------------------------

### Process Multimodal Request and Log Results

Source: https://github.com/bowang-lab/medrax/blob/main/experiments/benchmark_medrax.ipynb

Handles the processing of multimodal requests, including prompt generation, calling a core agent function, logging successful responses, and capturing errors. It prepares structured log entries for analysis.

```python
def create_multimodal_request(question_data, case_details, case_id, question_id, agent, thread):
    subfigures = case_details.get("figures", [])
    image_urls = []
    for subfig in subfigures:
        if "url" in subfig:
            image_urls.append(subfig["url"])
        else:
            print(f"Subfigure missing URL: {subfig}")

    prompt = (
        f"Answer this question correctly using chain of thought reasoning and "
        "carefully evaluating choices. Solve using our own vision and reasoning and then"
        "use tools to complement your reasoning. Trust your own judgement over any tools.\n"
        f"{question_data['question']}\n{figure_prompt}"
    )

    try:
        start_time = time.time()

        final_response, agent_state = run_medrax(
            agent=agent, thread=thread, prompt=prompt, image_urls=image_urls
        )
        model_answer, agent_state = run_medrax(
            agent=agent,
            thread=thread,
            prompt="If you had to choose the best option, only respond with the letter of choice (only one of A, B, C, D, E, F)",
        )
        duration = time.time() - start_time

        log_entry = {
            "case_id": case_id,
            "question_id": question_id,
            "timestamp": datetime.now().isoformat(),
            "model": model_name,
            "temperature": temperature,
            "duration": round(duration, 2),
            "usage": "",
            "cost": 0,
            "raw_response": final_response,
            "model_answer": model_answer.strip(),
            "correct_answer": question_data["answer"][0],
            "input": {
                "messages": prompt,
                "question_data": {
                    "question": question_data["question"],
                    "explanation": question_data["explanation"],
                    "metadata": question_data.get("metadata", {},
                    "figures": question_data["figures"],
                },
                "image_urls": [subfig["url"] for subfig in subfigures if "url" in subfig],
                "image_captions": [subfig.get("caption", "") for subfig in subfigures],
            },
            "agent_state": agent_state,
        }
        logging.info(json.dumps(log_entry))
        return final_response, model_answer.strip()

    except Exception as e:
        log_entry = {
            "case_id": case_id,
            "question_id": question_id,
            "timestamp": datetime.now().isoformat(),
            "model": model_name,
            "temperature": temperature,
            "status": "error",
            "error": str(e),
            "cost": 0,
            "input": {
                "messages": prompt,
                "question_data": {
                    "question": question_data["question"],
                    "explanation": question_data["explanation"],
                    "metadata": question_data.get("metadata", {},
                    "figures": question_data["figures"],
                },
                "image_urls": [subfig["url"] for subfig in subfigures if "url" in subfig],
                "image_captions": [subfig.get("caption", "") for subfig in subfigures],
            },
        }
        logging.info(json.dumps(log_entry))
        print(f"Error processing case {case_id}, question {question_id}: {str(e)}")
        return "", ""

```

--------------------------------

### Count Total Benchmark Questions and Cases

Source: https://github.com/bowang-lab/medrax/blob/main/experiments/benchmark_medrax.ipynb

Calculates the total number of distinct cases and the total number of questions across all cases in the benchmark directory. It iterates through directories and counts JSON files.

```python
def count_total_questions():
    total_cases = len(glob.glob("../benchmark/questions/*"))
    total_questions = sum(
        len(glob.glob(f"../benchmark/questions/{case_id}/*.json"))
        for case_id in os.listdir("../benchmark/questions")
    )
    return total_cases, total_questions
```

--------------------------------

### Start MedRAX Gradio Interface

Source: https://github.com/bowang-lab/medrax/blob/main/README.md

Command to launch the Gradio-based user interface for the MedRAX project. This provides a web-based way to interact with the AI agent. An alternative command using `sudo` is provided for potential permission issues.

```Bash
python main.py
```

```Bash
sudo -E env "PATH=$PATH" python main.py
```

--------------------------------

### Run ChestAgentBench Benchmarks

Source: https://github.com/bowang-lab/medrax/blob/main/experiments/README.md

Execute Python scripts to benchmark different AI models (GPT-4o, Llama 3.2, CheXagent, LLaVA-Med) against the ChestAgentBench dataset. This includes specific setup for LLaVA-Med which requires cloning its repository.

```bash
cd experiments
python benchmark_gpt4o.py
```

```bash
cd experiments
python benchmark_llama.py
```

```bash
cd experiments
python benchmark_chexagent.py
```

```bash
# For LLaVA-Med, clone repo and copy script:
# git clone <LLaVA-Med-repo>
# cp benchmark_llavamed.py ~/LLaVA-Med/llava/serve
python -m llava.serve.benchmark_llavamed --model-name llava-med-v1.5-mistral-7b --controller http://localhost:10000
```

--------------------------------

### Configure Local LLM Environment Variables

Source: https://github.com/bowang-lab/medrax/blob/main/README.md

Sets environment variables to configure the agent to use a local LLM, such as those provided by Ollama or LM Studio. This example shows how to set the base URL and API key.

```bash
export OPENAI_BASE_URL="http://localhost:11434/v1"
export OPENAI_API_KEY="ollama"
```

--------------------------------

### CheXbench Dataset Preparation

Source: https://github.com/bowang-lab/medrax/blob/main/experiments/README.md

Instructions for preparing the CheXbench dataset, including downloading specific dataset files (SLAKE, Rad-ReStruct, Open-I) and fixing local paths in the configuration file using a provided script.

```bash
# Download SLAKE dataset
# Save to MedMAX/data/slake

# Download Rad-ReStruct images
# Save to MedMAX/data/rad-restruct/images

# Download Open-I images
# Save to MedMAX/data/openi/images

# Fix paths in chexbench.json
python MedMAX/data/fix_chexbench.py
```

--------------------------------

### Initialize MedRAX Agent with Tools

Source: https://github.com/bowang-lab/medrax/blob/main/README.md

Demonstrates how to initialize the MedRAX agent by specifying a list of desired tools. This allows for selective tool usage and efficient resource management.

```python
selected_tools = [
    "ImageVisualizerTool",
    "ChestXRayClassifierTool",
    "ChestXRaySegmentationTool",
    # Add or remove tools as needed
]

agent, tools_dict = initialize_agent(
    "medrax/docs/system_prompts.txt",
    tools_to_use=selected_tools,
    model_dir="/model-weights"
)
```

--------------------------------

### Initialize Visual QA Tool

Source: https://github.com/bowang-lab/medrax/blob/main/README.md

Initializes the XRayVQATool for Visual Question Answering on X-rays. It automatically downloads CheXagent weights and requires a specified cache directory and device.

```python
XRayVQATool(
    cache_dir=model_dir, 
    device=device
)
```

--------------------------------

### Initialize Image Visualizer Tool

Source: https://github.com/bowang-lab/medrax/blob/main/README.md

Initializes the ImageVisualizerTool, a utility tool that does not require additional model weights.

```python
ImageVisualizerTool()
```

--------------------------------

### Initialize LLaVA-Med Tool

Source: https://github.com/bowang-lab/medrax/blob/main/README.md

Initializes the LLaVA-Med Tool, which automatically downloads weights to the cache directory. It supports 8-bit and 4-bit quantization for memory efficiency and requires a specified device.

```python
LlavaMedTool(
    cache_dir=model_dir, 
    device=device, 
    load_in_8bit=True
)
```

--------------------------------

### Initialize Chest X-Ray Classification Tool

Source: https://github.com/bowang-lab/medrax/blob/main/README.md

Initializes the ChestXRayClassifierTool, which automatically downloads its model weights. Requires a specified device (e.g., 'cuda' or 'cpu').

```python
ChestXRayClassifierTool(device=device)
```

--------------------------------

### Initialize Chest X-Ray Report Generator Tool

Source: https://github.com/bowang-lab/medrax/blob/main/README.md

Initializes the ChestXRayReportGeneratorTool. This tool requires model weights to be downloaded to the specified cache directory and a device to run on.

```python
ChestXRayReportGeneratorTool(
    cache_dir=model_dir, 
    device=device
)
```

--------------------------------

### Initialize Chest X-Ray Segmentation Tool

Source: https://github.com/bowang-lab/medrax/blob/main/README.md

Initializes the ChestXRaySegmentationTool, which automatically downloads its model weights. Requires a specified device.

```python
ChestXRaySegmentationTool(device=device)
```

--------------------------------

### Initialize X-Ray Phrase Grounding Tool

Source: https://github.com/bowang-lab/medrax/blob/main/README.md

Initializes the XRayPhraseGroundingTool, downloading Maira-2 weights to the specified cache directory. Supports 8-bit and 4-bit quantization for reduced memory usage.

```python
XRayPhraseGroundingTool(
    cache_dir=model_dir, 
    temp_dir=temp_dir, 
    load_in_8bit=True, 
    device=device
)
```

--------------------------------

### Compare Benchmark Runs

Source: https://github.com/bowang-lab/medrax/blob/main/experiments/README.md

Utilize the `compare_runs.py` script to analyze and compare benchmark results from one or multiple model runs. The script can compare single files, two specific files, or all provided model log files.

```bash
# Compare a single file
python compare_runs.py results/medmax.json
```

```bash
# Compare two models
python compare_runs.py results/medmax.json results/gpt4o.json
```

```bash
# Compare all models
python compare_runs.py results/medmax.json results/gpt4o.json results/llama.json results/chexagent.json results/llavamed.json
```

--------------------------------

### Initialize Dicom Processor Tool

Source: https://github.com/bowang-lab/medrax/blob/main/README.md

Initializes the DicomProcessorTool, a utility tool that does not require additional model weights but uses a temporary directory for processing.

```python
DicomProcessorTool(temp_dir=temp_dir)
```

--------------------------------

### Download ChestAgentBench Dataset

Source: https://github.com/bowang-lab/medrax/blob/main/README.md

Command to download the ChestAgentBench dataset from Hugging Face. This dataset contains 2,500 complex medical queries across 7 categories for evaluating CXR interpretation models.

```Bash
huggingface-cli download wanglab/chestagentbench --repo-type dataset --local-dir chestagentbench
```

--------------------------------

### Evaluate Model with GPT-4o

Source: https://github.com/bowang-lab/medrax/blob/main/README.md

Script to run an evaluation using the GPT-4o model. It requires setting the OpenAI API key, specifying the model, temperature, and the number of cases to process. The `--use-urls` flag indicates processing data from URLs.

```Bash
export OPENAI_API_KEY="<your-openai-api-key>"
python quickstart.py \
    --model chatgpt-4o-latest \
    --temperature 0.2 \
    --max-cases 2 \
    --log-prefix chatgpt-4o-latest \
    --use-urls
```

--------------------------------

### Inspect and Analyze Benchmark Logs

Source: https://github.com/bowang-lab/medrax/blob/main/experiments/README.md

Scripts to inspect generated logs and analyze benchmark results. The log inspection script selects the most recent log file by default, while the analysis script requires a log file and model name.

```bash
python inspect_logs.py [optional: log-file] -n [num-logs]
```

```bash
python analyze_axes.py results/[logfile].json ../benchmark/questions/ --model [gpt4|llama|chexagent|llava-med] --max-questions [optional:int]
```

--------------------------------

### MedRAX Citation

Source: https://github.com/bowang-lab/medrax/blob/main/README.md

BibTeX entry for citing the MedRAX project in academic publications.

```bibtex
@misc{fallahpour2025medraxmedicalreasoningagent,
      title={MedRAX: Medical Reasoning Agent for Chest X-ray},
      author={Adibvafa Fallahpour and Jun Ma and Alif Munim and Hongwei Lyu and Bo Wang},
      year={2025},
      eprint={2502.02673},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2502.02673},
}
```

--------------------------------

### Unzip Dataset Figures

Source: https://github.com/bowang-lab/medrax/blob/main/README.md

Command to unzip the figures associated with the Eurorad dataset, likely used within the ChestAgentBench context.

```Bash
unzip chestagentbench/figures.zip
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.