### Install dependencies with uv Source: https://github.com/dwzhu-pku/paperbanana/blob/main/skill/SKILL.md Install the project dependencies using uv. Run from the repository root. ```bash cd uv pip install -r requirements.txt ``` -------------------------------- ### Step 3: Install dependencies and create placeholder files Source: https://github.com/dwzhu-pku/paperbanana/blob/main/notebooks/PaperBanana_Colab_End_to_End.ipynb Installs the repository requirements plus pypdf and ipyfilechooser, and creates empty ref.json placeholder files for the diagram and plot tasks so the planner works even with retrieval_setting='none'. Prints a reminder to restart the runtime on a fresh session. ```python import os import sys import subprocess from pathlib import Path os.chdir(REPO_DIR) if str(REPO_DIR) not in sys.path: sys.path.insert(0, str(REPO_DIR)) print('Working directory:', Path.cwd()) print('Python executable:', sys.executable) subprocess.run([sys.executable, '-m', 'pip', 'install', '-q', '--upgrade', 'pip'], check=True) subprocess.run([ sys.executable, '-m', 'pip', 'install', '-q', '-r', str(REPO_DIR / 'requirements.txt'), 'pypdf', 'ipyfilechooser' ], check=True) # PaperBanana planner expects these ref.json files to exist even when retrieval_setting='none'. for task in ['diagram', 'plot']: ref_path = REPO_DIR / 'data' / 'PaperBananaBench' / task / 'ref.json' ref_path.parent.mkdir(parents=True, exist_ok=True) if not ref_path.exists(): ref_path.write_text('[]\n', encoding='utf-8') print('Dependencies installed and placeholder data files are ready.') print() print('If this is your FIRST run on a fresh runtime, restart now:') print(' Runtime -> Restart runtime, then re-run from Step 1.') print('If you already restarted once, ignore this message.') ``` -------------------------------- ### Usage example with PlannerAgent Source: https://github.com/dwzhu-pku/paperbanana/blob/main/_autodocs/api-reference/base-agent.md Shows how to instantiate a concrete agent (PlannerAgent) using BaseAgent's constructor and call its process method. The example creates an ExpConfig, passes it to the agent, and prints model_name and system_prompt. ```python from agents.planner_agent import PlannerAgent from utils.config import ExpConfig cfg = ExpConfig(dataset_name="PaperBananaBench", task_name="diagram", exp_mode="dev_full", work_dir=repo_root) agent = PlannerAgent(exp_config=cfg) # BaseAgent.__init__(**kwargs) print(agent.model_name) # from exp_config.main_model_name print(agent.system_prompt[:40]) # DIAGRAM_PLANNER_AGENT_SYSTEM_PROMPT data = {"content": "...", "visual_intent": "..."} result = await agent.process(data) # returns the same dict, extended ``` -------------------------------- ### Install required packages with uv Source: https://github.com/dwzhu-pku/paperbanana/blob/main/README.md Installs the project's Python dependencies from requirements.txt using uv. ```bash uv pip install -r requirements.txt ``` -------------------------------- ### Install Python 3.12 with uv Source: https://github.com/dwzhu-pku/paperbanana/blob/main/README.md Installs Python 3.12 using uv, which is required for the project. ```bash uv python install 3.12 ``` -------------------------------- ### Usage example for legacy_generation_options Source: https://github.com/dwzhu-pku/paperbanana/blob/main/_autodocs/api-reference/legacy-generation-options.md Shows how to import and use generation_additional_info, image_size_from_data, and normalize_legacy_input_content. Demonstrates expected outputs and assertions. ```python from utils.legacy_generation_options import ( generation_additional_info, image_size_from_data, normalize_legacy_input_content, ) info = generation_additional_info("21:9", figure_size="7-9cm") # {"rounded_ratio": "21:9", "figure_size": "7-9cm", "image_size": "2k"} data = {"additional_info": info} assert image_size_from_data(data) == "2k" content = normalize_legacy_input_content('[{"x": 1, "y": 2}]', task_name="plot") assert isinstance(content, list) # now structured ``` -------------------------------- ### Initialize PaperVizProcessor and run directory Source: https://github.com/dwzhu-pku/paperbanana/blob/main/notebooks/PaperBanana_Colab_End_to_End.ipynb Initializes the PaperVizProcessor with all required agents and sets up the run directory and variables for later steps. This is the setup block for the optional viewer settings step. ```python processor = PaperVizProcessor( exp_config=exp_config, vanilla_agent=VanillaAgent(exp_config=exp_config), planner_agent=PlannerAgent(exp_config=exp_config), visualizer_agent=VisualizerAgent(exp_config=exp_config), stylist_agent=StylistAgent(exp_config=exp_config), critic_agent=CriticAgent(exp_config=exp_config), retriever_agent=RetrieverAgent(exp_config=exp_config), polish_agent=PolishAgent(exp_config=exp_config), ) run_stamp = datetime.now().strftime('%Y%m%d_%H%M%S') run_dir = OUTPUT_DIR / f'run_{task_name}_{run_stamp}' run_dir.mkdir(parents=True, exist_ok=True) # Expose these for Step 10 result = None results = [] selected_result = None selected_final_image = None selected_final_key = None active_task_name = task_name refined_path = None ``` -------------------------------- ### PolishAgent constructor initializes model names Source: https://github.com/dwzhu-pku/paperbanana/blob/main/_autodocs/api-reference/polish-agent.md Initializes the agent by calling super().__init__ and sets image_gen_model_name and main_model_name from the experiment config. Example values are gemini-3-pro-image-preview and gemini-3-pro-preview. ```python def __init__(self, **kwargs): super().__init__(**kwargs) self.image_gen_model_name = self.exp_config.image_gen_model_name # e.g., gemini-3-pro-image-preview self.main_model_name = self.exp_config.main_model_name # e.g., gemini-3-pro-preview ``` -------------------------------- ### Usage example for convert_png_b64_to_jpg_b64 Source: https://github.com/dwzhu-pku/paperbanana/blob/main/_autodocs/api-reference/image-utils.md Example usage of convert_png_b64_to_jpg_b64. It imports the function, passes a base64 PNG string, and checks for None to handle conversion failure. ```python from utils.image_utils import convert_png_b64_to_jpg_b64 png_b64 = "" jpg_b64 = convert_png_b64_to_jpg_b64(png_b64) if jpg_b64 is None: print("conversion failed; skipping") else: # jpg_b64 is a JPEG for storage/display pass ``` -------------------------------- ### Use legacy UI results utilities (resolve_final_output, text_key_for_image_key, build_evolution_stages) Source: https://github.com/dwzhu-pku/paperbanana/blob/main/_autodocs/api-reference/legacy-ui-results.md Shows how to use the legacy UI results utilities: resolve_final_output returns an object with image_key and text_key, text_key_for_image_key maps an image key to its text key (e.g., "vanilla_plot_base64_jpg" -> "visual_intent"), and build_evolution_stages returns a list of stages each with "name" and "image_key". The example uses exp_mode="dev_full" and a result dict with task_name "diagram". ```python from utils.legacy_ui_results import ( build_evolution_stages, resolve_final_output, resolve_display_mode_output, text_key_for_image_key, ) result = { "task_name": "diagram", "eval_image_field": "target_diagram_stylist_desc0_base64_jpg", "target_diagram_stylist_desc0_base64_jpg": "", "target_diagram_stylist_desc0": "description text", "vanilla_diagram_base64_jpg": "", } sel = resolve_final_output(result, exp_mode="dev_full") print(sel.image_key) # target_diagram_stylist_desc0_base64_jpg print(sel.text_key) # target_diagram_stylist_desc0 print(text_key_for_image_key("vanilla_plot_base64_jpg", result)) # "visual_intent" stages = build_evolution_stages(result, exp_mode="dev_full") for stage in stages: print(stage["name"], "->", stage["image_key"]) ``` -------------------------------- ### Generate diagram from transformer description Source: https://github.com/dwzhu-pku/paperbanana/blob/main/skill/SKILL.md Example command that generates a diagram from a transformer architecture description and a figure caption. The output is saved as architecture.png. ```bash python skill/run.py \ --content "We propose a transformer-based encoder-decoder architecture. The encoder consists of 12 self-attention layers with residual connections. The decoder uses cross-attention to attend to encoder outputs and generates the target sequence autoregressively." \ --caption "Figure 1: Overview of the proposed transformer architecture" \ --task diagram \ --output architecture.png ``` -------------------------------- ### Run demo.py Streamlit app Source: https://github.com/dwzhu-pku/paperbanana/blob/main/_autodocs/api-reference/entry-points.md Run the Streamlit demo UI. The first command starts the app on port 8501; the commented alternatives show the provided shell script (which creates a virtual environment and data directories) and the Dockerfile CMD that binds to 0.0.0.0 on port 8080. ```bash streamlit run demo.py --server.port 8501 # or: bash scripts/run_demo.sh (creates .venv + data dirs, then runs Streamlit) # Dockerfile CMD: streamlit run demo.py --server.port 8080 --server.address 0.0.0.0 ``` -------------------------------- ### Create ExpConfig with explicit parameters Source: https://github.com/dwzhu-pku/paperbanana/blob/main/_autodocs/api-reference/exp-config.md Constructs an ExpConfig instance with explicit parameters for dataset, task, split, experiment mode, retrieval setting, and model names. The empty model names are resolved from configs/model_config.yaml. The resulting exp_name and result_dir are printed as examples. ```python from pathlib import Path from utils.config import ExpConfig cfg = ExpConfig( dataset_name="PaperBananaBench", task_name="diagram", split_name="test", exp_mode="dev_full", retrieval_setting="auto", max_critic_rounds=3, main_model_name="", # resolved from configs/model_config.yaml image_gen_model_name="", work_dir=Path("/path/to/paperbanana"), ) print(cfg.exp_name) # e.g. 0310_1430_autoret_dev_full_test print(cfg.result_dir) # /path/to/paperbanana/results/PaperBananaBench_diagram ``` -------------------------------- ### Async process method for two-step polish Source: https://github.com/dwzhu-pku/paperbanana/blob/main/_autodocs/api-reference/polish-agent.md Defines the main async method that performs the two-step polish workflow: generating suggestions and then applying them to produce a polished image. It resolves the ground-truth image path, loads the image and style guide, and on success stores the polished image as a JPG base64 string. ```python async def process(self, data: Dict[str, Any]) -> Dict[str, Any]: """ Two-step polish process: 1. Generate improvement suggestions 2. Apply suggestions to generate polished image """ ``` -------------------------------- ### Usage example for diagram task Source: https://github.com/dwzhu-pku/paperbanana/blob/main/_autodocs/api-reference/planner-agent.md Shows how to use PlannerAgent for a diagram task. The retriever populates top10_references before calling process. The output contains the detailed figure description under 'target_diagram_desc0'. ```python import asyncio from agents.planner_agent import PlannerAgent from agents.retriever_agent import RetrieverAgent from utils.config import ExpConfig async def main(): cfg = ExpConfig(dataset_name="PaperBananaBench", task_name="diagram", exp_mode="dev_full", retrieval_setting="auto", work_dir=repo_root) planner = PlannerAgent(exp_config=cfg) data = {"content": "We propose a two-stage framework ...", "visual_intent": "Figure 1: Overview of the proposed framework"} # Retriever populates top10_references first data = await RetrieverAgent(exp_config=cfg).process(data, retrieval_setting="auto") out = await planner.process(data) print(out["target_diagram_desc0"]) # detailed figure description ``` -------------------------------- ### Error handling for missing planner output Source: https://github.com/dwzhu-pku/paperbanana/blob/main/_autodocs/api-reference/stylist-agent.md Error handling example showing that the planner output (target_{task}_desc0) must exist before calling process. If missing, a KeyError is caught and printed. ```python # Error handling: planner output must exist first try: out = await stylist.process({"content": "x", "visual_intent": "y"}) except KeyError as e: print(f"Planner description missing: {e}") ``` -------------------------------- ### Usage example for StylistAgent Source: https://github.com/dwzhu-pku/paperbanana/blob/main/_autodocs/api-reference/stylist-agent.md Example usage of StylistAgent. It creates an ExpConfig, instantiates the agent, and processes a data dictionary containing content, visual_intent, and target_diagram_desc0. The output is printed. ```python import asyncio from agents.stylist_agent import StylistAgent from utils.config import ExpConfig async def main(): cfg = ExpConfig(dataset_name="PaperBananaBench", task_name="diagram", exp_mode="dev_full", work_dir=repo_root) stylist = StylistAgent(exp_config=cfg) data = { "content": "Our method ...", "visual_intent": "Figure 1: Overview ...", "target_diagram_desc0": "The diagram shows a pipeline with three stages ...", } out = await stylist.process(data) print(out["target_diagram_stylist_desc0"]) ``` -------------------------------- ### Step 1: Mount Google Drive and define project paths Source: https://github.com/dwzhu-pku/paperbanana/blob/main/notebooks/PaperBanana_Colab_End_to_End.ipynb Mounts Google Drive and defines the project paths used throughout the notebook. The mount point can be overridden with the PAPERBANANA_DRIVE_MOUNT_POINT environment variable. Creates the PaperBanana root, input, and output folders if they do not exist. ```python from pathlib import Path import os from google.colab import drive # Mount Drive (configurable via PAPERBANANA_DRIVE_MOUNT_POINT) DRIVE_MOUNT_POINT = Path(os.environ.get('PAPERBANANA_DRIVE_MOUNT_POINT', str(Path.home() / 'drive'))) drive.mount(str(DRIVE_MOUNT_POINT)) # All notebook artifacts are stored under this Drive folder DRIVE_ROOT = DRIVE_MOUNT_POINT / 'MyDrive' PB_DRIVE_ROOT = DRIVE_ROOT / 'PaperBanana' REPO_DIR = PB_DRIVE_ROOT / 'PaperBanana' INPUT_DIR = PB_DRIVE_ROOT / 'input' OUTPUT_DIR = PB_DRIVE_ROOT / 'outputs' PB_DRIVE_ROOT.mkdir(parents=True, exist_ok=True) INPUT_DIR.mkdir(parents=True, exist_ok=True) OUTPUT_DIR.mkdir(parents=True, exist_ok=True) print(f'Drive root: {DRIVE_ROOT}') print(f'Project root in Drive: {PB_DRIVE_ROOT}') print(f'Repo path: {REPO_DIR}') print(f'Input path: {INPUT_DIR}') print(f'Output path: {OUTPUT_DIR}') print('Upload your paper to the input folder before Step 5.') ``` -------------------------------- ### Generate suggestions from style guide Source: https://github.com/dwzhu-pku/paperbanana/blob/main/_autodocs/api-reference/polish-agent.md Generates improvement suggestions based on the ground-truth image and style guide. Returns the model's suggestion text or an empty string on failure. ```python async def _generate_suggestions(self, gt_image_b64: str, style_guide: str) -> str: """Step 1: Generate improvement suggestions based on style guide""" ``` -------------------------------- ### Initialize VisualizerAgent and process diagram request Source: https://github.com/dwzhu-pku/paperbanana/blob/main/_autodocs/api-reference/visualizer-agent.md Initializes a VisualizerAgent with an ExpConfig for the PaperBananaBench diagram task, then calls process() with a data dictionary containing content, visual intent, stylist description, and additional info. Checks the output for the key 'target_diagram_stylist_desc0_base64_jpg' to see if an image is available. ```python import asyncio from agents.visualizer_agent import VisualizerAgent from utils.config import ExpConfig async def main(): cfg = ExpConfig(dataset_name="PaperBananaBench", task_name="diagram", exp_mode="dev_full", work_dir=repo_root) viz = VisualizerAgent(exp_config=cfg) data = { "content": "...", "visual_intent": "...", "target_diagram_stylist_desc0": "A detailed description of the figure ...", "additional_info": {"rounded_ratio": "21:9", "figure_size": "7-9cm"}, } out = await viz.process(data) if "target_diagram_stylist_desc0_base64_jpg" in out: # image is available for display/download pass ``` -------------------------------- ### _load_manual_references Source: https://github.com/dwzhu-pku/paperbanana/blob/main/_autodocs/api-reference/retriever-agent.md Loads manual references from a JSON file. Returns a tuple of IDs and full examples. ```APIDOC ## `_load_manual_references` ### Description Returns a tuple of (list of IDs, list of full examples) from `data/PaperBananaBench/diagram/agent_selected_12.json` (first 10). Plot mode returns `([], [])`. ### Method _load_manual_references(cfg: dict) -> tuple ### Parameters - **cfg** (dict) - Required - Configuration dictionary. ### Returns - **tuple** - (list of IDs, list of full examples). ### Source `agents/retriever_agent.py:127-145`. ``` -------------------------------- ### Run PaperVizProcessor batch with agents Source: https://github.com/dwzhu-pku/paperbanana/blob/main/_autodocs/api-reference/paperviz-processor.md Shows the standard async workflow for PaperVizProcessor: build an ExpConfig with dataset_name="PaperBananaBench", task_name="diagram", and exp_mode="dev_full", then instantiate the processor with all required agents. Feed a data_list containing content, visual_intent, path_to_gt_image, additional_info, and max_critic_rounds, and iterate over process_queries_batch with max_concurrent=10. The result includes target_diagram_stylist_desc0_base64_jpg and overall_outcome (e.g. "Model" / "Human" / "Tie"). ```python import asyncio from utils.paperviz_processor import PaperVizProcessor from utils import config from agents.vanilla_agent import VanillaAgent from agents.planner_agent import PlannerAgent from agents.visualizer_agent import VisualizerAgent from agents.stylist_agent import StylistAgent from agents.critic_agent import CriticAgent from agents.retriever_agent import RetrieverAgent from agents.polish_agent import PolishAgent from pathlib import Path async def main(): exp_config = config.ExpConfig( dataset_name="PaperBananaBench", task_name="diagram", split_name="test", exp_mode="dev_full", retrieval_setting="auto", work_dir=Path("/path/to/paperbanana"), ) processor = PaperVizProcessor( exp_config=exp_config, vanilla_agent=VanillaAgent(exp_config=exp_config), planner_agent=PlannerAgent(exp_config=exp_config), visualizer_agent=VisualizerAgent(exp_config=exp_config), stylist_agent=StylistAgent(exp_config=exp_config), critic_agent=CriticAgent(exp_config=exp_config), retriever_agent=RetrieverAgent(exp_config=exp_config), polish_agent=PolishAgent(exp_config=exp_config), ) data_list = [{"content": "...", "visual_intent": "Figure 1: ...", "path_to_gt_image": "images/ref_1.png", "additional_info": {"rounded_ratio": "21:9"}, "max_critic_rounds": 3}] results = [] async for result in processor.process_queries_batch(data_list, max_concurrent=10): results.append(result) print(results[0]["target_diagram_stylist_desc0_base64_jpg"] is not None) print(results[0]["overall_outcome"]) # e.g. "Model" / "Human" / "Tie" ``` -------------------------------- ### Clone the PaperBanana repository Source: https://github.com/dwzhu-pku/paperbanana/blob/main/README.md Clones the PaperBanana repository and changes into its directory. Run this first to get the source code. ```bash git clone https://github.com/dwzhu-pku/PaperBanana.git cd PaperBanana ``` -------------------------------- ### Step 5: Choose input file (upload or Drive path) Source: https://github.com/dwzhu-pku/paperbanana/blob/main/notebooks/PaperBanana_Colab_End_to_End.ipynb Step 5 of the notebook lets you choose an input file source: upload from computer, use a Drive path, or browse Drive with a widget. It sets SELECTED_DRIVE_FILE and FALLBACK_SELECTED_PATH for later steps. The browse_drive branch requires ipyfilechooser and ipywidgets; if unavailable, it prints a fallback message. ```python #@title Step 5: Choose input file (upload or Drive path) from pathlib import Path from IPython.display import display, Markdown FILE_INPUT_METHOD = 'upload_from_computer' # @param ['upload_from_computer', 'use_drive_path', 'browse_drive'] FALLBACK_SELECTED_PATH = str(INPUT_DIR / 'paper.pdf') # @param {type:"string"} BROWSER_ROOT = str(INPUT_DIR) # @param {type:"string"} # Step 8 will read this variable. SELECTED_DRIVE_FILE = '' USE_WIDGET_SELECTION = True ALLOWED_PATTERNS = ['*.pdf', '*.txt', '*.md', '*.tex', '*.csv', '*.json', '*.tsv'] def _set_selected(path_text: str): global SELECTED_DRIVE_FILE path = Path(str(path_text or '').strip()).expanduser() if path.exists() and path.is_file(): SELECTED_DRIVE_FILE = str(path) print('Selected file:', SELECTED_DRIVE_FILE) return True return False if FILE_INPUT_METHOD == 'upload_from_computer': try: from google.colab import files print('Click "Choose Files" and upload your paper/data file from your computer.') uploaded = files.upload() if not uploaded: print('No file uploaded. You can re-run Step 5.') else: first_name = list(uploaded.keys())[0] target_path = INPUT_DIR / first_name target_path.write_bytes(uploaded[first_name]) SELECTED_DRIVE_FILE = str(target_path) FALLBACK_SELECTED_PATH = str(target_path) print('Uploaded to Drive:', target_path) display(Markdown('**Next:** run **Step 6** then **Step 8**.')) except Exception as e: print('Upload widget failed:', e) print('Use FILE_INPUT_METHOD=use_drive_path and set FALLBACK_SELECTED_PATH.') elif FILE_INPUT_METHOD == 'use_drive_path': if _set_selected(FALLBACK_SELECTED_PATH): print('Using FALLBACK_SELECTED_PATH.') display(Markdown('**Next:** run **Step 6** then **Step 8**.')) else: print('Path not found:', FALLBACK_SELECTED_PATH) print('Update FALLBACK_SELECTED_PATH and re-run Step 5.') elif FILE_INPUT_METHOD == 'browse_drive': try: import ipywidgets as widgets from ipyfilechooser import FileChooser root = Path(BROWSER_ROOT) if not root.exists(): root = INPUT_DIR chooser = FileChooser(str(root)) chooser.title = 'Select a file, then click Use selected file' chooser.show_hidden = False chooser.use_dir_icons = True chooser.filter_pattern = ALLOWED_PATTERNS # Keep legacy name for compatibility with Step 8. pdf_chooser = chooser use_btn = widgets.Button(description='Use selected file', button_style='success') out = widgets.Output() def on_use_clicked(_): chosen = str(getattr(chooser, 'selected', '') or '').strip() with out: out.clear_output() if not chosen: print('No file selected yet.') return if _set_selected(chosen): display(Markdown('**Next:** run **Step 6** then **Step 8**.')) else: print('Selected path is invalid:', chosen) use_btn.on_click(on_use_clicked) display(chooser, use_btn, out) except Exception as e: print('Browser widget unavailable:', e) print('Use FILE_INPUT_METHOD=use_drive_path and set FALLBACK_SELECTED_PATH.') else: raise ValueError(f'Unknown FILE_INPUT_METHOD: {FILE_INPUT_METHOD}') ``` -------------------------------- ### RetrieverAgent class definition Source: https://github.com/dwzhu-pku/paperbanana/blob/main/_autodocs/api-reference/retriever-agent.md Defines the RetrieverAgent class as a subclass of BaseAgent. The docstring states its purpose: to retrieve relevant reference examples. ```python class RetrieverAgent(BaseAgent): """Retriever Agent to retrieve relevant reference examples""" ``` -------------------------------- ### Run batch experiment CLI (main.py) Source: https://github.com/dwzhu-pku/paperbanana/blob/main/_autodocs/api-reference/entry-points.md Runs the batch experiment CLI with a development experiment mode, test split, diagram task, auto retrieval, and three critic rounds. The command builds an ExpConfig with the repo root as work_dir, constructs a PaperVizProcessor with all seven agents, and streams process_queries_batch with hard-coded concurrency 10, writing results every 10 records. ```bash python main.py --exp_mode dev_full --split_name test --task_name diagram \ --retrieval_setting auto --max_critic_rounds 3 ``` -------------------------------- ### Run skill/run.py CLI with content and caption Source: https://github.com/dwzhu-pku/paperbanana/blob/main/_autodocs/api-reference/entry-points.md Invokes the skill's CLI to generate a diagram from method text and a caption, producing `output.png` with 4 parallel candidates. Requires `--content` or `--content-file`; exits with code 1 if both are absent. ```bash python skill/run.py \ --content "METHOD_TEXT" --caption "FIGURE_CAPTION" \ --task diagram --output output.png --num-candidates 4 ``` -------------------------------- ### VisualizerAgent constructor Source: https://github.com/dwzhu-pku/paperbanana/blob/main/_autodocs/api-reference/visualizer-agent.md Initializes the VisualizerAgent by calling the parent BaseAgent constructor. Task-dependent setup is performed in the parent class based on the task configuration. ```python def __init__(self, **kwargs): super().__init__(**kwargs) ```