### Install dependencies and initialize Langfuse client Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/integrations/langfuse.md Required setup: install dependencies, configure environment variables for Langfuse and OpenAI, and initialize the Langfuse client. The assertion verifies authentication, failing if credentials are invalid. ```bash pip install py-openjudge langfuse ``` ```bash export LANGFUSE_PUBLIC_KEY="pk-lf-your-public-key" export LANGFUSE_SECRET_KEY="sk-lf-your-secret-key" export LANGFUSE_HOST="https://cloud.langfuse.com" # Or your self-hosted URL # OpenAI API configuration (required for LLM-based graders) export OPENAI_API_KEY="sk-your-api-key" export OPENAI_BASE_URL="https://api.openai.com/v1" # Optional, defaults to OpenAI ``` ```python import os from langfuse import Langfuse # Initialize Langfuse client langfuse = Langfuse( public_key=os.getenv("LANGFUSE_PUBLIC_KEY"), secret_key=os.getenv("LANGFUSE_SECRET_KEY"), host=os.getenv("LANGFUSE_HOST", "https://cloud.langfuse.com") ) # Verify connection assert langfuse.auth_check(), "Langfuse authentication failed" ``` -------------------------------- ### Run agentic judge examples (01_claude_code_judge.py, 02_codex_judge.py, 03_cursor_agent_judge.py) Source: https://github.com/agentscope-ai/openjudge/blob/main/cookbooks/agentic_judge/README.md Run each example script from the repository root. Each requires its specific CLI to be installed and authenticated on the machine running the example, as listed in the Prerequisites table. ```bash python cookbooks/agentic_judge/01_claude_code_judge.py python cookbooks/agentic_judge/02_codex_judge.py python cookbooks/agentic_judge/03_cursor_agent_judge.py ``` -------------------------------- ### Set up Ray cluster and run pairwise training Source: https://github.com/agentscope-ai/openjudge/blob/main/cookbooks/training_judge_model/grpo/README.md Commands to start a Ray cluster and execute the pairwise training script. Run these in a terminal after installing dependencies. ```bash # 1. Set up Ray cluster (if not already running) ray start --head --port=6379 --dashboard-port=8265 # 2. Run training cd cookbooks/training_judge_model/grpo bash pairwise/run_pairwise.sh ``` -------------------------------- ### Set up Ray cluster and run pointwise training Source: https://github.com/agentscope-ai/openjudge/blob/main/cookbooks/training_judge_model/grpo/README.md Commands to start a Ray cluster and execute the pointwise training script. Run these in a terminal after installing dependencies. ```bash # 1. Set up Ray cluster (if not already running) ray start --head --port=6379 --dashboard-port=8265 # 2. Run training cd cookbooks/training_judge_model/grpo bash pointwise/run_pointwise.sh ``` -------------------------------- ### Start Ray head node for multi-node setup Source: https://github.com/agentscope-ai/openjudge/blob/main/cookbooks/training_judge_model/grpo/README.md Starts the head node of a multi-node Ray cluster with 8 GPUs, using port 6379 and dashboard port 8265. ```bash ray start --head \ --port=6379 \ --dashboard-port=8265 \ --num-gpus=8 ``` -------------------------------- ### Install OpenJudge from Source Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/get_started/quickstart.md Clone the repository and install with standard, development, or VerL dependencies for training scenarios. ```bash git clone https://github.com/agentscope-ai/OpenJudge.git cd OpenJudge # Install based on your needs: pip install -e . # Standard installation pip install -e .[dev] # With development dependencies pip install -e .[verl] # With VerL option for training scenarios ``` -------------------------------- ### Set up OpenJudge development environment Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/community/contributing.md Commands for cloning the repository, installing dependencies, and verifying the installation with the RelevanceGrader import in a development environment. ```bash git clone https://github.com/YOUR_USERNAME/OpenJudge.git cd OpenJudge ``` ```bash pip install -e ".[dev]" ``` ```bash python -c "from openjudge.graders.common import RelevanceGrader; print('✓ Installation successful')" ``` -------------------------------- ### Install dependencies Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/integrations/agentscope.md Install the required Python packages for the integration. ```bash pip install py-openjudge agentscope ``` -------------------------------- ### Install OpenJudge Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/integrations/verl.md Installs the OpenJudge package and its dependencies. Run this from the OpenJudge directory. ```bash # Install dependencies cd OpenJudge pip install openjudge ``` -------------------------------- ### Verify installation and run tests Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/community/contributing.md Commands to verify the installation and optionally run the test suite with verbose output before submitting changes. ```bash # Verify installation python -c "from openjudge.graders.common import RelevanceGrader; print('✓ Works')" # Run tests (optional) pytest tests/ -v ``` -------------------------------- ### Quick Start: Evaluate Correctness with GradingRunner Source: https://github.com/agentscope-ai/openjudge/blob/main/skills/openjudge-core/01-graders-and-pipeline/SKILL.md Complete example of configuring an OpenAIChatModel, creating a CorrectnessGrader, preparing a dataset, and running evaluation with GradingRunner. Expected output shows scores and reasons for each response. ```python import asyncio from openjudge.models.openai_chat_model import OpenAIChatModel from openjudge.graders.common.correctness import CorrectnessGrader from openjudge.runner.grading_runner import GradingRunner # 1. Configure the judge model (OpenAI-compatible endpoint) model = OpenAIChatModel( model="qwen-plus", api_key="sk-xxx", base_url="https://dashscope.aliyuncs.com/compatible-mode/v1", ) # 2. Instantiate a grader grader = CorrectnessGrader(model=model) # 3. Prepare dataset dataset = [ { "query": "What is the capital of France?", "response": "Paris is the capital of France.", "reference_response": "Paris.", }, { "query": "What is 2 + 2?", "response": "The answer is five.", "reference_response": "4.", }, ] # 4. Run evaluation async def main(): runner = GradingRunner( grader_configs={"correctness": grader}, max_concurrency=8, ) results = await runner.arun(dataset) for i, result in enumerate(results["correctness"]): print(f"[{i}] score={result.score} reason={result.reason}") asyncio.run(main()) ``` ```text [0] score=5 reason=The response accurately states Paris as capital... [1] score=1 reason=The response gives the wrong answer (five vs 4)... ``` -------------------------------- ### Strategy B: Minimum Dependencies - Subtask Table and Install Command Source: https://github.com/agentscope-ai/openjudge/blob/main/skills/find-skills-combo/SKILL.md Strategy B covers all subtasks with the fewest skills, possibly downgrading coverage from High to Medium. The table includes a `vs Strategy A` column to show trade-offs, and the install command installs only the minimal skill set. ```markdown | Subtask | Handled By | Coverage | vs Strategy A | |---------|-----------|----------|---------------| | S1 | skill-name-a | High | Same | | S2 | skill-name-a | Medium | ↓ High → Medium | | S3 | skill-name-a | Medium | ↓ High → Medium | ``` ```bash npx skills add owner/repo@skill-a -g -y ``` -------------------------------- ### Install dependencies Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/integrations/langsmith.md Install the required packages: py-openjudge, langsmith, and python-dotenv. ```bash pip install py-openjudge langsmith python-dotenv ``` -------------------------------- ### Prepare Dataset: Download from HuggingFace or Use Bundled Example Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/validating_graders/ref_hallucination_arena.md Two options for obtaining the evaluation dataset: use the bundled example queries file or download the official dataset from HuggingFace using the huggingface_hub library. ```bash # Option 1: Use the bundled example queries ls cookbooks/ref_hallucination_arena/examples/queries_example.json # Option 2: Download from HuggingFace pip install huggingface_hub python -c " from huggingface_hub import hf_hub_download hf_hub_download( repo_id='OpenJudge/ref-hallucination-arena', filename='ref_hallucination_query.json', repo_type='dataset', local_dir='./data' ) " ``` -------------------------------- ### Install mmx-cli via npm Source: https://github.com/agentscope-ai/openjudge/blob/main/skills/mmx-cli/SKILL.md Installs the @minimax-ai/cli package globally using npm. ```bash npm install -g @minimax-ai/cli ``` -------------------------------- ### Install OpenJudge from PyPI Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/get_started/quickstart.md Install the package with standard dependencies from PyPI, or with dev dependencies for development purposes. ```bash # Install with Standard dependencies from PyPI pip install py-openjudge # For development purposes, install with dev dependencies pip install -e .[dev] ``` -------------------------------- ### Manual installation of verl and OpenJudge Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/building_graders/training_judge_models.md Install verl and its dependencies manually, then install OpenJudge. Note that simply running 'pip install verl' is not sufficient; specific versions of vLLM, SGLang, FlashAttention, and other dependencies are required. ```bash # 1. Create conda environment conda create -n openjudge_train python==3.12 conda activate openjudge_train # 2. Install dependencies (vLLM, SGLang, etc.) git clone https://github.com/volcengine/verl && cd verl USE_MEGATRON=0 bash scripts/install_vllm_sglang_mcore.sh # 3. Install verl pip install --no-deps -e . # 4. Install OpenJudge (verl dependencies already installed in step 2) cd /path/to/OpenJudge pip install -e . ``` -------------------------------- ### Install dependencies for arena skills Source: https://github.com/agentscope-ai/openjudge/blob/main/skills/arena-eval/README.md Install the required Python packages for both arena skills. The first command installs the OpenJudge package, and the second installs matplotlib for chart generation, which is needed by both skills. ```bash pip install py-openjudge pip install matplotlib # chart generation, needed by both skills ``` -------------------------------- ### ToolSelectionGrader usage example Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/built_in_graders/agent_graders.md Shows how to instantiate ToolSelectionGrader with an OpenAIChatModel and evaluate tool selection quality using aevaluate. The example uses a query, tool definitions, and tool calls, then prints the score and reason. ```python import asyncio from openjudge.models import OpenAIChatModel from openjudge.graders.agent import ToolSelectionGrader async def main(): model = OpenAIChatModel(model="qwen3-32b") grader = ToolSelectionGrader(model=model) result = await grader.aevaluate( query="Find all Python files modified in the last week", tool_definitions=[ {"name": "search_files", "description": "Search for files by pattern"}, {"name": "git_log", "description": "Get git commit history"}, {"name": "read_file", "description": "Read file contents"} ], tool_calls=[ {"name": "search_files", "arguments": {"pattern": "*.py"}}, {"name": "git_log", "arguments": {"days": 7}} ] ) print(f"Score: {result.score}") # 4-5 - good tool selection print(f"Reason: {result.reason}") asyncio.run(main()) ``` ```text Score: 5.0 Reason: The selected tools are highly relevant and directly address the user's query. The 'search_files' tool with the pattern '*.py' effectively identifies all Python files in the system, while the 'git_log' tool with the argument 'days: 7' retrieves the commit history for the last week, which can be used to determine which of those Python files were modified recently. Together, these tools provide a complete and efficient solution without including any unnecessary or redundant tools. The selection demonstrates a clear understanding of both the task intent and the capabilities of the available tools. ``` -------------------------------- ### Start Ray head node and worker nodes for multi-node cluster Source: https://github.com/agentscope-ai/openjudge/blob/main/cookbooks/training_judge_model/grpo/README.md Start a Ray head node on the head node, then start worker nodes with the head node's IP address. Ensure RAY_ADDRESS is set to http://:8265 before training. ```bash # On head node ray start --head --port=6379 --dashboard-port=8265 # On worker nodes ray start --address=":6379" ``` -------------------------------- ### Install prerequisites with pip Source: https://github.com/agentscope-ai/openjudge/blob/main/skills/academic-eval/02-bib-verify/SKILL.md Install the required Python packages to run the BibTeX verification workflow. ```bash pip install py-openjudge litellm ``` -------------------------------- ### Install verl and run SFT training Source: https://github.com/agentscope-ai/openjudge/blob/main/cookbooks/training_judge_model/sft/README.md Install the verl library and run the SFT training script. The training script is located in the cookbooks/training_judge_model/sft directory. ```bash pip install verl==0.6.1 ``` ```bash cd cookbooks/training_judge_model/sft bash run_sft_rm.sh ``` -------------------------------- ### Install and Run Bradley-Terry Training Source: https://github.com/agentscope-ai/openjudge/blob/main/cookbooks/training_judge_model/bradley-terry/README.md Install the verl package and run the Bradley-Terry training script from the cookbook directory. ```bash # 1. Install dependencies pip install verl==0.6.1 # 2. Run training cd cookbooks/training_judge_model/bradley-terry bash run_bt_rm.sh ``` -------------------------------- ### Verify openjudge installation Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/community/style-guide.md Verify the installation by importing openjudge and printing its version. Expected output is the version number. ```python import openjudge print(openjudge.__version__) ``` -------------------------------- ### Install OpenJudge and matplotlib for ref_hallucination_arena Source: https://github.com/agentscope-ai/openjudge/blob/main/skills/academic-eval/03-ref-hallucination-arena/SKILL.md Install the OpenJudge package and the extra dependency for chart generation in the ref_hallucination_arena skill. ```bash # Install OpenJudge pip install py-openjudge # Extra dependency for ref_hallucination_arena (chart generation) pip install matplotlib ``` -------------------------------- ### Install mmx-cli via npx Source: https://github.com/agentscope-ai/openjudge/blob/main/skills/mmx-cli/SKILL.md Installs the mmx-cli skill globally using npx. Requires Node.js and npm. ```bash npx skills add MiniMax-AI/cli@skill -g -y ``` -------------------------------- ### Full example: generate grader and evaluate Source: https://github.com/agentscope-ai/openjudge/blob/main/skills/openjudge-core/01-graders-and-pipeline/generator.md Complete usage example. Creates a generator with pointwise config, generates a grader from training data, then evaluates a new sample. Generation may take several minutes for large datasets. ```python import asyncio from openjudge.generator.iterative_rubric.generator import ( IterativeRubricsGenerator, IterativePointwiseRubricsGeneratorConfig, ) config = IterativePointwiseRubricsGeneratorConfig( grader_name="Science QA Grader", model=model, task_description="Evaluate factual answers to science questions", min_score=0, max_score=1, max_epochs=3, batch_size=5, ) generator = IterativeRubricsGenerator(config) async def main(): train_data = [ {"query": "What is gravity?", "response": "A force attracting masses.", "label_score": 1}, {"query": "What is gravity?", "response": "Something heavy.", "label_score": 0}, {"query": "What is entropy?", "response": "Measure of disorder.", "label_score": 1}, {"query": "What is entropy?", "response": "A type of energy.", "label_score": 0}, ] # Generate grader — may take several minutes for large datasets grader = await generator.generate(dataset=train_data) # Evaluate new samples result = await grader.aevaluate( query="What is osmosis?", response="Osmosis is the movement of water across a semi-permeable membrane.", ) print(f"score={result.score} reason={result.reason}") asyncio.run(main()) ``` -------------------------------- ### ActionAlignmentGrader usage example Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/built_in_graders/agent_graders.md Shows how to use ActionAlignmentGrader to evaluate alignment between a plan and an action. Requires an OpenAIChatModel and an async main function. The example prints the score and reason; expected score is 1.0 for good alignment. ```python import asyncio from openjudge.models import OpenAIChatModel from openjudge.graders.agent import ActionAlignmentGrader async def main(): model = OpenAIChatModel(model="qwen3-32b") grader = ActionAlignmentGrader(model=model) result = await grader.aevaluate( plan="I will open drawer 1 to find the key.", action="open drawer 1", context="Task: Find the key to unlock the door" ) print(f"Score: {result.score}") # 1.0 - good alignment print(f"Reason: {result.reason}") asyncio.run(main()) ``` ```text Score: 1.0 Reason: The action 'open drawer 1' directly implements the stated plan 'I will open drawer 1 to find the key.' It targets the correct object (drawer 1), contributes to achieving the goal of finding the key, follows the logical order outlined in the plan, and respects any implied preconditions (e.g., needing to open the drawer to access its contents). The alignment is clear and direct, so confidence is high. ``` -------------------------------- ### Install legacy rm-gallery package Source: https://github.com/agentscope-ai/openjudge/blob/main/README.md Install the legacy v0.1.x package for users who choose to stay on the old version. This is not recommended for new projects as v0.2.0 is not backward compatible. ```bash pip install rm-gallery ``` -------------------------------- ### SkillThreatAnalysisGrader usage example Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/built_in_graders/skills.md Shows how to instantiate the grader with an OpenAIChatModel and evaluate a skill package. The example uses a threshold of 3 and prints the score, reason, and findings. The expected output is also shown. ```python import asyncio from openjudge.models import OpenAIChatModel from openjudge.graders.skills import SkillThreatAnalysisGrader async def main(): model = OpenAIChatModel(model="qwen3-32b") grader = SkillThreatAnalysisGrader(model=model, threshold=3) result = await grader.aevaluate( skill_name="code-formatter", skill_manifest="name: code-formatter\ndescription: Formats Python source files locally.", instruction_body="# Code Formatter\nFormat the provided Python code using black.", script_contents=["import black\nblack.format_str(code, mode=black.Mode())"], reference_contents=[], ) print(f"Score: {result.score}") # 4 — Clean print(f"Reason: {result.reason}") print(f"Findings: {result.metadata['findings']}") asyncio.run(main()) ``` ```text Score: 4 Reason: The skill package contains no security findings. The YAML manifest and instructions describe a legitimate local code-formatting operation matching the declared purpose. Findings: [] ``` -------------------------------- ### Start Ray single-machine cluster Source: https://github.com/agentscope-ai/openjudge/blob/main/cookbooks/training_judge_model/grpo/README.md Starts a single-machine Ray cluster with all resources, checks its status, and points to the dashboard at http://localhost:8265. Recommended for testing. ```bash # Start Ray with all resources ray start --head --port=6379 --dashboard-port=8265 # Check status ray status # View dashboard at http://localhost:8265 ``` -------------------------------- ### Start Ray Cluster for GRPO Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/building_graders/training_judge_models.md Commands to start a Ray head node and verify the cluster. Required before running GRPO training. ```bash # Start Ray head node ray start --head --port=6379 --dashboard-port=8265 # Verify cluster ray status ``` -------------------------------- ### Install httpx dependency Source: https://github.com/agentscope-ai/openjudge/blob/main/skills/claude-authenticity/SKILL.md Install the only required dependency for the Claude authenticity verification script. No other packages or setup are needed. ```bash pip install httpx ``` -------------------------------- ### Docker environment setup for training Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/building_graders/training_judge_models.md Build and run the pre-configured training Docker image. Requires the OpenJudge repository and mounts directories for models and data. ```bash cd OpenJudge # Build the training image docker build -f docker/Dockerfile.train -t openjudge-train:latest . # Run the container docker run --gpus all -it \ --shm-size=64g \ -v $(pwd):/workspace/OpenJudge \ -v /path/to/your/models:/models \ -v /path/to/your/data:/data \ --name openjudge-train \ openjudge-train:latest ``` -------------------------------- ### Prefix and Suffix Matching with StringMatchGrader Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/built_in_graders/text.md Shows how to use StringMatchGrader with algorithm='prefix_match' and 'suffix_match' to check if the response starts or ends with the reference. The prefix example returns a score of 1.0 for 'Hello' in 'Hello World', and the suffix example returns 1.0 for 'World' in 'Hello World'. ```python import asyncio from openjudge.graders.text.string_match import StringMatchGrader async def main(): # Prefix matching prefix_grader = StringMatchGrader(algorithm="prefix_match") result = await prefix_grader.aevaluate( reference_response="Hello", response="Hello World", ) print(f"Prefix Score: {result.score}") # 1.0 # Suffix matching suffix_grader = StringMatchGrader(algorithm="suffix_match") result = await suffix_grader.aevaluate( reference_response="World", response="Hello World", ) print(f"Suffix Score: {result.score}") # 1.0 asyncio.run(main()) ``` -------------------------------- ### Quick examples for text, image, video, speech, music, and search Source: https://github.com/agentscope-ai/openjudge/blob/main/skills/mmx-cli/SKILL.md Shows basic usage of each mmx command: text generation, image generation, video generation, speech synthesis, music generation, and web search. Each command uses the default model unless specified. ```bash # Generate text mmx text generate "Summarize the key benefits of reinforcement learning" # Generate an image mmx image generate "A futuristic city skyline at sunset, photorealistic" # Generate a short video clip mmx video generate "A golden retriever playing in autumn leaves" # Synthesize speech from text mmx speech generate --text "Hello, welcome to the demo." --voice Calm_Woman # Generate music with lyrics mmx music generate --lyrics "Rise and shine, a brand new day" --style "pop upbeat" # Web search mmx search "latest advances in LLM evaluation" ``` -------------------------------- ### Labeling guide template Source: https://github.com/agentscope-ai/openjudge/blob/main/skills/eval_pipeline/01-eval-design/SKILL.md Template for human annotation tasks. Provides structure for defining the criterion, pass/fail conditions, examples, and edge cases. Use when human annotation is needed. ```markdown ## Annotation Task: [dimension_name] **Criterion**: [what the dimension measures] **Pass**: [concrete, observable conditions for pass] **Fail**: [concrete, observable conditions for fail] **Examples**: - Input: "..." | Output: "..." | Judgment: Pass | Reason: ... - Input: "..." | Output: "..." | Judgment: Fail | Reason: ... **Edge cases**: - If X happens but Y doesn't → [how to judge] - If both A and B are present → [which takes priority] ``` -------------------------------- ### Per-Field Accuracy Output Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/validating_graders/ref_hallucination_arena.md Example output showing per-field accuracy for a model, breaking down title, author, year, and DOI accuracy. This breakdown helps identify patterns where a model may get titles right but fabricate author names or DOIs. ```text Per-Field Accuracy (Model A): Title Accuracy : 82.3% # Percentage of titles matching real papers Author Accuracy : 68.5% # Percentage of correct author lists Year Accuracy : 71.2% # Percentage of correct publication years DOI Accuracy : 45.8% # Percentage of valid DOIs ``` -------------------------------- ### Minimal YAML configuration for arena evaluation Source: https://github.com/agentscope-ai/openjudge/blob/main/skills/arena-eval/01-auto-arena/SKILL.md Minimal YAML configuration for the arena evaluation tool. Defines the task description, two target endpoints (model_v1 and model_v2) with environment-variable API keys, and a judge endpoint. Use this as a starting point for a basic evaluation setup. ```yaml task: description: "Academic GPT assistant for research and writing tasks" target_endpoints: model_v1: base_url: "https://api.openai.com/v1" api_key: "${OPENAI_API_KEY}" model: "gpt-4" model_v2: base_url: "https://api.openai.com/v1" api_key: "${OPENAI_API_KEY}" model: "gpt-3.5-turbo" judge_endpoint: base_url: "https://api.openai.com/v1" api_key: "${OPENAI_API_KEY}" model: "gpt-4" ``` -------------------------------- ### Install OpenJudge with Docker Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/get_started/quickstart.md Clone the repository, build the Docker image, and run the container with a mounted workspace. ```bash git clone https://github.com/agentscope-ai/OpenJudge.git cd OpenJudge # Build the Docker image docker build -f docker/Dockerfile.base -t openjudge:latest . # Run the container docker run -it \ -v $(pwd):/workspace/OpenJudge \ --name openjudge \ openjudge:latest ``` -------------------------------- ### Run GradingRunner with CorrectnessGrader and RelevanceGrader Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/running_graders/run_tasks.md This example shows the basic setup of a GradingRunner with two graders, a weighted sum aggregator, and concurrency control. It uses OpenAIChatModel with the 'qwen3-32b' model. The runner's arun method is asynchronous and awaits results, with max_concurrency set to 5. ```python from openjudge.runner.grading_runner import GradingRunner from openjudge.runner.aggregator.weighted_sum_aggregator import WeightedSumAggregator from openjudge.graders.common.correctness import CorrectnessGrader from openjudge.graders.common.relevance import RelevanceGrader from openjudge.models.openai_chat_model import OpenAIChatModel # Prepare your data in whatever format works for you data = [ { "query": "What is the capital of France?", "response": "Paris", "reference_answer": "Paris" }, { "query": "Who wrote Romeo and Juliet?", "response": "Shakespeare", "reference_answer": "William Shakespeare" } ] # Configure graders with mappers to connect your data fields grader_configs = { "correctness": { "grader": CorrectnessGrader(model=OpenAIChatModel("qwen3-32b")), "mapper": {"query": "query", "response": "response", "reference_response": "reference_answer"} }, "relevance": { "grader": RelevanceGrader(model=OpenAIChatModel("qwen3-32b")), "mapper": {"query": "query", "response": "response", "reference_response": "reference_answer"} } } # Configure aggregators to combine results aggregators = [ WeightedSumAggregator(name="overall_quality", weights={"correctness": 0.6, "relevance": 0.4}) ] # Run evaluation with concurrency control runner = GradingRunner( grader_configs, aggregators=aggregators, max_concurrency=5 ) results = await runner.arun(data) ``` -------------------------------- ### Complete Pipeline Example Source: https://github.com/agentscope-ai/openjudge/blob/main/skills/openjudge-core/01-graders-and-pipeline/pipeline.md Sets up an OpenAIChatModel with the qwen-plus model, configures a VotingEvaluationStrategy with 3 votes, defines a dataset with a single query-response pair, and creates a GradingRunner with three graders: CorrectnessGrader, RelevanceGrader, and StringMatchGrader. The runner uses a WeightedSumAggregator to combine scores and runs asynchronously, printing scores or errors for each grader. ```python import asyncio from openjudge.models.openai_chat_model import OpenAIChatModel from openjudge.graders.common.correctness import CorrectnessGrader from openjudge.graders.common.relevance import RelevanceGrader from openjudge.graders.text.string_match import StringMatchGrader from openjudge.runner.grading_runner import GradingRunner, GraderConfig from openjudge.runner.aggregator.weighted_sum_aggregator import WeightedSumAggregator from openjudge.evaluation_strategy import VotingEvaluationStrategy from openjudge.graders.schema import GraderScore, GraderError model = OpenAIChatModel(model="qwen-plus", api_key="sk-xxx", base_url="https://dashscope.aliyuncs.com/compatible-mode/v1") # Voting strategy for LLM-based graders voting = VotingEvaluationStrategy(num_votes=3) dataset = [ { "query": "What is the capital of France?", "response": "Paris", "reference": "Paris", "reference_response": "The capital of France is Paris.", }, ] runner = GradingRunner( grader_configs={ "correctness": CorrectnessGrader(model=model, strategy=voting), "relevance": RelevanceGrader(model=model, strategy=voting), "exact_match": GraderConfig( grader=StringMatchGrader(), mapper={"response": "response", "reference_response": "reference"}, ), }, aggregators=WeightedSumAggregator( name="overall", weights={"correctness": 0.5, "relevance": 0.3, "exact_match": 0.2}, ), max_concurrency=8, ) async def main(): results = await runner.arun(dataset) for grader_name, grader_results in results.items(): for i, result in enumerate(grader_results): if isinstance(result, GraderScore): print(f"[{grader_name}][{i}] score={result.score:.3f}") elif isinstance(result, GraderError): print(f"[{grader_name}][{i}] ERROR: {result.error}") asyncio.run(main()) ``` -------------------------------- ### Prioritized Recommendations (P0/P1/P2) Example Source: https://github.com/agentscope-ai/openjudge/blob/main/skills/eval_pipeline/04-eval-report/SKILL.md Shows the format for generating prioritized actions. Each item includes priority, concrete action, current state, target state, expected impact, and the skill to use. The example includes P0, P1, and P2 items with specific metrics and thresholds. ```plaintext 🔴 P0 | Calibrate hallucination judge Current: TPR=0.74 (below 0.8 threshold) Target: TPR >= 0.8 Impact: Judge becomes usable as production gate Use: 03-align-human 🔴 P0 | Add boundary samples for hallucination Current: n=6 boundary samples (CI half-width ±18%) Target: n >= 20 (CI narrows to ±10%) Impact: Reliable per-stratum TPR measurement Use: 01-eval-design 🟡 P1 | Align tone_consistency judge Current: kappa=0.72, bias=-0.15 (lenient) Target: kappa >= 0.8, |bias| < 0.1 Impact: Reduce false positive rate ~15% Use: 03-align-human 🟢 P2 | Enable auto-gate for factuality judge Current: kappa=0.87, TPR=0.92, TNR=0.88 Impact: Eliminate 90% of human review for this dimension Use: 03-align-human (mark Phase 3) ``` -------------------------------- ### Start Training Container Source: https://github.com/agentscope-ai/openjudge/blob/main/docker/README.md Runs the OpenJudge training container with GPU support, shared memory, and mounted model and data directories. ```bash docker run --gpus all -it \ --shm-size=64g \ -v $(pwd):/workspace/OpenJudge \ -v /path/to/your/models:/models \ -v /path/to/your/data:/data \ --name openjudge-train \ openjudge-train:latest ``` -------------------------------- ### Install py-openjudge with pip Source: https://github.com/agentscope-ai/openjudge/blob/main/README.md Install the py-openjudge package via pip. ```bash pip install py-openjudge ``` -------------------------------- ### Start Base Container Source: https://github.com/agentscope-ai/openjudge/blob/main/docker/README.md Runs the OpenJudge container for evaluation, mounting the current directory and setting the OpenAI API key. ```bash docker run -it \ -v $(pwd):/workspace/OpenJudge \ -e OPENAI_API_KEY=your_api_key \ --name openjudge \ openjudge:latest ``` -------------------------------- ### Component 3: Few-Shot Examples (PASS, FAIL, BORDERLINE) Source: https://github.com/agentscope-ai/openjudge/blob/main/skills/eval_pipeline/02-metric-design/SKILL.md The third component: at least one pass, one fail, and one borderline example. The borderline example teaches the judge where the boundary is. ```text Example 1 (PASS): Context: "Order #12345: shipped May 10" Response: "Your order #12345 was shipped on May 10 and arrives May 12." Critique: The response uses the exact order number (#12345) and matches the ship date from context. No fabrication or omission. Result: Pass Example 2 (FAIL): Context: "Order #12345: shipped May 10" Response: "Your order #12346 is on its way!" Critique: The response uses order #12346 but the context only mentions #12345. This is a fabricated order number, not a typo — #12346 doesn't exist. Result: Fail Example 3 (BORDERLINE PASS): Context: "Orders #12345 (shipped), #12346 (processing)" Response: "Your recent order has shipped and should arrive soon." Critique: The response doesn't specify which order, but says "recent order" which could reasonably refer to either. If the customer only asked about shipped items, this is fine. If they asked about a specific order, it's insufficient. Given the generic phrasing, this passes but is weak. Result: Pass ``` -------------------------------- ### Configure and run GradingRunner with graders, mappers, and aggregators Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/get_started/core_concepts.md Complete configuration example showing how to set up a GradingRunner with two graders (HelpfulnessGrader and RelevanceGrader), data mappers, a WeightedSumAggregator, and concurrency control. The runner is executed asynchronously with await runner.arun(data). ```python from openjudge.runner.grading_runner import GradingRunner from openjudge.runner.aggregator.weighted_sum_aggregator import WeightedSumAggregator from openjudge.graders.common.helpfulness import HelpfulnessGrader from openjudge.graders.common.relevance import RelevanceGrader from openjudge.models.openai_chat_model import OpenAIChatModel # Prepare your data in whatever format works for you data = [ { "query": "What is the capital of France?", "response": "Paris", "reference_answer": "Paris" }, { "query": "Who wrote Romeo and Juliet?", "response": "Shakespeare", "reference_answer": "William Shakespeare" } ] # Configure graders with mappers to connect your data fields graders = { "helpfulness": { "grader": HelpfulnessGrader(model=OpenAIChatModel("qwen3-32b")), "mapper": {"question": "query", "answer": "response"} }, "relevance": { "grader": RelevanceGrader(model=OpenAIChatModel("qwen3-32b")), "mapper": {"q": "query", "a": "response", "ref": "reference_answer"} } } # Configure aggregators to combine results aggregators = [ WeightedSumAggregator(weights={"helpfulness": 0.6, "relevance": 0.4}) ] # Run evaluation with concurrency control runner = GradingRunner( graders, aggregators=aggregators, max_concurrency=5 ) results = await runner.arun(data) ``` -------------------------------- ### Display help message Source: https://github.com/agentscope-ai/openjudge/blob/main/experiments/README.md Shows full usage instructions for the script. ```bash ./run_grader_evals.sh --help ``` -------------------------------- ### Install pypdfium2 Source: https://github.com/agentscope-ai/openjudge/blob/main/skills/academic-eval/01-paper-review/reference.md Install the pypdfium2 package to resolve the ModuleNotFoundError when using vision mode. ```bash pip install pypdfium2 ``` -------------------------------- ### Create a dataset Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/integrations/langsmith.md Create a LangSmith dataset with labeled examples containing questions and expected answers. Uses client.create_dataset and client.create_examples. ```python from langsmith import Client client = Client() # Create examples with inputs and expected outputs examples = [ { "inputs": {"question": "What is the capital of France?"}, "outputs": {"expected_answer": "Paris"} }, { "inputs": {"question": "How many planets are in our solar system?"}, "outputs": {"expected_answer": "8"} }, { "inputs": {"question": "Who wrote Romeo and Juliet?"}, "outputs": {"expected_answer": "William Shakespeare"} } ] # Create the dataset dataset = client.create_dataset(dataset_name="QA Evaluation Dataset") client.create_examples( dataset_id=dataset.id, examples=examples ) ``` -------------------------------- ### Install litellm Source: https://github.com/agentscope-ai/openjudge/blob/main/skills/academic-eval/01-paper-review/reference.md Install the litellm package to resolve the ModuleNotFoundError when running scripts that depend on it. ```bash pip install litellm ``` -------------------------------- ### Quick Start: Pointwise GRPO Training Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/building_graders/training_judge_models.md Command to run pointwise GRPO training from the cookbook directory. ```bash cd cookbooks/training_judge_model/grpo bash pointwise/run_pointwise.sh ``` -------------------------------- ### Install Python dependencies Source: https://github.com/agentscope-ai/openjudge/blob/main/experiments/README.md Installs the required Python packages: py-openjudge, datasets, and huggingface_hub. ```bash pip install py-openjudge datasets huggingface_hub ``` -------------------------------- ### Strategy A: Maximum Quality - Subtask Table and Install Commands Source: https://github.com/agentscope-ai/openjudge/blob/main/skills/find-skills-combo/SKILL.md Strategy A assigns the best-fit skill to every subtask for high coverage. The table lists each subtask with its skill and coverage level, followed by the install commands using `npx skills add` with `-g -y` flags. ```markdown | Subtask | Handled By | Coverage | |---------|-----------|----------| | S1 | skill-name-a | High | | S2 | skill-name-b | High | | S3 | skill-name-c | High | ``` ```bash npx skills add owner/repo@skill-a -g -y npx skills add owner/repo@skill-b -g -y npx skills add owner/repo@skill-c -g -y ``` -------------------------------- ### Install verl==0.6.1 Source: https://github.com/agentscope-ai/openjudge/blob/main/cookbooks/training_judge_model/grpo/README.md Install the required verl package version 0.6.1 using pip. ```bash pip install verl==0.6.1 ``` -------------------------------- ### Quick Start: Pairwise GRPO Training Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/building_graders/training_judge_models.md Command to run pairwise GRPO training from the cookbook directory. ```bash cd cookbooks/training_judge_model/grpo bash pairwise/run_pairwise.sh ``` -------------------------------- ### Start Ray Head Node Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/building_graders/training_judge_models.md Command to start a Ray head node, required for GRPO training. ```APIDOC ## Start Ray Head Node ### Description Starts a Ray head node on the specified port and dashboard port, required for GRPO training. ### Command ```bash ray start --head --port=6379 --dashboard-port=8265 ``` ### Parameters - `--port` (integer) - Required - Port for Ray head node (default: 6379) - `--dashboard-port` (integer) - Required - Port for Ray dashboard (default: 8265) ### Example ```bash ray start --head --port=6379 --dashboard-port=8265 ``` ``` -------------------------------- ### Python example: Using InstructionFollowingGrader with OpenAIChatModel Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/built_in_graders/general.md Use this snippet to evaluate instruction following with a model. It creates an InstructionFollowingGrader with an OpenAIChatModel, then calls aevaluate with an instruction and response. The example shows a good response scoring 5.0 and a poor response scoring 4.0, with reasons printed. ```python import asyncio from openjudge.models import OpenAIChatModel from openjudge.graders.common.instruction_following import InstructionFollowingGrader async def main(): model = OpenAIChatModel(model="qwen3-32b") grader = InstructionFollowingGrader(model=model) # Good instruction following result = await grader.aevaluate( instruction="Write exactly 3 bullet points about AI benefits.", response="• AI automates repetitive tasks\n• AI improves decision-making\n• AI enables personalization", ) print(f"Score: {result.score}") # Output: Score: 5.0 print(f"Reason: {result.reason}") # Output: Reason: The response perfectly adheres to the instruction. It provides exactly 3 bullet points about AI benefits, as required, without any additional or missing information. # Poor instruction following result = await grader.aevaluate( instruction="Write exactly 3 bullet points about AI benefits.", response="AI has many benefits. It can automate tasks, improve decisions, and personalize experiences. These benefits are significant.", ) print(f"Score: {result.score}") # Output: Score: 4.0 print(f"Reason: {result.reason}") # Output: Reason: The response provides exactly 3 bullet points as instructed, but combines them into a single sentence. While the content addresses the benefits of AI, the format deviates slightly from the expected bullet point structure. asyncio.run(main()) ``` -------------------------------- ### Install development dependencies Source: https://github.com/agentscope-ai/openjudge/blob/main/skills/eval_pipeline/README.md Installs the project in editable mode with dev dependencies, which includes the openai and python-dotenv packages. ```bash pip install -e ".[dev]" # brings in openai + python-dotenv (already project deps) ``` -------------------------------- ### Install OpenJudge and matplotlib Source: https://github.com/agentscope-ai/openjudge/blob/main/skills/arena-eval/01-auto-arena/SKILL.md Install py-openjudge and matplotlib (extra dependency for auto_arena chart generation) via pip. ```bash pip install py-openjudge pip install matplotlib ``` -------------------------------- ### Single Paper Review Quick Start Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/applications/paper_review.md Defines a PipelineConfig with the recommended gemini-3-pro-preview model and optional OpenAI-compatible proxy. Sets timeouts and enables safety, correctness, review, and criticality checks. Initiates review_and_report on a PDF and prints the review score and report path. ```python import asyncio import os from cookbooks.paper_review import PaperReviewPipeline, PipelineConfig, generate_report async def main(): config = PipelineConfig( model_name="gemini-3-pro-preview", # Recommended api_key=os.environ.get("OPENAI_API_KEY", ""), base_url=os.environ.get("OPENAI_BASE_URL", ""), # OpenAI-compatible proxy timeout=1500, enable_safety_checks=True, enable_correctness=True, enable_review=True, enable_criticality=True, ) pipeline = PaperReviewPipeline(config) # Review paper and generate report result, report = await pipeline.review_and_report( pdf_input="paper.pdf", paper_name="My Research Paper", output_path="review_report.md", ) print(f"Review Score: {result.review.score}/6") print(f"Report saved to: review_report.md") asyncio.run(main()) ``` -------------------------------- ### ToolSelectionGrader Example Output Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/get_started/evaluate_ai_agents.md Example output of a successful ToolSelectionGrader evaluation, showing a perfect score and the reasoning behind it. ```python GraderScore( name='tool_selection', score=5.0, reason="The agent selected the 'calculator' tool with the expression '45 * 0.15', which is the most direct and efficient tool for computing a percentage-based tip. The query is purely mathematical, requiring no external information. The calculator tool is fully capable of performing this arithmetic operation accurately." ) ``` -------------------------------- ### Evaluation Report Configuration Source: https://github.com/agentscope-ai/openjudge/blob/main/docs/applications/auto_arena.md YAML configuration for generating a comprehensive Markdown report with concrete examples, including language and example count. ```yaml report: enabled: true # Enable report generation language: "zh" # "zh" (Chinese) or "en" (English) include_examples: 3 # Examples per section (1-10) ``` -------------------------------- ### Executive Summary Example Source: https://github.com/agentscope-ai/openjudge/blob/main/skills/eval_pipeline/04-eval-report/SKILL.md Shows the format for a one-page executive summary for non-technical stakeholders. Includes system, stakes, report date, ship readiness, top risks, eval maturity, and next actions. The example uses a specific chatbot system and includes conditional ship readiness with two blocking items. ```plaintext Executive Summary ================= System: Customer support chatbot for e-commerce Stakes: Production Report Date: 2026-05-12 SHIP READINESS: Conditional 2 items must be resolved before production gate: 1. Hallucination judge TPR below threshold (0.74 < 0.8) 2. No safety/redteam evaluation has been run TOP 3 RISKS: 1. Hallucination detection unreliable — severity: HIGH The judge measuring whether the bot fabricates information itself has poor recall (TPR=0.74), meaning ~26% of hallucinations go undetected. Mitigation: Calibrate with more boundary labels (2-3 weeks). 2. Safety coverage missing — severity: MEDIUM No jailbreak, injection, or PII leakage testing has been performed. Mitigation: Run redteam skill this sprint (1-2 days). 3. Tone evaluation is biased lenient — severity: LOW The tone judge systematically rates responses as better than humans do. Mitigation: Refine judge prompt with borderline examples. EVAL MATURITY: L2 (Developing) → Target L3 in 3-4 weeks Strongest: Failure Discovery (L3) Weakest: Safety Coverage (L1), Human Alignment (L2) NEXT ACTIONS (this sprint): [P0] Add boundary labels + recalibrate hallucination judge [P0] Run initial redteam evaluation [P1] Refine tone judge alignment ```