### Generate Example Instance via API Request

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Illustrates the API request format for generating a valid JSON instance against a given schema. Requires the `requests` and `json` libraries.

```python
import requests, json

schema = {
    "type": "object",
    "properties": {"name": {"type": "string"}, "age": {"type": "integer"}},
    "required": ["name"],
    "additionalProperties": False
}

# Request a valid instance (constrained to schema)
req = {
    "model": "model",
    "messages": [
        {"role": "system", "content": "You are an expert in generating testcases for JSON schema validators."},
        {"role": "user", "content": "Please generate an example data that validates against the following schema:\n" + json.dumps(schema, indent=4)}
    ],
    "response_format": {
        "type": "json_schema",
        "json_schema": {"strict": True, "schema": schema}
    },
    "max_tokens": 8000,
    "temperature": 0.7
}
response = requests.post("http://localhost:3001/v1/chat/completions",
                         headers={"Content-Type": "application/json"},
                         json=req)
result = response.json()
instance = json.loads(result["choices"Пожалуйста, предоставьте мне JSON, соответствующий предоставленной схеме.][0]["message"]["content"])
print(instance)  # e.g. {"name": "Alice", "age": 30}
```

--------------------------------

### Install MaskBench Dependencies

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Installs the necessary Python packages for running MaskBench benchmarks. This should be run from the maskbench/ directory.

```bash
pip install llguidance xgrammar outlines-core llama-cpp-python transformers
```

--------------------------------

### Implement Abstract Engine Base Class in Python

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Subclasses must implement init(), compile_grammar(), reset(), compute_mask(), and commit_token(). This example shows a minimal skeleton.

```python
from maskbench.engine import Engine

class MyEngine(Engine):
    """Minimal example engine skeleton."""

    def get_id(self) -> str:
        return "my_engine"

    def init(self):
        # Called once after tokenizer is set; initialize backend state here.
        self.grammar_state = None

    def compile_grammar(self, schema: dict):
        # Parse JSON Schema and build internal grammar/automaton.
        # Raises an exception on unsupported schemas.
        import json
        self.grammar_state = json.dumps(schema)  # placeholder

    def reset(self):
        # Called before each test instance; reset to start-of-sequence state.
        self.current_state = self.grammar_state

    def compute_mask(self):
        # Compute which tokens are allowed at the current position.
        # Store result internally; accessed later by commit_token.
        self.allowed_tokens = set(range(self.tokenizer.vocab_size))  # placeholder

    def commit_token(self, token_id: int) -> bool:
        # Advance the grammar state by accepting token_id.
        # Returns True if token was allowed, False otherwise.
        ok = token_id in self.allowed_tokens
        if ok:
            pass  # update self.current_state
        return ok
```

--------------------------------

### Generate Positive Instances with Constrained Decoding

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Generates valid JSON instances using constrained decoding, guiding the LLM to adhere strictly to the schema. Requires the JSB_DATA environment variable.

```bash
python creation/gen_tests_with_llm.py -c data/Github_easy/
```

--------------------------------

### Display MaskBench Help Information

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Shows all available command-line options and arguments for the run_maskbench.py script.

```bash
./scripts/run_maskbench.py --help
```

--------------------------------

### Run MaskBench with LLGuidance Engine

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Executes the MaskBench benchmark using the LLGuidance engine on the Github_easy dataset. This command assumes you are in the maskbench/ directory.

```bash
cd maskbench/
./scripts/run_maskbench.py --llg data/
```

--------------------------------

### Run MaskBench with Outlines Engine

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Executes the MaskBench benchmark using the Outlines engine on the full dataset. This command assumes you are in the maskbench/ directory.

```bash
./scripts/run_maskbench.py --outlines data/
```

--------------------------------

### Run Decoding Simulation with GrammarMatcher

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Simulates decoding a JSON instance using a GrammarMatcher to check token acceptance. Ensure the grammar is compiled and the tokenizer is initialized.

```python
matcher = xgr.GrammarMatcher(compiled)
instance = '{"value": 42}'
tokens = tokenizer.encode(instance, add_special_tokens=False)

for token_id in tokens:
    matcher.fill_next_token_bitmask(token_bitmask)     # compute mask
    accepted = matcher.accept_token(token_id)
    if not accepted:
        print(f"Token {token_id} rejected")
        break
print("All tokens accepted.")
```

--------------------------------

### Load JSONSchemaBench Dataset with Hugging Face

Source: https://github.com/guidance-ai/jsonschemabench/blob/main/README.md

Demonstrates how to load the JSONSchemaBench dataset from the Hugging Face Hub using the `datasets` library. This is the primary method for accessing the benchmark data.

```python
from datasets import load_dataset

dataset = load_dataset("epfl-dlab/JSONSchemaBench")
print(dataset)
```

--------------------------------

### Fetch and Convert BFCL Schemas (Python)

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Python script to download function-calling schemas from the Gorilla BFCL v3 dataset. It converts Gorilla-style function definitions to JSON Schema and validates ground-truth completions.

```python
# Programmatic usage – fetches, converts, validates, and saves BFCL schemas
from creation.fetch_bfcl import load_gorilla_data, convert_function_to_schema, convert_ground_truth_to_completion
import os, json

os.makedirs("out", exist_ok=True)

# Download all BFCL v3 files and produce out/BFCL_<id>.json schema files
gorilla_data = load_gorilla_data()
# Output: out/BFCL_java_0.json, out/BFCL_javascript_0.json, ...

# Convert a single Gorilla function definition to JSON Schema
function_def = {
    "name": "get_weather",
    "parameters": {
        "type": "object",
        "properties": {
            "city":  {"type": "string", "description": "City name"},
            "units": {"type": "string", "enum": ["celsius", "fahrenheit"]}
        },
        "required": ["city"]
    }
}
schema = convert_function_to_schema(function_def)
print(json.dumps(schema, indent=2))
# {
#   "type": "object",
#   "properties": {
#     "get_weather": {
#       "type": "object",
#       "properties": {
#         "city":  {"type": "string"},
#         "units": {"type": "string", "enum": ["celsius", "fahrenheit"]}
#       },
#       "required": ["city"],
#       "additionalProperties": false
#     }
#   },
#   "required": ["get_weather"],
#   "additionalProperties": false
# }

# Convert a ground-truth answer into the expected completion format
ground_truth = {"get_weather": {"city": ["Paris"], "units": ["celsius"]}}
completion = convert_ground_truth_to_completion(ground_truth)
print(completion)  # {"get_weather": {"city": "Paris", "units": "celsius"}}
```

--------------------------------

### Load and Validate Local JSON Schema Files

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Shows how to load individual JSON schema files from the local data directory and validate JSON instances against them using the `jsonschema` library. Includes parsing the file and iterating through test cases.

```python
import json

# Load a single schema file (e.g., from Github_easy split)
with open("data/Github_easy/o10008.json") as f:
    entry = json.load(f)

print(entry["schema"])
# Example output:
# {
#   "type": "object",
#   "properties": {
#     "name": {"type": "string"},
#     "count": {"type": "integer", "minimum": 0}
#   },
#   "required": ["name"],
#   "additionalProperties": false
# }

for test in entry["tests"]:
    print(test["valid"], test["data"])
# True  {'name': 'example', 'count': 5}
# False {'name': 123}                   # violates type constraint

# Validate instances with jsonschema
from jsonschema import Draft202012Validator, validate

schema = entry["schema"]
for test in entry["tests"]:
    try:
        validate(test["data"], schema,
                 format_checker=Draft202012Validator.FORMAT_CHECKER)
        result = "valid"
    except Exception as e:
        result = f"invalid: {e.message}"
    expected = "valid" if test["valid"] else "invalid"
    print(f"Expected {expected}, got {result}")
```

--------------------------------

### Run MaskBench with XGrammar in Compliant Mode

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Runs MaskBench with the XGrammar engine in compliant mode, specifying a custom output path, thread count, time limit, and memory limit. Ensure you are in the maskbench/ directory.

```bash
./scripts/run_maskbench.py --xgr-compliant --output tmp/out-xgr-compliant \
    --num-threads 16 --time-limit 600 --mem-limit 20 data/
```

--------------------------------

### Run MaskBench Script

Source: https://github.com/guidance-ai/jsonschemabench/blob/main/maskbench/README.md

Execute the main script to run MaskBench tests. Use the --xgr-compliant flag for specific data processing. Results are saved in the 'tmp/out--xgr-compliant' directory. Refer to the script's help for more options, especially resource limits.

```bash
./scripts/run_maskbench.py --xgr-compliant data/
```

--------------------------------

### Programmatic Single-File MaskBench Runner Invocation

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Demonstrates how to programmatically invoke the MaskBench single-file runner using sys.argv to simulate command-line arguments. This is useful for debugging or custom execution flows.

```python
import sys
sys.argv = [
    "runner",
    "--llg",
    "--tokenizer", "unsloth/Meta-Llama-3.1-8B-Instruct",
    "--time-limit", "900",
    "--mem-limit", "40",
    "data/Github_easy---o10008.json"
]
from maskbench.runner import main
main()
```

--------------------------------

### Analyze and Compare Multiple MaskBench Engines

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Analyzes output directories for multiple MaskBench engines side-by-side and updates the README.md file with aggregated results. Ensure you are in the maskbench/ directory.

```bash
./scripts/maskbench_results.py tmp/out--llg tmp/out--xgr tmp/out--llamacpp
```

--------------------------------

### Validate and Add Single Generated Instance to Schema File (Python)

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Python script that mirrors the logic of `add_tests.py` for validating and adding a single generated instance to a schema file. Requires `jsonschema` library.

```python
# Validate and add a single generated instance to a schema file (mirrors add_tests logic)
import json
from jsonschema import Draft202012Validator, validate

schema_file = "data/Github_easy/o10008.json"
with open(schema_file) as f:
    entry = json.load(f)

schema = entry["schema"]
existing = set(json.dumps(t["data"]) for t in entry["tests"])

new_instance = {"name": "example", "count": 7}
serialized = json.dumps(new_instance)

if serialized not in existing:
    try:
        validate(new_instance, schema,
                 format_checker=Draft202012Validator.FORMAT_CHECKER)
        # Instance is valid – add as positive test
        entry["tests"].append({
            "description": "llama 70b generated positive",
            "valid": True,
            "data": new_instance
        })
        with open(schema_file, "w") as f:
            json.dump(entry, f, indent=2, ensure_ascii=False)
        print("Test added.")
    except Exception as e:
        print(f"Validation failed (expected for negative tests): {e}")
```

--------------------------------

### Analyze MaskBench Output for a Single Engine

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Analyzes the output directory of a single MaskBench engine (e.g., LLGuidance) to compute statistics and generate a Markdown table. Run this from the maskbench/ directory.

```bash
cd maskbench/
./scripts/maskbench_results.py tmp/out--llg
```

--------------------------------

### Generate Positive Test Instances with LLM

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Generates valid JSON instances for schemas in the specified directory using an LLM. This script requires the JSB_DATA environment variable to be set.

```bash
python creation/gen_tests_with_llm.py data/Github_easy/
```

--------------------------------

### Add Generated Tests to Schema Files (Bash)

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Bash commands to execute the `add_tests.py` script. It merges LLM-generated responses back into schema files, validating and deduplicating tests.

```bash
export JSB_DATA=/path/to/jsonschemabench_data

# Add generated tests from the responses directory to schema files
python creation/add_tests.py "$JSB_DATA/work/responses/"

# Add tests from a specific file
python creation/add_tests.py "$JSB_DATA/work/responses/o10008.json"
```

--------------------------------

### Programmatic MaskBench Results Analysis

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Shows how to programmatically use the maskbench_results.py script by setting sys.argv to specify the output directories to analyze. The script will generate various output files including CSVs and plots.

```python
import sys
sys.argv = ["maskbench_results.py", "tmp/out--llg", "tmp/out--xgr"]
# Script reads tmp/out--llg/*.json, tmp/out--llg/meta.txt, etc.
# Produces:
#   tmp/out--llg/stats.txt      – full Stats object as JSON
#   tmp/out--llg/entries.txt    – TBM/TTFM percentiles as JSON
#   tmp/out--llg/ttfm_us.csv    – log-fraction CDF data for TTFM
#   tmp/out--llg/masks_us.csv   – log-fraction CDF data for TBM
#   tmp/out--llg/histogram.csv  – fraction of masks above each time bucket
#   plots/tbm.png               – horizontal bar chart of TBM percentiles
#   plots/ttfm.png              – horizontal bar chart of TTFM percentiles

# Example entries.txt content (for LLGuidance):
# {
#   "TBM avg": 64,
#   "TBM p25": 29,   "TBM p50": 46,  "TBM p75": 55,
#   "TBM p90": 77,   "TBM p95": 119, "TBM p99": 533,
#   "TTFM avg": 1850, "TTFM p75": 1534,
#   "schemas": 11306, "passing": 8909,
#   "compile error": 2377, "timeout": 0, "validation error": 20
# }
```

--------------------------------

### LLGuidance Engine for Constrained Decoding in Python

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Wraps the llguidance library to compile JSON Schema into a grammar and fill token bitmasks. Best average TBM and lowest tail latency.

```python
from transformers import AutoTokenizer
import llguidance as llg
import llguidance.hf
from llguidance.numpy import fill_next_token_bitmask, allocate_token_bitmask
import json

# Initialize tokenizer
tokenizer = AutoTokenizer.from_pretrained("unsloth/Meta-Llama-3.1-8B-Instruct")
llg_tokenizer = llguidance.hf.from_tokenizer(tokenizer)

# Allocate bitmask buffer (batch_size=1)
mask_data = allocate_token_bitmask(1, llg_tokenizer.vocab_size)

# Define a JSON Schema
schema = {
    "type": "object",
    "properties": {
        "city": {"type": "string"},
        "temperature": {"type": "number"}
    },
    "required": ["city", "temperature"],
    "additionalProperties": False
}

# Compile grammar
grammars = json.dumps({"grammars": [{"json_schema": schema}]})
matcher = llg.LLMatcher(llg_tokenizer, grammars)
if matcher.is_error():
    raise ValueError(f"Compile error: {matcher.get_error()}")

# Generate token-by-token (simulate decoding)
instance = '{"city": "Paris", "temperature": 18.5}'
tokens = tokenizer.encode(instance, add_special_tokens=False)

interp = matcher.deep_copy()  # reset per test instance
for token_id in tokens:
    fill_next_token_bitmask(interp, mask_data, 0)         # compute mask
    allowed = (mask_data[0, token_id // 32] & (1 << (token_id % 32))) != 0
    if allowed:
        interp.consume_token(token_id)
    else:
        print(f"Token {token_id!r} rejected")
        break
print("Decoding complete, all tokens accepted.")
```

--------------------------------

### Convert JSON to Unique Tests

Source: https://github.com/guidance-ai/jsonschemabench/blob/main/maskbench/creation/README.md

Use this script to create or update JSON files in the unique_tests directory from JSON files in the data directory. Ensure the JSB_DATA environment variable is set.

```bash
json_to_unique_tests.py $JSB_DATA/json/*
```

--------------------------------

### Quick Analysis of MaskBench Results

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Performs a quick analysis of MaskBench results by skipping re-aggregation and loading cached statistics. Useful for rapidly viewing results from multiple engines.

```bash
./scripts/maskbench_results.py -q tmp/out--llg tmp/out--xgr
```

--------------------------------

### Debug MaskBench on a Single Schema File

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Runs the MaskBench runner on a specific JSON schema file for debugging purposes. This command uses the LLGuidance engine.

```bash
python -m maskbench.runner --llg --debug data/Github_easy---o13947.json
```

--------------------------------

### Analyze MaskBench Results

Source: https://github.com/guidance-ai/jsonschemabench/blob/main/maskbench/README.md

Generate tables and plots from the MaskBench results using the provided analysis script. This helps in visualizing and comparing performance data.

```bash
./scripts/maskbench_results.py
```

--------------------------------

### Generate Tests with LLM

Source: https://github.com/guidance-ai/jsonschemabench/blob/main/maskbench/creation/README.md

This script generates new test cases using a Large Language Model, saving the responses in the work/responses directory. It requires the unique_tests directory as input.

```bash
gen_tests_with_llm.py $JSB_DATA/unique_tests
```

--------------------------------

### Debug MaskBench Runner

Source: https://github.com/guidance-ai/jsonschemabench/blob/main/maskbench/README.md

Run the MaskBench runner in debug mode for a specific JSON configuration file. Use the --debug flag for detailed output. Replace --llg with other engine options (e.g., --xgr, --llm) as needed to test different engines.

```python
python -m maskbench.runner data/Github_easy---o13947.json --debug --llg
```

--------------------------------

### Generate Tests for a Single Schema File

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Generates negative (invalid) test instances for a specific JSON schema file using an LLM. Requires the JSB_DATA environment variable.

```bash
python creation/gen_tests_with_llm.py -n data/Github_easy/o10008.json
```

--------------------------------

### Generate Negative Test Instances with LLM

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Generates invalid JSON instances for schemas in the specified directory using an LLM, focusing on violating specific schema features. Requires the JSB_DATA environment variable.

```bash
python creation/gen_tests_with_llm.py -n data/Github_easy/
```

--------------------------------

### Set JSB Data Directory Environment Variable

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Sets the JSB_DATA environment variable, which is required by the gen_tests_with_llm.py script to locate the JSON schema benchmark data.

```bash
export JSB_DATA=/path/to/jsonschemabench_data
```

--------------------------------

### XGrammar Engine for Constrained Decoding in Python

Source: https://context7.com/guidance-ai/jsonschemabench/llms.txt

Wraps the xgrammar library's GrammarCompiler and GrammarMatcher. Supports strict and compliant modes. Degrades sharply at p99 TBM.

```python
from transformers import AutoTokenizer, AutoConfig
import xgrammar as xgr
import json

tokenizer_model_id = "unsloth/Meta-Llama-3.1-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(tokenizer_model_id)
config = AutoConfig.from_pretrained(tokenizer_model_id)

# Build tokenizer info (use full vocab size from config, not tokenizer)
tokenizer_info = xgr.TokenizerInfo.from_huggingface(
    tokenizer, vocab_size=config.vocab_size
)
token_bitmask = xgr.allocate_token_bitmask(1, tokenizer_info.vocab_size)

# Compile grammar in compliant mode (any whitespace, non-strict)
compiler = xgr.GrammarCompiler(tokenizer_info, max_threads=1)
schema = {"type": "object", "properties": {"value": {"type": "integer"}}, "required": ["value"]}
compiled = compiler.compile_json_schema(
    json.dumps(schema),
    any_whitespace=True,   # compliant mode
    strict_mode=False
)

```

--------------------------------

### Reorder JSON Keys

Source: https://github.com/guidance-ai/jsonschemabench/blob/main/maskbench/creation/README.md

Reorders keys within JSON files in the unique_tests directory to a consistent 'llguidance' format. This is useful for standardizing data structure.

```bash
reorder.py $JSB_DATA/unique_tests
```

--------------------------------

### Add Generated Tests to Unique Tests

Source: https://github.com/guidance-ai/jsonschemabench/blob/main/maskbench/creation/README.md

Integrates tests generated by the LLM into the unique_tests directory. This script updates existing JSON files with the new test data.

```bash
add_tests.py $JSB_DATA/work/responses
```

--------------------------------

### Convert Unique Tests to JSON

Source: https://github.com/guidance-ai/jsonschemabench/blob/main/maskbench/creation/README.md

This script updates JSON files in the data directory based on the content of the unique_tests directory. It serves as the inverse operation to `json_to_unique_tests.py`.

```bash
unique_tests_to_json.py $JSB_DATA/unique_tests
```

--------------------------------

### Add Reasons to Python Validation Errors

Source: https://github.com/guidance-ai/jsonschemabench/blob/main/maskbench/creation/README.md

Updates Python validation errors within the unique_tests directory. This script helps in providing more detailed explanations for validation failures.

```bash
add_reasons.py $JSB_DATA/unique_tests
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.