### Install luna-bench with uv

Source: https://github.com/aqarios/luna-bench/wiki/Home

Use uv for recommended installation of the luna-bench package.

```bash
uv add luna-bench
```

--------------------------------

### Install luna-bench with pip

Source: https://github.com/aqarios/luna-bench/wiki/Home

Install the luna-bench package using pip.

```bash
pip install luna-bench
```

--------------------------------

### Install Luna-Bench

Source: https://context7.com/aqarios/luna-bench/llms.txt

Install the core library and optional extras for metrics, plotting, or QUBO features.

```bash
pip install luna-bench
# Optional extras:
pip install luna-bench[opt-metrics]   # pyscipopt for ScipAlgorithm / OptSolFeature
pip install luna-bench[plot]          # matplotlib + seaborn for plots
pip install luna-bench[qubo-features] # networkx + scipy for QUBO features
```

--------------------------------

### Quick Example: Run a Benchmark

Source: https://github.com/aqarios/luna-bench/wiki/Home

This example demonstrates setting up and running a benchmark using Luna-Bench. It includes creating a model set, adding models, configuring the benchmark with features, metrics, and algorithms, and then running the benchmark pipeline.

```python
import multiprocessing

# Required on macOS to avoid multiprocessing issues
multiprocessing.set_start_method("fork")

from luna_quantum import Model
from luna_bench.components import Benchmark, ModelSet
from luna_bench.components.algorithms import ScipAlgorithm
from luna_bench.components.features import VarNumberFeature
from luna_bench.components.metrics import FeasibilityRatio, Runtime

# Create a model set and add models
model_set = ModelSet.create("my_models")
model_set.add(model)  # model is a luna_quantum.Model instance

# Create a benchmark and assign the model set
bench = Benchmark.create("my_benchmark")
bench.set_modelset(model_set)

# Register components
bench.add_feature(name="var_num", feature=VarNumberFeature())
bench.add_metric(name="runtime", metric=Runtime())
bench.add_metric(name="feasibility", metric=FeasibilityRatio())
bench.add_algorithm(name="scip", algorithm=ScipAlgorithm(max_runtime=30))

# Run the full pipeline: features -> algorithms -> metrics -> plots
bench.run()

# Or run each stage individually:
# bench.run_features()
# bench.run_algorithms()
# bench.run_metrics()
# bench.run_plots()
```

--------------------------------

### Install Dependencies with uv

Source: https://github.com/aqarios/luna-bench/blob/main/README.md

Use the 'uv sync' command to install all project dependencies. This command ensures your environment matches the project's requirements.

```bash
# Install dependencies
uv sync
```

--------------------------------

### Clone Repository and Install Dependencies

Source: https://github.com/aqarios/luna-bench/wiki/Contributing

Clone the Luna-Bench repository and install its dependencies using uv.

```bash
git clone https://github.com/aqarios/luna-bench.git
cd luna-bench
uv sync
```

--------------------------------

### macOS Multiprocessing Setup

Source: https://github.com/aqarios/luna-bench/blob/main/README.md

On macOS, set the multiprocessing start method to 'fork' before other imports to avoid known issues.

```python
import multiprocessing
multiprocessing.set_start_method("fork")
```

--------------------------------

### Install and Run Pre-commit Hooks

Source: https://github.com/aqarios/luna-bench/blob/main/README.md

Install pre-commit hooks to automate linting, formatting, type checking, and testing on each commit. Run 'pre-commit run . --all-files' to apply these checks to all files in the repository.

```bash
# Install pre-commit hooks
pre-commit run . --all-files
```

--------------------------------

### Feature/Bug Fix/CI Branch Creation

Source: https://github.com/aqarios/luna-bench/blob/main/CONTRIBUTING.md

Use this command to start a new feature, bug fix, or CI branch from the main branch.

```bash
git checkout -b f/y_xxxx  main
```

--------------------------------

### Combined Metrics and Features Plot Example

Source: https://github.com/aqarios/luna-bench/wiki/Plots

Implement a plot that visualizes both metric and feature data. The `@plot` decorator specifies IDs for both, and the `run` method receives a `FeaturesAndMetricsValidationResult`.

```python
from luna_bench.components.plots.generics.features_metrics_plot import (
    GenericFeaturesMetricsPlot,
    FeaturesAndMetricsValidationResult,
)
from luna_bench.helpers import plot


@plot(
    metrics_ids=("my_module.MyMetric",),
    features_ids=("my_module.MyFeature",),
)
class MyCombinedPlot(GenericFeaturesMetricsPlot):
    def run(self, data: FeaturesAndMetricsValidationResult) -> None:
        metric_entity = data.metrics["my_module.MyMetric"]
        feature_entity = data.features["my_module.MyFeature"]
        # combine metric and feature data in your visualization
```

--------------------------------

### Create, Load, and List Benchmarks

Source: https://github.com/aqarios/luna-bench/wiki/Benchmark

Demonstrates how to create a new benchmark, load an existing one by name, and retrieve a list of all available benchmarks. Benchmarks are persisted in SQLite and survive across sessions.

```python
from luna_bench.components import Benchmark

# Create a new benchmark (persisted to SQLite)
bench = Benchmark.create("my_benchmark")

# Load an existing benchmark by name
bench = Benchmark.load("my_benchmark")

# List all benchmarks
all_benchmarks = Benchmark.load_all()
```

--------------------------------

### Run a Benchmark with Solvers and Metrics

Source: https://github.com/aqarios/luna-bench/blob/main/README.md

Set up and run a benchmark using Luna-Bench. This involves creating a Benchmark instance, setting a model set, adding algorithms (solvers), features, metrics, and plots, and then executing the benchmark.

```python
from luna_bench import Benchmark
from luna_bench.algorithms import ScipAlgorithm
from luna_bench.features import OptSolFeature
from luna_bench.metrics import ApproximationRatio
from luna_bench.plots import AverageFeasibilityRatioPlot
from luna_quantum.algorithms import FlexQAOA

benchmark = Benchmark.create("my_benchmark")
benchmark.set_modelset(modelset)

# Add a solver
benchmark.add_algorithm("scip", ScipAlgorithm(max_runtime=60))

# Add any luna_quantum algorithm directly
benchmark.add_algorithm("flexqaoa", FlexQAOA())

# Add a feature that computes the optimal solution (used by metrics)
benchmark.add_feature("optimal_solution", OptSolFeature())

# Add a metric to evaluate solution quality
benchmark.add_metric("approx_ratio", ApproximationRatio())

# Add a plot to visualize metric results
benchmark.add_plot("approx_plot", AverageFeasibilityRatioPlot())

# Run everything: features, algorithms, metrics, plots
benchmark.run()
```

--------------------------------

### Configure Benchmark Components

Source: https://github.com/aqarios/luna-bench/wiki/Benchmark

Shows how to configure a benchmark by setting a model set and adding features, metrics, and algorithms. Each component can be added with a unique name. Components can also be removed by name or entity.

```python
from luna_bench.components import Benchmark, ModelSet
from luna_bench.components.algorithms import ScipAlgorithm
from luna_bench.components.metrics import Runtime, FeasibilityRatio
from luna_bench.components.features import VarNumberFeature
bench = Benchmark.create("my_benchmark")
bench.set_modelset(model_set)

bench.add_feature(name="var_num", feature=VarNumberFeature())
bench.add_metric(name="runtime", metric=Runtime())
bench.add_metric(name="feasibility", metric=FeasibilityRatio())
bench.add_algorithm(name="scip", algorithm=ScipAlgorithm(max_runtime=30))
```

```python
bench.remove_metric("runtime")
# or
entity = bench.add_metric(name="runtime", metric=Runtime())
bench.remove_metric(entity)
```

--------------------------------

### Metrics-only Plot Example

Source: https://github.com/aqarios/luna-bench/wiki/Plots

Define a plot that requires only metric results. The `@plot` decorator specifies the metric IDs, and the `run` method receives a `MetricsValidationResult` object.

```python
from luna_bench.components.plots.generics.metrics_plot import GenericMetricsPlot, MetricsValidationResult
from luna_bench.helpers import plot


@plot(metrics_ids=("my_module.MyMetric",))
class MyMetricPlot(GenericMetricsPlot):
    def run(self, data: MetricsValidationResult) -> None:
        metric_entity = data.metrics["my_module.MyMetric"]
        # metric_entity contains the metric results for all (algorithm, model) pairs
        # generate your plot with matplotlib/seaborn
```

--------------------------------

### Create and Add Models to a ModelSet

Source: https://github.com/aqarios/luna-bench/wiki/ModelSet

Use `ModelSet.create()` to initialize an empty model set and `.add()` to populate it with models loaded from files. Ensure models are deserialized correctly before adding.

```python
from luna_quantum import Model
from luna_bench.components import ModelSet
import glob

model_set = ModelSet.create("my_tsp_models")

model_files = glob.glob("models/*") # directory of serialized Model objects
for path in model_files:
    with open(path, "rb") as f:
        model = Model.deserialize(f.read())
    model_set.add(model)
```

--------------------------------

### Features-only Plot Example

Source: https://github.com/aqarios/luna-bench/wiki/Plots

Create a plot that depends solely on feature results. The `@plot` decorator lists the required feature IDs, and the `run` method processes a `FeaturesValidationResult`.

```python
from luna_bench.components.plots.generics.features_plot import GenericFeaturesPlot, FeaturesValidationResult
from luna_bench.helpers import plot


@plot(features_ids=("my_module.MyFeature",))
class MyFeaturePlot(GenericFeaturesPlot):
    def run(self, data: FeaturesValidationResult) -> None:
        feature_entity = data.features["my_module.MyFeature"]
        # feature_entity contains the feature results for all models
```

--------------------------------

### Run Test Suite

Source: https://github.com/aqarios/luna-bench/wiki/Contributing

Execute the project's test suite using pytest via uv.

```bash
uv run pytest tests
```

--------------------------------

### Run Benchmark Stages Individually

Source: https://github.com/aqarios/luna-bench/wiki/Benchmark

Illustrates how to execute specific stages of the benchmark pipeline independently. This allows for granular control over the execution flow, such as running only features or algorithms.

```python
bench.run_features()
bench.run_algorithms()
bench.run_metrics()
bench.run_plots()
```

--------------------------------

### Access Model Metadata and Full Models

Source: https://github.com/aqarios/luna-bench/wiki/ModelSet

Iterate through `model_set.models` to get `ModelMetadata` objects, which contain `id`, `name`, and `hash`. Access the full `Model` object via the `.model` property, which performs lazy loading from the database.

```python
for meta in model_set.models:
    print(meta.name, meta.hash)
    full_model = meta.model  # fetches from DB on access
```

--------------------------------

### Full CI Check

Source: https://github.com/aqarios/luna-bench/wiki/Contributing

Run all continuous integration checks, including tests, linting, and type checking, using uv.

```bash
uv run pytest tests && uv run ruff check . && uv run mypy luna_bench
```

--------------------------------

### Benchmark Run Pipeline Stages

Source: https://github.com/aqarios/luna-bench/wiki/Architecture

Illustrates the sequential execution of the four main stages in a benchmark run: Features, Algorithms, Metrics, and Plots. Each stage can also be invoked individually.

```mermaid
flowchart LR
    F["run_features()"] --> A["run_algorithms()"] --> M["run_metrics()"] --> P["run_plots()"]
```

--------------------------------

### Implement a Custom Algorithm

Source: https://github.com/aqarios/luna-bench/blob/main/README.md

Create a custom optimization algorithm by subclassing BaseAlgorithmSync and registering it with the @algorithm decorator. Implement the run method with your solver logic.

```python
from luna_bench.base_components import BaseAlgorithmSync
from luna_bench.helpers import algorithm
from luna_model import Model, Solution

@algorithm()
class MyAlgorithm(BaseAlgorithmSync):
    max_iterations: int = 1000

    def run(self, model: Model) -> Solution:
        # Your solver logic here
        ...
```

--------------------------------

### Using ScipAlgorithm

Source: https://context7.com/aqarios/luna-bench/llms.txt

The ScipAlgorithm is a built-in solver that wraps the SCIP optimization solver. It translates Luna models, solves them, and returns the optimal solution. It can be configured with a maximum runtime.

```APIDOC
## ScipAlgorithm — Built-in classical exact solver

`ScipAlgorithm` wraps the SCIP mixed-integer programming solver. It translates `luna_model.Model` objects to LP format, solves them, and returns the optimal solution.

```python
from luna_bench.components.algorithms import ScipAlgorithm
from luna_bench.components import Benchmark

bench = Benchmark.create("scip_demo")
# Set time limit; default is 3600s (1 hour). Set None for no limit.
bench.add_algorithm("scip_fast", ScipAlgorithm(max_runtime=30, quiet_output=True))
bench.add_algorithm("scip_thorough", ScipAlgorithm(max_runtime=300))

# ScipAlgorithm raises InfeasibleModelError if the model has no feasible solution:
from luna_bench.components.algorithms.scip import InfeasibleModelError
try:
    result = ScipAlgorithm().run(infeasible_model)
except InfeasibleModelError:
    print("Model is infeasible — no solution exists.")
```
```

--------------------------------

### Add Luna-Quantum Algorithms

Source: https://github.com/aqarios/luna-bench/wiki/Algorithms

Integrate algorithms directly from the `luna-quantum` library by passing instances to `bench.add_algorithm()`. They are automatically wrapped.

```python
from luna_quantum.algorithms import SimulatedAnnealing

bench.add_algorithm(name="sa", algorithm=SimulatedAnnealing())
```

--------------------------------

### Registering a Custom Asynchronous Algorithm

Source: https://context7.com/aqarios/luna-bench/llms.txt

For cloud or quantum backends, subclass BaseAlgorithmAsync and use the @algorithm decorator. Implement `run_async` to submit jobs and `fetch_result` to retrieve solutions.

```APIDOC
## @algorithm decorator — Register an asynchronous algorithm (cloud/quantum backends)

For backends where you submit a job and poll for results later (e.g., quantum hardware), subclass `BaseAlgorithmAsync[T]` where `T` is the retrieval data type.

```python
from luna_model import Model, Solution
from returns.result import Failure, Result, Success

from luna_bench.base_components import BaseAlgorithmAsync
from luna_bench.helpers import algorithm


@algorithm()
class MyCloudSolver(BaseAlgorithmAsync[str]):
    api_endpoint: str = "https://my-cloud-solver.example.com"
    num_shots: int = 1000

    @property
    def model_type(self) -> type[str]:
        return str  # T = str (job ID is the retrieval token)

    def run_async(self, model: Model) -> str:
        # Submit job to cloud; return job ID for later retrieval
        # (In practice: call your cloud API here)
        job_id = f"job-{model.name}-{self.num_shots}"
        return job_id

    def fetch_result(self, model: Model, retrieval_data: str) -> Result[Solution, str]:
        # Poll the cloud API using the job_id
        # Return Success(solution) or Failure("error message")
        try:
            solution = Solution.from_dict(data={}, model=model)
            return Success(solution)
        except Exception as e:
            return Failure(str(e))

bench.add_algorithm("cloud", MyCloudSolver(num_shots=2000))
```
```

--------------------------------

### Run All Benchmark Stages

Source: https://github.com/aqarios/luna-bench/wiki/Benchmark

Executes all stages of the benchmark in the predefined sequence: features, algorithms, metrics, and plots. This is a convenient way to run a complete benchmark cycle.

```python
bench.run()
```

--------------------------------

### Compute Fraction of Overall Best Solution

Source: https://context7.com/aqarios/luna-bench/llms.txt

Calculates the fraction of samples that match the best known solution. Requires OptSolFeature.

```python
from luna_bench.components.metrics import FractionOfOverallBestSolution
from luna_bench.components.features import OptSolFeature
from luna_bench.components import Benchmark

bench = Benchmark.create("fob_demo")
bench.set_modelset(model_set)
bench.add_feature("opt", OptSolFeature())
bench.add_algorithm("solver", ScipAlgorithm())
bench.add_metric("fob", FractionOfOverallBestSolution(abs_tol=1e-6))
bench.run()

df = bench.all_metrics_to_dataframe()
print(df["fob/fraction_of_overall_best_solution"])
```

--------------------------------

### Define a Simple Optimization Model

Source: https://github.com/aqarios/luna-bench/blob/main/README.md

Create an optimization model with variables, an objective function, and constraints using luna_model. This model can then be added to a ModelSet.

```python
from luna_model import Model, Variable
from luna_bench.components import ModelSet

# Build a simple optimization model
model = Model("example")
with model.environment:
    x = Variable("x")
    y = Variable("y")
model.objective = x * y + x
model.constraints += x >= 0
model.constraints += y <= 5

# Group models into a set
modelset = ModelSet.create("my_models")
modelset.add(model)
```

--------------------------------

### Use ScipAlgorithm for Exact Solutions

Source: https://context7.com/aqarios/luna-bench/llms.txt

Wraps the SCIP solver to translate luna_model.Model objects, solve them, and return optimal solutions. Handles infeasible models by raising InfeasibleModelError.

```python
from luna_bench.components.algorithms import ScipAlgorithm
from luna_bench.components import Benchmark

bench = Benchmark.create("scip_demo")
# Set time limit; default is 3600s (1 hour). Set None for no limit.
bench.add_algorithm("scip_fast", ScipAlgorithm(max_runtime=30, quiet_output=True))
bench.add_algorithm("scip_thorough", ScipAlgorithm(max_runtime=300))

# ScipAlgorithm raises InfeasibleModelError if the model has no feasible solution:
from luna_bench.components.algorithms.scip import InfeasibleModelError
try:
    result = ScipAlgorithm().run(infeasible_model)
except InfeasibleModelError:
    print("Model is infeasible — no solution exists.")
```

--------------------------------

### Dependency Injection Container Relationships

Source: https://github.com/aqarios/luna-bench/wiki/Architecture

Illustrates the relationships between the various dependency injection containers in Luna-Bench. It shows how containers like `RegistryContainer`, `DaoContainer`, and `MapperContainer` provide dependencies to `UsecaseContainer` and `BackgroundTaskContainer`.

```mermaid
graph LR
    RC[RegistryContainer] --> MC[MapperContainer]
    MC --> UC[UsecaseContainer]
    DC[DaoContainer] --> UC
    BTC[BackgroundTaskContainer] --> UC
    UC --> Comp[components/]
```

--------------------------------

### Creating a ModelSet

Source: https://github.com/aqarios/luna-bench/wiki/ModelSet

Use `ModelSet.create()` to initialize a new, empty model set with a given name. Models can then be added using the `.add()` method. A `RuntimeError` is raised if a model set with the same name already exists.

```APIDOC
## ModelSet.create(name: str)

### Description
Creates a new, empty model set with the specified name.

### Method
`ModelSet.create()`

### Parameters
#### Path Parameters
- **name** (str) - Required - The unique name for the new model set.

### Raises
- `RuntimeError`: If a model set with the given name already exists.

## ModelSet.add(model: Model)

### Description
Adds a `Model` object to the model set. If a model with the same hash already exists, the addition is a no-op.

### Method
`model_set.add()`

### Parameters
#### Path Parameters
- **model** (Model) - Required - The `Model` object to add to the set.
```

--------------------------------

### Luna-Bench Decorators for Component Registration

Source: https://context7.com/aqarios/luna-bench/llms.txt

Use decorators like `@feature`, `@metric`, and `@plot` to register components with the Luna-Bench registry. These decorators automatically inject components, and can be used with or without arguments to specify IDs or dependencies.

```python
from luna_bench.helpers import algorithm, feature, metric, plot

# Use as a plain decorator (no arguments needed for most cases)
@feature
class MyFeature(BaseFeature): ...

# Or as a factory with an explicit ID
@feature(feature_id="my_project.v2.MyFeature")
class MyFeatureV2(BaseFeature): ...

# Metrics declare required features as a class (not instance)
@metric(required_features=MyFeatureV2)
class MyMetric(BaseMetric): ...

# Multiple required features
@metric(required_features=[FeatureA, FeatureB])
class MultiFeatureMetric(BaseMetric): ...

# Plots declare required metric and feature IDs (registered_id strings)
@plot(
    metrics_ids=(MyMetric.registered_id,),
    features_ids=(MyFeatureV2.registered_id,),
)
class MyPlot(GenericFeaturesMetricsPlot): ...

# Inspect what is registered
from luna_bench.helpers import features, metrics, algorithms_sync, plots
print(features().ids())        # all registered feature IDs
print(algorithms_sync().ids()) # all registered sync algorithm IDs
```

--------------------------------

### Call Flow Through Architectural Layers

Source: https://github.com/aqarios/luna-bench/wiki/Architecture

Visualizes the sequence of interactions between different architectural layers for a user action, specifically demonstrating the `bench.run_features()` call. It highlights the path from user code through components, use cases, DAOs, and the database.

```mermaid
sequenceDiagram
    participant U as User Code
    participant C as Component<br/>(Benchmark)
    participant UC as Use Case
    participant DAO as DAO + Transaction
    participant DB as SQLite

    U->>C: bench.run_features()
    C->>UC: feature_run_uc(benchmark_entity)
    UC->>DAO: transaction.model.load(model_id)
    DAO->>DB: SELECT from model_table
    DB-->>DAO: model bytes
    DAO-->>UC: Model
    UC->>UC: feature.run(model) → result
    UC->>DAO: transaction.feature.set_result(...)
    DAO->>DB: INSERT into feature_result_table
    UC-->>C: Result[None, Error]
    C->>C: Unwrap result or raise
    C-->>U: returns
```

--------------------------------

### Compute Optimal Solution Value with OptSolFeature

Source: https://context7.com/aqarios/luna-bench/llms.txt

The OptSolFeature computes the optimal solution value using SCIP, providing bounds or proven optimality. It's required by several metrics and can be configured with a maximum runtime.

```python
from luna_bench.components.features import OptSolFeature
from luna_bench.components.features.optsol_feature import OptSolFeatureResult, InfeasibleModelError

bench.add_feature("optimal", OptSolFeature())              # solve to proven optimality
bench.add_feature("optimal_60s", OptSolFeature(max_runtime=60))  # upper/lower bound in 60s

# Direct usage (outside benchmark pipeline):
feature = OptSolFeature(max_runtime=30)
try:
    result: OptSolFeatureResult = feature.run(model)
    if result.pre_terminated:
        print(f"Best bound found in time limit: {result.best_sol}")
    else:
        print(f"Proven optimal: {result.best_sol} (solved in {result.runtime:.2f}s)")
except InfeasibleModelError:
    print("No feasible solution exists.")
```

--------------------------------

### Add Built-in Features to Benchmark

Source: https://github.com/aqarios/luna-bench/wiki/Features

Use `bench.add_feature` to include predefined features like `VarNumberFeature` and `OptSolFeature`. `OptSolFeature` can be configured with a `max_runtime`.

```python
from luna_bench.components.features import VarNumberFeature, OptSolFeature

bench.add_feature(name="var_num", feature=VarNumberFeature())
bench.add_feature(name="opt_sol", feature=OptSolFeature(max_runtime=60))
```

--------------------------------

### Create and Manage Model Sets

Source: https://context7.com/aqarios/luna-bench/llms.txt

Define and persist collections of optimization models using ModelSet. Models are stored in SQLite and can be loaded across sessions.

```python
from luna_model import Model, Variable
from luna_bench.components import ModelSet

# Build a simple optimization model
model = Model("knapsack_5")
with model.environment:
    x = [Variable(f"x{i}") for i in range(5)]
# (set objective and constraints on model here)

# Create a new model set and populate it
model_set = ModelSet.create("knapsack_instances")
model_set.add(model)

# Load across sessions
model_set = ModelSet.load("knapsack_instances")

# List all model sets in the database
all_sets = ModelSet.load_all()
print([s.name for s in all_sets])  # ['knapsack_instances']

# Inspect models (lazy – returns ModelMetadata, not full Model objects)
for meta in model_set.models:
    print(meta.name, meta.hash)
    full_model = meta.model  # fetches full Model from DB on demand

# Load metadata for ALL models across ALL model sets
all_models = ModelSet.load_all_models()  # list[ModelMetadata]

# Remove a model or delete the entire set
model_set.remove_model(model)
model_set.delete()
```

--------------------------------

### Implement Sync Algorithm

Source: https://github.com/aqarios/luna-bench/wiki/Algorithms

Subclass `BaseAlgorithmSync` and implement the `run` method to create a synchronous algorithm. The `@algorithm` decorator automatically registers the class.

```python
from luna_quantum import Model, Solution
from luna_bench.base_components import BaseAlgorithmSync
from luna_bench.helpers import algorithm

@algorithm
class MySolver(BaseAlgorithmSync):
    max_iterations: int = 1000

    def run(self, model: Model) -> Solution:
        # solve the model
        return solution
```

--------------------------------

### Release Branch Creation

Source: https://github.com/aqarios/luna-bench/blob/main/CONTRIBUTING.md

Use this command to create a new release branch from the main branch. Delete any existing release branch first if necessary.

```bash
git checkout -b release  main
```

--------------------------------

### Orchestrate Benchmarking Pipeline

Source: https://context7.com/aqarios/luna-bench/llms.txt

Use Benchmark to link ModelSets with algorithms, features, metrics, and plots. Results are persisted to SQLite.

```python
from luna_bench.components import Benchmark, ModelSet
from luna_bench.components.algorithms import ScipAlgorithm
from luna_bench.components.features import OptSolFeature, VarNumberFeature
from luna_bench.components.metrics import ApproximationRatio, FeasibilityRatio, Runtime
from luna_bench.components.plots import AverageFeasibilityRatioPlot, AverageApproximationRatioPlot

model_set = ModelSet.load("knapsack_instances")

# Create or load an existing benchmark
bench = Benchmark.create("knapsack_comparison")
# bench = Benchmark.load("knapsack_comparison")   # reload across sessions
# bench = Benchmark.open("knapsack_comparison")   # create-or-load

bench.set_modelset(model_set)

# Register components
bench.add_feature("var_count", VarNumberFeature())
bench.add_feature("optimal", OptSolFeature(max_runtime=60))

bench.add_algorithm("scip_30s", ScipAlgorithm(max_runtime=30))
bench.add_algorithm("scip_300s", ScipAlgorithm(max_runtime=300))

bench.add_metric("runtime", Runtime())
bench.add_metric("feasibility", FeasibilityRatio())
bench.add_metric("approx_ratio", ApproximationRatio())

bench.add_plot("feasibility_plot", AverageFeasibilityRatioPlot())
bench.add_plot("approx_plot", AverageApproximationRatioPlot())

# Run all stages in order: features → algorithms → metrics → plots
bench.run()

# Or run stages individually
bench.run_features()
bench.run_algorithms()
bench.run_metrics()
bench.run_plots()

# Remove a component by name or entity
bench.remove_metric("runtime")

# List all registered benchmarks
all_benches = Benchmark.load_all()
```

--------------------------------

### Code Quality Checks

Source: https://github.com/aqarios/luna-bench/wiki/Contributing

Perform code formatting, linting, and type checking using ruff and mypy via uv.

```bash
uv run ruff check .
uv run ruff format .
uv run mypy luna_bench
```

--------------------------------

### Estimate Time to Solution with Target Probability

Source: https://context7.com/aqarios/luna-bench/llms.txt

Estimates the time to find an optimal solution with a specified target probability. Requires OptSolFeature.

```python
from luna_bench.components.metrics import TimeToSolution
from luna_bench.components.metrics.time_to_solution import TimeToSolutionResult
from luna_bench.components.features import OptSolFeature
from luna_bench.components import Benchmark

bench = Benchmark.create("tts_demo")
bench.set_modelset(model_set)
bench.add_feature("opt", OptSolFeature())
bench.add_algorithm("solver", ScipAlgorithm())
bench.add_metric("tts", TimeToSolution(target_probability=0.99, abs_tol=1e-6))
bench.run()

df = bench.all_metrics_to_dataframe()
# Columns: tts/time_to_solution, tts/probability_optimal,
#          tts/num_optimal_found, tts/num_samples
print(df[["algorithm", "model", "tts/time_to_solution", "tts/probability_optimal"]])
```

--------------------------------

### Git Graph Visualization of Branching Strategy

Source: https://github.com/aqarios/luna-bench/blob/main/CONTRIBUTING.md

Visual representation of the project's branching workflow, including feature, bug fix, CI, release, and hotfix branches.

```mermaid
gitGraph
commit
branch fix/y_xxxx
branch f/y_xxxx
branch ci/y_xxxx
checkout fix/y_xxxx
commit
checkout main
merge fix/y_xxxx
checkout f/y_xxxx
commit
checkout main
merge f/y_xxxx
checkout ci/y_xxxx
commit
checkout main
merge ci/y_xxxx
```

```mermaid
gitGraph
commit
branch release
checkout main
commit
checkout release
commit id: "bump v1.1.0" tag: "v1.1.0"
checkout main
merge release
```

```mermaid
gitGraph
commit
branch hot/y_xxxx
checkout main
commit
checkout hot/y_xxxx
commit id: "fix something"
commit id: "bump v1.1.1" tag: "v1.1.1"
checkout main
merge hot/y_xxxx
```

--------------------------------

### Add Built-in Algorithms

Source: https://github.com/aqarios/luna-bench/wiki/Algorithms

Register built-in algorithms like `ScipAlgorithm` and `FakeAlgorithm` with custom names and configurations using `bench.add_algorithm()`.

```python
from luna_bench.components.algorithms import ScipAlgorithm, FakeAlgorithm

bench.add_algorithm(name="scip", algorithm=ScipAlgorithm(max_runtime=30))
bench.add_algorithm(name="fake", algorithm=FakeAlgorithm(time_to_sleep=1.0))
```

--------------------------------

### Instantiate Custom Metric with Non-Default Parameters

Source: https://github.com/aqarios/luna-bench/wiki/Metrics

Pass non-default values when instantiating a custom metric. This allows for dynamic configuration of metric behavior.

```python
bench.add_metric(name="my_metric", metric=MyMetric(threshold=0.8))
```

--------------------------------

### Database Schema ER Diagram

Source: https://github.com/aqarios/luna-bench/wiki/Architecture

This diagram illustrates the database schema used for storing benchmark configurations and results, primarily utilizing SQLite and Peewee ORM.

```mermaid
erDiagram
    BenchmarkTable ||--o| ModelSetTable : "has modelset"
    BenchmarkTable ||--o{ FeatureTable : "has features"
    BenchmarkTable ||--o{ AlgorithmTable : "has algorithms"
    BenchmarkTable ||--o{ MetricTable : "has metrics"
    BenchmarkTable ||--o{ PlotConfigTable : "has plots"
    ModelSetTable }o--o{ ModelMetadataTable : "contains models"
    ModelMetadataTable ||--|| ModelTable : "has binary data"
    FeatureTable ||--o{ FeatureResultTable : "has results"
    FeatureResultTable }o--|| ModelMetadataTable : "per model"
    AlgorithmTable ||--o{ AlgorithmResultTable : "has results"
    MetricTable ||--o{ MetricResultTable : "has results"

    BenchmarkTable {
        int id PK
        string name UK
        string status
    }
    ModelSetTable {
        int id PK
        string name UK
    }
    ModelMetadataTable {
        int id PK
        string name
        string hash UK
    }
    ModelTable {
        int model_id PK,FK
        blob encoded_model
    }
    FeatureTable {
        int id PK
        string name
        string registered_id
        json config_data
        int benchmark_id FK
    }
    FeatureResultTable {
        int id PK
        int processing_time_ms
        string status
        json result_data
        int feature_id FK
        int model_metadata_id FK
    }
    AlgorithmTable {
        int id PK
        string name
        string algorithm_type
        string registered_id
        json config_data
        int benchmark_id FK
    }
    AlgorithmResultTable {
        int id PK
        int processing_time_ms
        string status
        json solution_data
        string model_name
        int algorithm_id FK
    }
    MetricTable {
        int id PK
        string name
        string registered_id
        json config_data
        int benchmark_id FK
    }
    MetricResultTable {
        int id PK
        int processing_time_ms
        string status
        json result_data
        string algorithm_name
        string model_name
        int metric_id FK
    }
```

--------------------------------

### Register Custom Synchronous Algorithm with @algorithm

Source: https://context7.com/aqarios/luna-bench/llms.txt

Subclass BaseAlgorithmSync and use the @algorithm decorator to register a custom synchronous solver. Configuration parameters are defined as Pydantic fields.

```python
from luna_model import Model, Solution
from luna_bench.base_components import BaseAlgorithmSync
from luna_bench.helpers import algorithm

@algorithm()
class GreedyAlgorithm(BaseAlgorithmSync):
    max_iterations: int = 100
    tolerance: float = 1e-4

    def run(self, model: Model) -> Solution:
        # Implement your solver logic
        # Must return a luna_model Solution object
        solution_dict = {var.name: 0.0 for var in model.variables()}
        return Solution.from_dict(
            data=solution_dict,
            model=model,
        )

# Use in a benchmark
from luna_bench.components import Benchmark
bench = Benchmark.create("greedy_test")
bench.add_algorithm("greedy", GreedyAlgorithm(max_iterations=500))
```

--------------------------------

### Implement a Custom Metric in Python

Source: https://github.com/aqarios/luna-bench/wiki/Metrics

Subclass `BaseMetric`, implement the `run` method, and register with the `@metric` decorator. Return a Pydantic model extending `MetricResult`. Configuration parameters are Pydantic fields.

```python
from luna_quantum import Solution
from luna_bench.base_components import BaseMetric
from luna_bench.base_components.data_types.feature_results import FeatureResults
from luna_bench.helpers import metric
from luna_bench.types import MetricResult


class MyMetricResult(MetricResult):
    score: float


@metric
class MyMetric(BaseMetric):
    threshold: float = 0.5  # config parameter

    def run(self, solution: Solution, feature_results: FeatureResults) -> MyMetricResult:
        feasible = sum(1 for s in solution.samples if s.is_feasible)
        return MyMetricResult(score=feasible / len(solution.samples))
```

--------------------------------

### Loading and Reusing ModelSets

Source: https://github.com/aqarios/luna-bench/wiki/ModelSet

Model sets can be loaded from the database by their name using `ModelSet.load()`. `ModelSet.load_all()` retrieves all existing model sets.

```APIDOC
## ModelSet.load(name: str)

### Description
Loads a specific model set from the database by its name.

### Method
`ModelSet.load()`

### Parameters
#### Path Parameters
- **name** (str) - Required - The name of the model set to load.

## ModelSet.load_all()

### Description
Retrieves all model sets currently stored in the database.

### Method
`ModelSet.load_all()`

### Returns
- `list[ModelSet]`: A list of all loaded `ModelSet` objects.
```

--------------------------------

### Register Custom Asynchronous Algorithm with @algorithm

Source: https://context7.com/aqarios/luna-bench/llms.txt

Subclass BaseAlgorithmAsync and use the @algorithm decorator for cloud or quantum backends. Implement run_async to submit jobs and fetch_result to retrieve outcomes.

```python
from luna_model import Model, Solution
from returns.result import Failure, Result, Success

from luna_bench.base_components import BaseAlgorithmAsync
from luna_bench.helpers import algorithm


@algorithm()
class MyCloudSolver(BaseAlgorithmAsync[str]):
    api_endpoint: str = "https://my-cloud-solver.example.com"
    num_shots: int = 1000

    @property
    def model_type(self) -> type[str]:
        return str  # T = str (job ID is the retrieval token)

    def run_async(self, model: Model) -> str:
        # Submit job to cloud; return job ID for later retrieval
        # (In practice: call your cloud API here)
        job_id = f"job-{model.name}-{self.num_shots}"
        return job_id

    def fetch_result(self, model: Model, retrieval_data: str) -> Result[Solution, str]:
        # Poll the cloud API using the job_id
        # Return Success(solution) or Failure("error message")
        try:
            solution = Solution.from_dict(data={}, model=model)
            return Success(solution)
        except Exception as e:
            return Failure(str(e))

bench.add_algorithm("cloud", MyCloudSolver(num_shots=2000))
```

--------------------------------

### Load and List ModelSets

Source: https://github.com/aqarios/luna-bench/wiki/ModelSet

Load a specific model set by its name or retrieve all available model sets from the database. Model sets are persistent and can be reused across different sessions.

```python
# Load a specific model set by name
model_set = ModelSet.load("my_tsp_models")

# List all model sets in the database for the given ModelSet
all_sets = ModelSet.load_all()
```

--------------------------------

### Benchmark Execution Model Flow

Source: https://github.com/aqarios/luna-bench/wiki/Architecture

Details the execution flow and data consumption for each stage of the benchmark pipeline: Features, Algorithms, Metrics, and Plots. This diagram clarifies the relationships and data dependencies between stages.

```mermaid
flowchart TD
    subgraph "Feature Stage"
        direction LR
        F1["Feature A"] --> M1["Model 1"]
        F1 --> M2["Model 2"]
        F1 --> M3["Model 3"]
    end

    subgraph "Algorithm Stage"
        direction LR
        A1["Algo X × Model 1"]
        A2["Algo X × Model 2"]
        A3["Algo X × Model 3"]
        A4["Algo Y × Model 1"]
        A5["Algo Y × Model 2"]
        A6["Algo Y × Model 3"]
    end

    subgraph "Metric Stage"
        direction LR
        MR["Metric per (algo, model) pair\n→ uses Solution + FeatureResults"]
    end

    subgraph "Plot Stage"
        direction LR
        PL["Plots consume all\nmetric + feature results"]
    end

    F1 --> A1
    F1 --> A4
    A1 --> MR
    A2 --> MR
    A3 --> MR
    A4 --> MR
    A5 --> MR
    A6 --> MR
    MR --> PL
```

--------------------------------

### Export Benchmark Results to DataFrame

Source: https://context7.com/aqarios/luna-bench/llms.txt

Convert all algorithm, feature, and metric results into a single pandas DataFrame for analysis.

```python
from luna_bench.components import Benchmark

bench = Benchmark.load("knapsack_comparison")
bench.run()

```

--------------------------------

### Add Built-in Plots to Benchmark

Source: https://context7.com/aqarios/luna-bench/llms.txt

Integrates pre-built plots for visualizing aggregated metric data. Plots run automatically during `bench.run()` or `bench.run_plots()`. Error handling can be configured to skip broken plots.

```python
from luna_bench.components.plots import (
    AverageRuntimePlot,
    AverageFeasibilityRatioPlot,
    AverageApproximationRatioPlot,
    AverageFoBRatioPlot,
    AverageBestSolutionFoundRatioPlot,
)
from luna_bench.components.plots.metrics_plots.per_model_plots import RuntimePerModelPlot
from luna_bench.components import Benchmark

bench = Benchmark.create("plot_demo")
# ... add modelset, features, algorithms, metrics ...

bench.add_plot("avg_runtime", AverageRuntimePlot())          # avg runtime per algorithm
bench.add_plot("avg_feas", AverageFeasibilityRatioPlot())    # avg feasibility per algorithm
bench.add_plot("avg_ar", AverageApproximationRatioPlot())    # avg approx ratio per algorithm
bench.add_plot("rt_per_model", RuntimePerModelPlot())        # runtime grouped by model

# Error handling: skip broken plots instead of failing
from luna_bench.entities.enums import ErrorHandlingMode
bench.run_plots(error_handling_mode=ErrorHandlingMode.CONTINUE_ON_ERROR)
```

--------------------------------

### Implement Async Algorithm

Source: https://github.com/aqarios/luna-bench/wiki/Algorithms

Subclass `BaseAlgorithmAsync[T]` for asynchronous operations, implementing `run_async` to submit jobs and `fetch_result` to retrieve solutions. The `model_type` property specifies the retrieval data type.

```python
from luna_quantum import Model, Solution
from returns.result import Result
from luna_bench.base_components import BaseAlgorithmAsync
from luna_bench.helpers import algorithm

@algorithm
class MyCloudSolver(BaseAlgorithmAsync[str]):
    @property
    def model_type(self) -> type[str]:
        return str

    def run_async(self, model: Model) -> str:
        # submit job, return a job ID or retrieval token
        return job_id

    def fetch_result(self, model: Model, retrieval_data: str) -> Result[Solution, str]:
        # poll for results using the retrieval data
        return Success(solution)
```

--------------------------------

### Listing All Models Across All Sets

Source: https://github.com/aqarios/luna-bench/wiki/ModelSet

The `ModelSet.load_all_models()` method provides metadata for every model stored across all model sets in the database.

```APIDOC
## ModelSet.load_all_models()

### Description
Retrieves metadata for all models present in all model sets within the database.

### Method
`ModelSet.load_all_models()`

### Returns
- `list[ModelMetadata]`: A list containing `ModelMetadata` for every model across all model sets.
```

--------------------------------

### Register Components with Decorators

Source: https://github.com/aqarios/luna-bench/wiki/Architecture

Use decorators like `@feature`, `@metric`, `@algorithm`, and `@plot` to register custom components. These decorators automatically generate a `registered_id`, set it as a `ClassVar`, and store the class in the appropriate registry. Ensure required features or metrics are specified when necessary.

```python
from luna_bench.helpers import feature, metric, algorithm, plot

@feature
class MyFeature(BaseFeature):
    def run(self, model: Model) -> MyResult: ...

@metric(required_features=MyFeature)
class MyMetric(BaseMetric):
    def run(self, solution: Solution, feature_results: FeatureResults) -> MyMetricResult: ...

@algorithm
class MySolver(BaseAlgorithmSync):
    def run(self, model: Model) -> Solution: ...

@plot(metrics_ids=("my_module.MyMetric",))
class MyPlot(GenericMetricsPlot):
    def run(self, data: MetricsValidationResult) -> None: ...

```

--------------------------------

### Convert Benchmark Results to DataFrame

Source: https://context7.com/aqarios/luna-bench/llms.txt

Converts benchmark results into a pandas DataFrame for analysis. Includes options to include raw solution objects.

```python
df = bench.results_to_dataframe()
print(df.columns.tolist())
# ['algorithm', 'model', 'runtime/runtime_seconds',
#  'feasibility/feasibility_ratio', 'approx_ratio/approximation_ratio',
#  'var_count/var_number', 'optimal/best_sol', 'optimal/pre_terminated', ...]

print(df[["algorithm", "model", "approx_ratio/approximation_ratio"]])
#            algorithm       model  approx_ratio/approximation_ratio
# 0         scip_30s  knapsack_5                               1.0
# 1        scip_300s  knapsack_5                               1.0

# Include raw Solution objects (off by default to keep DataFrame light)
df_with_solutions = bench.results_to_dataframe(inlcude_solution=True)

# Per-component DataFrames
algo_df = bench.algorithms_to_dataframe()
metrics_df = bench.all_metrics_to_dataframe()
features_df = bench.all_features_to_dataframe()
```

--------------------------------

### Run Problem Size Feature and Access Boolean Variable Stats

Source: https://github.com/aqarios/luna-bench/wiki/Features

This snippet demonstrates running the `ProblemSizeFeatures` and accessing specific statistics, such as the count and fraction of boolean variables, using typed access via `.get()`.

```python
from luna_bench.components.features.mip.problem_size_feature import (
    ProblemSizeFeatures, VarTypeKey, VarType,
)

result = ProblemSizeFeatures().run(model)
boolean_stats = result.var_counts.get(VarTypeKey(var_type=VarType.BOOLEAN))
print(f"Boolean vars: {boolean_stats.count} ({boolean_stats.fraction:.1%})")
```

--------------------------------

### Compute Feasibility Ratio

Source: https://context7.com/aqarios/luna-bench/llms.txt

Measures the proportion of samples that satisfy all model constraints. Ranges from 0.0 (none feasible) to 1.0 (all feasible). Does not require any features.

```python
from luna_bench.components.metrics import FeasibilityRatio
from luna_bench.components import Benchmark

bench = Benchmark.create("feasibility_demo")
bench.set_modelset(model_set)
bench.add_algorithm("scip", ScipAlgorithm(max_runtime=30))
bench.add_metric("feas", FeasibilityRatio())
bench.run()

df = bench.all_metrics_to_dataframe()
print(df[["algorithm", "model", "feas/feasibility_ratio"]])
# algorithm       model  feas/feasibility_ratio
# scip      problem_01                     1.0
# scip      problem_02                     0.75
```

--------------------------------

### Write a Custom Plot from Scratch

Source: https://context7.com/aqarios/luna-bench/llms.txt

Subclass `GenericMetricsPlot` and use the `@plot` decorator to register a custom plot. The framework validates required data before calling the `run` method, which should contain your plotting logic.

```python
import matplotlib.pyplot as plt
import seaborn as sns
from luna_bench.components.plots.generics.metrics_plot import GenericMetricsPlot, MetricsValidationResult
from luna_bench.components.plots.utils.dataframe_conversion import metric_to_dataframe
from luna_bench.components.metrics.runtime import Runtime, RuntimeResult
from luna_bench.helpers import plot


@plot(metrics_ids=(Runtime.registered_id,))
class ViolinRuntimePlot(GenericMetricsPlot):
    def run(self, data: MetricsValidationResult) -> None:
        df = metric_to_dataframe(
            data.metrics[Runtime.registered_id],
            RuntimeResult,
            "runtime_seconds",
        )
        if df.empty:
            return
        plt.figure(figsize=(10, 6))
        sns.violinplot(data=df, x="algorithm", y="runtime_seconds")
        plt.title("Runtime Distribution per Algorithm")
        plt.tight_layout()
        plt.show()


from luna_bench.components import Benchmark
bench = Benchmark.create("violin_demo")
bench.add_metric("runtime", Runtime())
bench.add_plot("violin", ViolinRuntimePlot())
bench.run()
```

--------------------------------

### Registering a Custom Synchronous Algorithm

Source: https://context7.com/aqarios/luna-bench/llms.txt

Subclass BaseAlgorithmSync and use the @algorithm decorator to register a custom synchronous algorithm. Configuration parameters can be defined as Pydantic fields.

```APIDOC
## @algorithm decorator — Register a custom synchronous algorithm

Subclass `BaseAlgorithmSync`, implement `run(model) -> Solution`, and register with `@algorithm`. Configuration parameters are declared as Pydantic fields.

```python
from luna_model import Model, Solution
from luna_bench.base_components import BaseAlgorithmSync
from luna_bench.helpers import algorithm

@algorithm()
class GreedyAlgorithm(BaseAlgorithmSync):
    max_iterations: int = 100
    tolerance: float = 1e-4

    def run(self, model: Model) -> Solution:
        # Implement your solver logic
        # Must return a luna_model Solution object
        solution_dict = {var.name: 0.0 for var in model.variables()}
        return Solution.from_dict(
            data=solution_dict,
            model=model,
        )

# Use in a benchmark
from luna_bench.components import Benchmark
bench = Benchmark.create("greedy_test")
bench.add_algorithm("greedy", GreedyAlgorithm(max_iterations=500))
```
```

--------------------------------

### Define a Custom Metric in Python

Source: https://github.com/aqarios/luna-bench/blob/main/README.md

Create a custom metric by inheriting from BaseMetric and implementing the run method. Ensure your result class inherits from MetricResult. Use the @metric() decorator to register the metric.

```python
from luna_bench.base_components import BaseMetric
from luna_bench.base_components.data_types.feature_results import FeatureResults
from luna_bench.helpers import metric
from luna_bench.types import MetricResult
from luna_model import Solution

class MyMetricResult(MetricResult):
    score: float

@metric()
class MyMetric(BaseMetric):
    def run(self, solution: Solution, feature_results: FeatureResults) -> MyMetricResult:
        score = solution.expectation_value()
        return MyMetricResult(score=score)
```