### Example Install and Run Steps

Source: https://github.com/autogluon/tabarena/blob/main/README.md

A step-by-step guide to setting up a project, installing dependencies including AutoGluon and TabArena, and running a quickstart example script. This is useful for quickly getting started with TabArena.

```bash
pip install uv
uv init -p 3.12
uv sync
git clone https://github.com/autogluon/autogluon.git
./autogluon/full_install.sh
git clone https://github.com/autogluon/tabarena.git
cd tabarena
uv pip install --prerelease=allow -e "./tabarena[benchmark]"
cd examples/benchmarking
python run_quickstart_tabarena.py 
```

--------------------------------

### Launch TabRepo HPO Experiments with Syne Tune

Source: https://github.com/autogluon/tabarena/blob/main/tabrepo.md

Example script to get started with launching HPO experiments on TabRepo using Syne Tune's wrapper. This simulates methods like random-search or Bayesian optimization.

```python
examples/launch_tabrepo.py
```

--------------------------------

### Install TabArena for Benchmarking

Source: https://github.com/autogluon/tabarena/blob/main/README.md

Install TabArena with benchmarking capabilities, which includes dependencies required for fitting models. This is necessary if you plan to train or benchmark models.

```bash
uv sync --extra benchmark
```

--------------------------------

### Install TabArena for Evaluation

Source: https://github.com/autogluon/tabarena/blob/main/README.md

Install TabArena with minimal dependencies if you only intend to evaluate models or use the leaderboard/metrics. This command synchronizes the necessary packages.

```bash
uv sync
```

--------------------------------

### Install TabRepo

Source: https://github.com/autogluon/tabarena/blob/main/tabrepo.md

Clone the repository and install the package in editable mode. Requires Python 3.9-3.11.

```bash
git clone https://github.com/autogluon/tabrepo.git
pip install -e tabrepo/
```

--------------------------------

### Reproduce TabArena Benchmarking with LightGBM and RealMLP

Source: https://github.com/autogluon/tabarena/blob/main/examples/README.md

Reproduces running LightGBM and RealMLP on three datasets from TabArena. This script is part of the benchmarking examples.

```python
run_quickstart_tabarena.py
```

--------------------------------

### Run TabArena's RealMLP Model

Source: https://github.com/autogluon/tabarena/blob/main/examples/README.md

A simple example for running TabArena's specific implementation of the RealMLP model. Useful for direct usage of this SOTA model.

```python
run_tabarena_realmlp.py
```

--------------------------------

### Run Tuned and Ensembled TabArena Model

Source: https://github.com/autogluon/tabarena/blob/main/examples/README.md

A minimal example for running a tuned and ensembled TabArena model. This demonstrates using more advanced model configurations.

```python
run_tuned_ensemble_model.py
```

--------------------------------

### Install TabRepo 2.0 Dependencies

Source: https://github.com/autogluon/tabarena/blob/main/tabrepo.md

Install dependencies for TabRepo 2.0, requiring the latest AutoGluon mainline or 1.3+. Includes benchmark extras.

```bash
# Requires latest mainline AutoGluon (or AutoGluon 1.3+)
git clone https://github.com/autogluon/autogluon
./autogluon/full_install.sh

git clone https://github.com/autogluon/tabrepo.git
pip install -e tabrepo/[benchmark]
```

--------------------------------

### Run Default TabArena Model with Cross-Validation Bagging

Source: https://github.com/autogluon/tabarena/blob/main/examples/README.md

A minimal example for running a default TabArena model, including cross-validation bagging. This serves as a basic integration example.

```python
run_default_model.py
```

--------------------------------

### Developer Install with Editable AutoGluon

Source: https://github.com/autogluon/tabarena/blob/main/README.md

Install TabArena in editable mode along with AutoGluon. This setup is for developers who intend to modify both TabArena and AutoGluon codebases. It involves cloning AutoGluon, running its full install script, cloning TabArena, and then installing TabArena with benchmark extras in editable mode.

```bash
git clone https://github.com/autogluon/autogluon.git
./autogluon/full_install.sh

git clone https://github.com/autogluon/tabarena.git
uv pip install --prerelease=allow -e "./tabarena/tabarena[benchmark]"
```

--------------------------------

### Reproduce AutoML Conf 2024 Paper Experiments

Source: https://github.com/autogluon/tabarena/blob/main/tabrepo.md

Install extra dependencies for reproducing paper experiments, including AG benchmark and other Python packages.

```bash
# Install AG benchmark, required only to reproduce results showing win-rate tables

git clone https://github.com/autogluon/autogluon-bench.git
pip install -e autogluon-bench/

git clone https://github.com/Innixma/autogluon-benchmark.git
pip install -e autogluon-benchmark/

# Install extra dependencies used for results scripts
pip install autorank seaborn
```

--------------------------------

### Install TabArena as a Dependency

Source: https://github.com/autogluon/tabarena/blob/main/README.md

Install TabArena directly from GitHub as a dependency in your project using pip. This is useful for incorporating TabArena into other projects without cloning the repository directly.

```bash
"tabarena @ git+https://github.com/autogluon/tabarena.git#subdirectory=tabarena"
```

--------------------------------

### Load and Compare New Results Artifacts using TabArena

Source: https://github.com/autogluon/tabarena/blob/main/examples/README.md

An example of how to load and compare new results artifacts using TabArena. This is useful for analyzing and validating benchmark outcomes.

```python
run_beta_tabpfn_end_to_end.py
```

--------------------------------

### Run Standalone TabArena Models

Source: https://context7.com/autogluon/tabarena/llms.txt

Utilize TabArena's SOTA models directly on your data. Load example data, get model configuration using get_configs_generator_from_name, and select the model class and configuration.

```python
from autogluon.core.data import LabelCleaner
from autogluon.core.models import BaggedEnsembleModel
from autogluon.features.generators import AutoMLPipelineFeatureGenerator
from tabarena.models.utils import get_configs_generator_from_name
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer

# Load example data
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42
)

# Get model configuration (RealMLP, LightGBM, CatBoost, XGBoost, TabPFNv2, etc.)
model_to_run = "RealMLP"
model_meta = get_configs_generator_from_name(model_name=model_to_run)
model_cls = model_meta.model_cls
model_config = model_meta.manual_configs[0]
```

--------------------------------

### Run AutoGluon on OpenML Task

Source: https://github.com/autogluon/tabarena/blob/main/examples/README.md

A minimal example for running AutoGluon on any OpenML task. This allows leveraging AutoGluon's capabilities within the TabArena context.

```python
run_autogluon_on_openml_task.py
```

--------------------------------

### Get Predictions for Multiple Configurations

Source: https://context7.com/autogluon/tabarena/llms.txt

Retrieve predictions for a specified subset of configurations on a given dataset and fold. Set `binary_as_multiclass=False` to get predictions in a shape of (n_configs, n_rows).

```python
configs_subset = ["CatBoost_r1_BAG_L1", "LightGBM_r1_BAG_L1", "RandomForest_r1_BAG_L1"]
predictions_multi = repo.predict_test_multi(
    dataset=dataset,
    fold=fold,
    configs=configs_subset,
    binary_as_multiclass=False  # Return shape (n_configs, n_rows) for binary
)
print(f"Multi-config predictions shape: {predictions_multi.shape}")
```

--------------------------------

### Activate Virtual Environment and Navigate Directory

Source: https://github.com/autogluon/tabarena/blob/main/tabflow_slurm/README.md

This command activates a Python virtual environment and changes the current directory to the TabArena SLURM code location. Ensure the paths are correct for your SLURM setup.

```bash
source /work/dlclarge2/purucker-tabarena/venvs/tabarena_25032026/bin/activate && cd /work/dlclarge2/purucker-tabarena/code/tabarena_new/tabarena/tabflow_slurm
```

--------------------------------

### List and Load Benchmark Contexts

Source: https://context7.com/autogluon/tabarena/llms.txt

Use `list_contexts` to see available benchmark contexts and `get_context` to retrieve details. Load a repository with `load_repository`, specifying version, cache, prediction loading, and prediction format.

```python
from tabarena import load_repository, list_contexts, get_context

# List all available benchmark contexts
context_names = list_contexts()
print(f"Available Contexts: {context_names}")
# Output: ['D244_F3_C1530', 'D244_F3_C1530_10', 'D244_F3_C1530_100', ...]

# Get context details
for name in context_names:
    context = get_context(name)
    print(f"{name}: {context.description}")

# Load a repository (auto-downloads required files)
repo = load_repository(
    version="D244_F3_C1530_30",  # 30 datasets, 1.1 GB
    cache=True,                   # Cache for faster subsequent loads
    load_predictions=True,        # Load model predictions for ensembling
    prediction_format="memmap",   # Memory-efficient format
)

# Print repository summary
repo.print_info()
# Output: Datasets: 30, Folds: 3, Configs: 1530
```

--------------------------------

### Query Datasets and Configurations

Source: https://context7.com/autogluon/tabarena/llms.txt

Access dataset information using `repo.datasets()`, `repo.dataset_info()`, and `repo.dataset_metadata()`. Explore model configurations with `repo.configs()`, `repo.config_type()`, `repo.config_hyperparameters()`, and `repo.autogluon_hyperparameters_dict()`.

```python
from tabarena import load_repository

repo = load_repository("D244_F3_C1530_30", cache=True)

# Get all datasets
datasets = repo.datasets()
print(f"Datasets: {datasets}")
# Output: ['Australian', 'balance-scale', 'blood-transfusion', ...]

# Get dataset info and metadata
dataset = "Australian"
dataset_info = repo.dataset_info(dataset=dataset)
print(f"Dataset Info: {dataset_info}")
# Output: {'metric': 'roc_auc', 'problem_type': 'binary', ...}

dataset_metadata = repo.dataset_metadata(dataset=dataset)
print(f"Metadata: {dataset_metadata}")
# Output: {'NumberOfInstances': 690, 'NumberOfFeatures': 14, ...}

# Get all configurations (model variants)
configs = repo.configs()
print(f"Total configs: {len(configs)}")
print(f"First 5 configs: {configs[:5]}")
# Output: ['CatBoost_r1_BAG_L1', 'CatBoost_r2_BAG_L1', ...]

# Get config hyperparameters and AutoGluon-compatible format
config = "CatBoost_r1_BAG_L1"
config_type = repo.config_type(config=config)
config_hp = repo.config_hyperparameters(config=config)
ag_hp = repo.autogluon_hyperparameters_dict(configs=[config])
print(f"Config type: {config_type}")
print(f"Hyperparameters: {config_hp}")
print(f"AutoGluon format: {ag_hp}")
```

--------------------------------

### Reproduce TabRepo Dataset Subset with Python

Source: https://github.com/autogluon/tabarena/blob/main/tabrepo.md

Run a small subset of the TabRepo dataset from scratch in a few minutes. Ensure the script is executable.

```python
examples/tabrepo/run_quickstart_from_scratch.py
```

--------------------------------

### Reproduce Paper Experiments

Source: https://github.com/autogluon/tabarena/blob/main/tabrepo.md

Executes a script to reproduce paper experiments, requiring the AutoML2024 branch of TabRepo. This process demands significant disk storage and memory, and may take considerable time.

```bash
python scripts/baseline_comparison/evaluate_baselines.py
```

--------------------------------

### Create Virtual Environment for TabArena

Source: https://github.com/autogluon/tabarena/blob/main/README.md

Create a dedicated virtual environment for TabArena using uv. This isolates project dependencies and ensures compatibility. Activate the environment using the provided source command.

```bash
uv venv --seed --python 3.12 ~/.venvs/tabarena
source ~/.venvs/tabarena/bin/activate
```

--------------------------------

### Download All Raw Data from TabArena Benchmarks

Source: https://github.com/autogluon/tabarena/blob/main/examples/README.md

Downloads all the raw data from prior TabArena benchmarks. This is useful for local analysis or re-processing of the data.

```python
run_download_raw.py
```

--------------------------------

### Benchmark New Models with AGModelBagExperiment

Source: https://context7.com/autogluon/tabarena/llms.txt

Set up and run benchmark experiments for new models against TabArena's standardized evaluation protocol using `AGModelBagExperiment` and `ExperimentBatchRunner`. This involves defining experiment paths, loading task metadata, selecting datasets and folds, and specifying the models to benchmark with their hyperparameters, bagging folds, and time limits.

```python
from pathlib import Path
from tabarena.benchmark.experiment import AGModelBagExperiment, ExperimentBatchRunner
from tabarena.nips2025_utils.tabarena_context import TabArenaContext
from tabarena.benchmark.models.ag import RealMLPModel
from autogluon.tabular.models import LGBModel

# Setup experiment paths
expname = str(Path("./experiments/my_benchmark"))

# Load TabArena task metadata
tabarena_context = TabArenaContext()
task_metadata = tabarena_context.task_metadata

# Select datasets and folds
datasets = ["anneal", "credit-g", "diabetes"]
folds = [0]

# Define methods to benchmark
methods = [
    AGModelBagExperiment(
        name="LightGBM_c1_BAG_L1_Custom",
        model_cls=LGBModel,
        model_hyperparameters={
            "learning_rate": 0.05,
            "num_leaves": 128,
        },
        num_bag_folds=8,    # 8-fold cross-validation bagging
        time_limit=3600,    # 1 hour time limit
    ),
    AGModelBagExperiment(
        name="RealMLP_c1_BAG_L1_Custom",
        model_cls=RealMLPModel,
        model_hyperparameters={},
        num_bag_folds=8,
        time_limit=3600,
    ),
]

# Run experiments
exp_batch_runner = ExperimentBatchRunner(
    expname=expname,
    task_metadata=task_metadata
)

results_lst = exp_batch_runner.run(
    datasets=datasets,
    folds=folds,
    methods=methods,
    ignore_cache=False,  # Set True to re-run from scratch
)

print(f"Completed {len(results_lst)} experiments")
```

--------------------------------

### Implement and Benchmark Custom Models in TabArena

Source: https://github.com/autogluon/tabarena/blob/main/examples/README.md

Provides a template for implementing your own model for TabArena and benchmarking it on TabArena-Lite. This is useful for custom model integration.

```python
custom_tabarena_model/
```

--------------------------------

### Accessing Processed Artifacts with EvaluationRepository

Source: https://context7.com/autogluon/tabarena/llms.txt

Downloads and inspects pre-computed benchmark artifacts for TabArena methods. Requires tabarena and pandas. Lists available methods, downloads artifacts if needed, and loads them into an EvaluationRepository for exploration.

```python
from tabarena import EvaluationRepository
from tabarena.nips2025_utils.artifacts import tabarena_method_metadata_collection
import pandas as pd

# List available methods
methods_info = tabarena_method_metadata_collection.info()
print(methods_info.to_markdown())
# Output:
# | method      | display_name | config_type | verified |
# |-------------|--------------|-------------|----------|
# | LightGBM    | LightGBM     | GBM         | True     |
# | CatBoost    | CatBoost     | CAT         | True     |
# ...

# Get metadata for a specific method
method = "LightGBM"
method_metadata = tabarena_method_metadata_collection.get_method_metadata(method=method)

# Download processed artifacts if not present
if not method_metadata.path_processed_exists:
    print(f"Downloading to {method_metadata.path_processed}...")
    method_metadata.method_downloader().download_processed()

# Load the method's repository
repo: EvaluationRepository = method_metadata.load_processed()
repo.print_info()

# Explore available data
datasets = repo.datasets()
configs = repo.configs()
print(f"Datasets: {len(datasets)}, Configs: {len(configs)}")

# Query metrics
metrics = repo.metrics(datasets=datasets[:2], configs=configs[:2])
with pd.option_context("display.width", 1000):
    print(metrics)
```

--------------------------------

### Generate and Format Leaderboards

Source: https://context7.com/autogluon/tabarena/llms.txt

Use TabArenaContext to compare methods against benchmarks and format leaderboards for website display. Ensure TabArenaContext is initialized correctly.

```python
from pathlib import Path
from tabarena.nips2025_utils.tabarena_context import TabArenaContext
from tabarena.nips2025_utils.end_to_end import EndToEnd
from tabarena.website.website_format import format_leaderboard

# Initialize TabArena context
tabarena_context = TabArenaContext(include_unverified=False)

# Generate main leaderboard comparing all methods
output_dir = Path("./output_leaderboard")
leaderboard = tabarena_context.compare(output_dir=output_dir)

# Format for website display
leaderboard_formatted = format_leaderboard(df_leaderboard=leaderboard)
print(leaderboard_formatted.to_markdown(index=False))
```

--------------------------------

### Preprocessing and Training with Bagged Ensemble

Source: https://context7.com/autogluon/tabarena/llms.txt

Demonstrates data preprocessing using AutoMLPipelineFeatureGenerator and LabelCleaner, followed by training a BaggedEnsembleModel with cross-validation. Ensure necessary imports and data are available.

```python
task_type = "binary"
feature_generator = AutoMLPipelineFeatureGenerator()
label_cleaner = LabelCleaner.construct(problem_type=task_type, y=y_train)

import pandas as pd
X_train_df = pd.DataFrame(X_train)
X_test_df = pd.DataFrame(X_test)
y_train_series = pd.Series(y_train)

X_train_processed = feature_generator.fit_transform(X_train_df)
y_train_processed = label_cleaner.transform(y_train_series)
X_test_processed = feature_generator.transform(X_test_df)
y_test_processed = label_cleaner.transform(pd.Series(y_test))

# Train with cross-validation bagging (recommended)
model = BaggedEnsembleModel(
    model_cls(problem_type=task_type, **model_config),
    hyperparameters=dict(refit_folds=False),
)
model.params["fold_fitting_strategy"] = "sequential_local"
model = model.fit(X=X_train_processed, y=y_train_processed, k_fold=8)

# Evaluate
print(f"Validation {model.eval_metric.name}:", model.score_with_oof(y=y_train_processed))
y_pred = model.predict(X=X_test_processed)
```

--------------------------------

### Benchmark Models on Custom Private Datasets with TabArena

Source: https://github.com/autogluon/tabarena/blob/main/examples/README.md

Demonstrates benchmarking models with TabArena on custom, private datasets. This is useful for evaluating models on your own data.

```python
run_quickstart_tabarena_custom_datasets.py
```

--------------------------------

### Benchmark and Evaluate Models on a Single Dataset with TabArena

Source: https://github.com/autogluon/tabarena/blob/main/examples/README.md

Focuses on benchmarking and evaluating models with TabArena on a single dataset. Useful for targeted model assessment.

```python
run_quickstart_tabarena_one_datasets.py
```

--------------------------------

### Compare Custom Results with TabArena

Source: https://context7.com/autogluon/tabarena/llms.txt

After running experiments, use EndToEnd to compute and compare custom results against TabArena benchmarks on matching tasks. Specify an output directory and whether to only include valid tasks.

```python
from tabarena.nips2025_utils.end_to_end import EndToEnd

# After running experiments, compute and compare results
end_to_end = EndToEnd.from_raw(
    results_lst=results_lst,  # From ExperimentBatchRunner
    task_metadata=task_metadata,
    cache=False
)
end_to_end_results = end_to_end.to_results()

# Compare against TabArena on matching tasks
comparison = end_to_end_results.compare_on_tabarena(
    output_dir=Path("./eval/comparison"),
    only_valid_tasks=True,
    new_result_prefix="My_"
)
```

--------------------------------

### Load Repository and Evaluate Ensemble

Source: https://github.com/autogluon/tabarena/blob/main/tabrepo.md

Loads a TabRepo repository and evaluates an ensemble of specified configurations for a given dataset and fold. This method computes ensemble weights using the Caruana procedure after loading model predictions and ground truth.

```python
from tabarena import load_repository

repo = load_repository("D244_F3_C1530_30")
print(repo.evaluate_ensemble(dataset="Australian", fold=0, configs=["CatBoost_r22_BAG_L1", "RandomForest_r12_BAG_L1"])))
```

--------------------------------

### Clone TabArena Repository

Source: https://github.com/autogluon/tabarena/blob/main/README.md

Clone the TabArena repository to your local machine. Ensure your working directory is the project root for subsequent commands to function correctly.

```bash
git clone https://github.com/autogluon/tabarena.git
cd tabarena  # ensure the working directory is the project root, otherwise the below commands won't work
```

--------------------------------

### Implement Custom TabArena Model

Source: https://context7.com/autogluon/tabarena/llms.txt

Create custom models compatible with TabArena by inheriting from AutoGluon's AbstractModel. Implement _preprocess and _fit methods. Ensure a unique ag_key is set.

```python
from autogluon.core.models import AbstractModel
from autogluon.features import LabelEncoderFeatureGenerator
import numpy as np
import pandas as pd

class CustomRandomForestModel(AbstractModel):
    """Custom model compatible with TabArena benchmarking."""

    ag_key = "CRF"      # Unique identifier for AutoGluon
    ag_name = "CustomRF"

    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self._feature_generator = None

    def _preprocess(self, X: pd.DataFrame, is_train=False, **kwargs) -> np.ndarray:
        X = super()._preprocess(X, **kwargs)
        if is_train:
            self._feature_generator = LabelEncoderFeatureGenerator(verbosity=0)
            self._feature_generator.fit(X=X)
        if self._feature_generator.features_in:
            X = X.copy()
            X[self._feature_generator.features_in] = self._feature_generator.transform(X=X)
        return X.fillna(0).to_numpy(dtype=np.float32)

    def _fit(self, X: pd.DataFrame, y: pd.Series, num_cpus: int = 1, **kwargs):
        if self.problem_type == "regression":
            from sklearn.ensemble import RandomForestRegressor
            model_cls = RandomForestRegressor
        else:
            from sklearn.ensemble import RandomForestClassifier
            model_cls = RandomForestClassifier

        X = self.preprocess(X, y=y, is_train=True)
        params = self._get_model_params()
        self.model = model_cls(**params)
        self.model.fit(X, y)

    def _set_default_params(self):
        default_params = {"n_estimators": 100, "n_jobs": -1, "random_state": 0}
        for param, val in default_params.items():
            self._set_default_param_value(param, val)
```

--------------------------------

### Generate Main Leaderboard from TabArena Results

Source: https://github.com/autogluon/tabarena/blob/main/examples/README.md

Generates the main leaderboard from TabArena benchmark results. This script is used for visualizing and comparing model performance.

```python
run_generate_main_leaderboard.py
```

--------------------------------

### Compute Bencheval Leaderboard with ELO Ratings

Source: https://context7.com/autogluon/tabarena/llms.txt

Computes a comprehensive leaderboard using the TabArena evaluator, including ELO ratings, win rates, and improvability scores. Requires pandas DataFrame with method, task, error, and optional extra columns like training/inference time.

```python
from bencheval.tabarena import TabArena
import pandas as pd

# Prepare results data (from experiments or loaded from files)
results_data = pd.DataFrame({
    "method": ["LightGBM", "LightGBM", "CatBoost", "CatBoost"],
    "task": ["dataset_A_fold_0", "dataset_B_fold_0", "dataset_A_fold_0", "dataset_B_fold_0"],
    "metric_error": [0.12, 0.15, 0.11, 0.16],
    "time_train_s": [10.5, 12.3, 15.2, 18.1],
    "time_infer_s": [0.01, 0.02, 0.02, 0.03],
})

# Initialize TabArena evaluator
evaluator = TabArena(
    method_col="method",
    task_col="task",
    error_col="metric_error",
    columns_to_agg_extra=["time_train_s", "time_infer_s"],
)

# Compute leaderboard
leaderboard = evaluator.leaderboard(
    data=results_data,
    include_elo=True,
    include_winrate=True,
    include_improvability=True,
    include_rank_counts=True,
    sort_by="rank",
)

print(leaderboard)
```

--------------------------------

### Load TabRepo Repository

Source: https://github.com/autogluon/tabarena/blob/main/tabrepo.md

Loads a TabRepo repository using a specified context name. This is a foundational step for accessing model evaluations, predictions, and ensemble simulations. The repository data is downloaded from the internet.

```python
from tabarena import load_repository

repo = load_repository(context_name)
```

--------------------------------

### Download TabArena Datasets from OpenML

Source: https://github.com/autogluon/tabarena/blob/main/examples/README.md

Retrieves the data used by TabArena from OpenML. This script allows access to the datasets without the full TabArena framework.

```python
run_get_tabarena_datasets_from_openml.py
```

--------------------------------

### Load Repository and Access Model Metrics

Source: https://github.com/autogluon/tabarena/blob/main/tabrepo.md

Loads a TabRepo repository and retrieves metrics for specified datasets and configurations. Use this to inspect model performance across different settings. The context name determines the scope of data downloaded.

```python
from tabarena import load_repository

repo = load_repository("D244_F3_C1530_30")
repo.metrics(datasets=["Australian"], configs=["CatBoost_r22_BAG_L1", "RandomForest_r12_BAG_L1"])
```

--------------------------------

### Load Repository and Query Model Predictions

Source: https://github.com/autogluon/tabarena/blob/main/tabrepo.md

Loads a TabRepo repository and queries predictions for a specific dataset, fold, and configurations. Use `predict_val_multi` for validation set predictions and `predict_test` for test set predictions. Ensure the specified context is loaded.

```python
from tabarena import load_repository

repo = load_repository("D244_F3_C1530_30")
print(repo.predict_val_multi(dataset="Australian", fold=0, configs=["CatBoost_r22_BAG_L1", "RandomForest_r12_BAG_L1"])))
```

--------------------------------

### Generate Plots of Pareto Trade-offs (Performance vs. Efficiency)

Source: https://github.com/autogluon/tabarena/blob/main/examples/README.md

Generates plots showing the trade-offs between predictive performance and efficiency (e.g., tuning time). Useful for analyzing model scalability.

```python
run_plot_pareto_over_tuning_time.py
```

--------------------------------

### Use Custom Model in TabArena Experiments

Source: https://context7.com/autogluon/tabarena/llms.txt

Integrate a custom model into TabArena experiments using AGModelBagExperiment. Specify the model class, hyperparameters, number of folds, and time limit.

```python
from tabarena.benchmark.experiment import AGModelBagExperiment

experiment = AGModelBagExperiment(
    name="CustomRF_c1_BAG_L1",
    model_cls=CustomRandomForestModel,
    model_hyperparameters={"n_estimators": 200},
    num_bag_folds=8,
    time_limit=3600,
)
```

--------------------------------

### Access Model Predictions and Labels

Source: https://context7.com/autogluon/tabarena/llms.txt

Retrieve test predictions using `repo.predict_test()`, validation predictions with `repo.predict_val()`, and ground truth labels via `repo.labels_test()` and `repo.labels_val()` for specific dataset-fold-config combinations.

```python
from tabarena import load_repository
import numpy as np

repo = load_repository("D244_F3_C1530_30", cache=True)

dataset = "Australian"
fold = 0
config = "CatBoost_r1_BAG_L1"

# Get test predictions (probability predictions for classification)
predictions_test = repo.predict_test(dataset=dataset, fold=fold, config=config)
print(f"Test predictions shape: {predictions_test.shape}")
print(f"Test predictions (first 5): {predictions_test[:5]}")
# Output: Test predictions shape: (69,)
# Output: [0.234, 0.891, 0.456, ...]

# Get validation predictions (out-of-fold)
predictions_val = repo.predict_val(dataset=dataset, fold=fold, config=config)
print(f"Validation predictions shape: {predictions_val.shape}")

# Get ground truth labels
y_test = repo.labels_test(dataset=dataset, fold=fold)
y_val = repo.labels_val(dataset=dataset, fold=fold)
print(f"Test labels: {y_test[:10]}")
print(f"Val labels: {y_val[:10]}")
```

--------------------------------

### Subsetting and Filtering Repositories with load_repository

Source: https://context7.com/autogluon/tabarena/llms.txt

Filters TabArena repositories by datasets, configurations, folds, or problem types. Uses `load_repository` and the `subset` method. `inplace=False` returns a new repository without modifying the original.

```python
from tabarena import load_repository

repo = load_repository("D244_F3_C1530_30", cache=True)

# Subset by datasets and configs
repo_subset = repo.subset(
    datasets=["Australian", "balance-scale", "blood-transfusion"],
    configs=repo.configs()[:50],  # First 50 configs
    folds=[0, 1],                  # Only folds 0 and 1
    problem_types=["binary"],      # Only binary classification
    inplace=False,                 # Return new repo, don't modify original
)

print(f"Original: {len(repo.datasets())} datasets, {len(repo.configs())} configs")
print(f"Subset: {len(repo_subset.datasets())} datasets, {len(repo_subset.configs())} configs")
```

--------------------------------

### Inspect Raw Data from TabArena Benchmarks

Source: https://github.com/autogluon/tabarena/blob/main/examples/README.md

Inspects the raw data collected from prior TabArena benchmarks. This allows for a deeper look into the original datasets.

```python
inspect_raw_data.py
```

--------------------------------

### Tuned and Ensembled Models with AutoGluon

Source: https://context7.com/autogluon/tabarena/llms.txt

Trains models with hyperparameter tuning and automatic ensembling using AutoGluon's TabularPredictor. Requires AutoGluon and scikit-learn. Data is loaded using fetch_california_housing and split.

```python
from autogluon.tabular import TabularPredictor
from tabarena.models.utils import get_configs_generator_from_name
import pandas as pd
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split

# Load data
data = fetch_california_housing()
X_train, X_test, y_train, y_test = train_test_split(
    pd.DataFrame(data.data, columns=data.feature_names),
    data.target, test_size=0.2, random_state=42
)
train_data = X_train.copy()
train_data["target"] = y_train

# Get hyperparameter configurations for tuning
model_to_run = "LightGBM"
model_meta = get_configs_generator_from_name(model_name=model_to_run)
model_cls = model_meta.model_cls
n_configs = 10  # TabArena uses 200 for full benchmark
hpo_configs = model_meta.generate_all_configs_lst(num_random_configs=n_configs)

# Train with tuning and ensembling
model_hyperparameters = {model_cls: hpo_configs}
predictor = TabularPredictor(
    label="target",
    eval_metric="rmse",
    problem_type="regression",
).fit(
    train_data,
    fit_weighted_ensemble=True,      # Enable post-hoc ensemble
    hyperparameters=model_hyperparameters,
    num_bag_folds=8,                 # 8-fold CV bagging
    time_limit=600,                  # 10 min time limit
)

# View validation performance
predictor.leaderboard(display=True)

# Predict
y_pred = predictor.predict(data=X_test)
rmse = ((y_pred - y_test) ** 2).mean() ** 0.5
print(f"Test RMSE: {rmse:.4f}")
```

--------------------------------

### Force Repository to Dense Representation

Source: https://context7.com/autogluon/tabarena/llms.txt

Converts a repository subset to a dense representation where all tasks share the same configurations. Useful for ensuring consistent task definitions.

```python
repo_dense = repo_subset.force_to_dense(inplace=False, verbose=True)
print(f"Dense: {len(repo_dense.datasets())} datasets, {len(repo_dense.configs())} configs")

tasks = repo_dense.tasks()
print(f"Tasks: {tasks[:5]}")
```

--------------------------------

### Query Performance Metrics

Source: https://context7.com/autogluon/tabarena/llms.txt

Retrieve performance metrics such as error, training time, and inference time for specific datasets and configurations. The `metrics` function can filter by dataset, fold, and configuration. Use `pd.option_context` to display all columns and adjust width for better readability of the output DataFrame.

```python
from tabarena import load_repository
import pandas as pd

repo = load_repository("D244_F3_C1530_30", cache=True)

# Get metrics for specific datasets and configs
metrics = repo.metrics(
    datasets=["Australian", "balance-scale"],
    configs=["CatBoost_r1_BAG_L1", "LightGBM_r41_BAG_L1"]
)

with pd.option_context("display.max_columns", None, "display.width", 1000):
    print(metrics)
```

```python
# Get metrics for all configs on a single dataset
all_metrics = repo.metrics(datasets=["Australian"])
print(f"Total metrics: {len(all_metrics)}")
```

```python
# Filter by problem type
regression_datasets = repo.datasets(problem_type="regression")
classification_datasets = repo.datasets(problem_type=["binary", "multiclass"])
```

--------------------------------

### Inspect Processed Data from TabArena Benchmarks

Source: https://github.com/autogluon/tabarena/blob/main/examples/README.md

Inspects the processed data generated from prior TabArena benchmarks. Useful for understanding the data used in evaluations.

```python
inspect_processed_data.py
```

--------------------------------

### Evaluate Ensembles Across All Tasks

Source: https://context7.com/autogluon/tabarena/llms.txt

Evaluate ensembles across multiple datasets in parallel using the `evaluate_ensembles` function. Specify the datasets, configurations, ensemble size, and the backend for parallelization (e.g., 'ray').

```python
from tabarena import load_repository

repo = load_repository("D244_F3_C1530_30", cache=True)

dataset = "Australian"
fold = 0
configs = repo.configs()

# Evaluate ensembles across all tasks (parallelized)
df_results_all, df_weights_all = repo.evaluate_ensembles(
    datasets=repo.datasets()[:5],  # First 5 datasets
    configs=configs[:100],          # First 100 configs
    ensemble_size=50,
    backend="ray",                  # Use Ray for parallelization
)
print(f"Results shape: {df_results_all.shape}")
```

--------------------------------

### Generate Figures for TabArena NeurIPS Paper

Source: https://github.com/autogluon/tabarena/blob/main/examples/README.md

Generates the figures used in the TabArena NeurIPS'2025 paper. This script is crucial for reproducing the paper's visualizations.

```python
run_generate_paper_figures.py
```

--------------------------------

### Evaluate Ensembles on a Single Task

Source: https://context7.com/autogluon/tabarena/llms.txt

Simulate ensemble selection using cached predictions for a given dataset and fold. The `evaluate_ensemble` function finds optimal model combinations. Specify `ensemble_size` for the number of iterations and `time_limit` for a training time budget. `fit_order` can be 'original' or 'random'.

```python
from tabarena import load_repository

repo = load_repository("D244_F3_C1530_30", cache=True)

dataset = "Australian"
fold = 0
configs = repo.configs()

# Evaluate ensemble on a single task
df_result, df_weights = repo.evaluate_ensemble(
    dataset=dataset,
    fold=fold,
    configs=configs,
    ensemble_size=100,      # Number of ensemble iterations
    time_limit=3600,        # Optional: max training time budget
    fit_order="original",   # or "random" to shuffle config order
)

print(f"Ensemble result:\n{df_result}")
```

```python
# Analyze ensemble weights
weights_sorted = df_weights.iloc[0].sort_values(ascending=False)
print(f"Top 5 ensemble models:\n{weights_sorted.head(5)}")
```

--------------------------------

### Bibtex Entry for TabArena Publication

Source: https://github.com/autogluon/tabarena/blob/main/README.md

Use this Bibtex entry when citing the TabArena paper in scientific publications.

```bibtex
@inproceedings{erickson2025tabarena,
title     = {TabArena: A Living Benchmark for Machine Learning on Tabular Data},
author    = {Erickson, Nick and Purucker, Lennart and Tschalzev, Andrej and Holzm{"u}ller, David and Desai, Prateek Mutalik and Salinas, David and Hutter, Frank},
booktitle = {Proceedings of the 39th Conference on Neural Information Processing Systems (NeurIPS)},
year      = {2025},
url       = {https://arxiv.org/abs/2506.16791}
}
```

--------------------------------

### TabRepo Citation

Source: https://github.com/autogluon/tabarena/blob/main/tabrepo.md

Citation details for the TabRepo work, suitable for academic use.

```bibtex
@inproceedings{
  tabrepo,
  title={TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applications},
  author={David Salinas and Nick Erickson},
  booktitle={AutoML Conference 2024 (ABCD Track)},
  year={2024},
  url={https://openreview.net/forum?id=03V2bjfsFC}
}
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.