### Install Matbench from PyPI

Source: https://github.com/materialsproject/matbench/blob/main/docs/How To Use/1install/index.html

Install the latest stable version of Matbench directly from the Python Package Index (PyPI).

```bash
pip install matbench
```

--------------------------------

### Install Matbench

Source: https://context7.com/materialsproject/matbench/llms.txt

Install Matbench using pip for standard usage or from source for development.

```bash
pip install matbench
```

```bash
git clone https://github.com/hackingmaterials/matbench
pip install -e ./matbench -r ./matbench/requirements.txt
```

--------------------------------

### Install Matbench for Current User

Source: https://github.com/materialsproject/matbench/blob/main/docs/How To Use/1install/index.html

Install Matbench for the current user after cloning the repository.

```bash
pip install --user ./matbench
```

--------------------------------

### Install Matbench for Development

Source: https://github.com/materialsproject/matbench/blob/main/docs/How To Use/1install/index.html

Install Matbench in editable mode for development purposes, including its requirements.

```bash
cd matbench pip install -e . -r requirements.txt
```

--------------------------------

### Install and Import Libraries

Source: https://github.com/materialsproject/matbench/blob/main/benchmarks/matbench_v0.1_lattice_xgboost/notebook.ipynb

Installs necessary libraries like matbench, matminer, pymatgen, and scikit-learn. Imports required classes for data handling and modeling.

```python
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error

from matbench.bench import MatBench
from matminer.datasets import load_dataset

from xgboost import XGBRegressor
```

--------------------------------

### Install Matbench and XGBoost

Source: https://github.com/materialsproject/matbench/blob/main/benchmarks/matbench_v0.1_lattice_xgboost/notebook.ipynb

Installs the necessary libraries for running Matbench benchmarks with XGBoost. This command should be run in a Python environment.

```bash
%pip install matbench xgboost
```

--------------------------------

### Matbench Benchmark Initialization

Source: https://github.com/materialsproject/matbench/blob/main/benchmarks/matbench_v0.1_GN-OA/GN_OA.ipynb

Initializes the Matbench benchmark and loads a specific task. This is the starting point for running benchmarks.

```python
from matbench.bench import MatbenchBenchmark


mb = MatbenchBenchmark(autoload=False)

task = mb.tasks_map['matbench_mp_e_form']
task.load()
```

--------------------------------

### Import Matbench and Dummy Model Classes

Source: https://github.com/materialsproject/matbench/blob/main/benchmarks/matbench_v0.1_dummy/notebook.ipynb

Import the required classes for dummy regression and classification, along with the MatbenchBenchmark class. This is necessary setup for running the benchmark.

```python
from sklearn.dummy import DummyRegressor, DummyClassifier
from matbench.bench import MatbenchBenchmark
```

--------------------------------

### Import Libraries for Matbench Benchmarking

Source: https://github.com/materialsproject/matbench/blob/main/benchmarks/matbench_v0.1_Ax_SAASBO_CrabNet_v1.2.7/notebook.ipynb

Imports essential libraries for Matbench benchmarking, including submitit for job execution, cloudpickle for serialization, and utility functions from 'utils.matbench'. Ensure 'pyro-ppl' is installed for FULLYBAYESIAN (SAASBO) functionality.

```python
# %% imports
# NOTE: `pip install pyro-ppl` to use FULLYBAYESIAN (SAASBO)
from submitit import AutoExecutor
import cloudpickle as pickle
from utils.matbench import matbench_fold, collect_results, task, savepath, dummy

print(f"dummy: {dummy}")
```

--------------------------------

### Import Matbench and necessary packages

Source: https://github.com/materialsproject/matbench/blob/main/benchmarks/matbench_v0.1_CrabNet/notebook.ipynb

Imports essential libraries for data manipulation, plotting, and the Matbench framework. Ensure these packages are installed before running.

```python
#import necessary packages including matbench

import os
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
from matbench.bench import MatbenchBenchmark
```

--------------------------------

### Retrieve Benchmark Information

Source: https://github.com/materialsproject/matbench/blob/main/benchmarks/matbench_v0.1_dummy/notebook.ipynb

Get additional information about the benchmark. This function call retrieves and potentially displays details related to the benchmark setup or execution.

```python
# Get some more info about the benchmark
mb.get_info()
```

--------------------------------

### Get Train/Validation Split

Source: https://context7.com/materialsproject/matbench/llms.txt

Retrieve training and validation data for a specific fold using task.get_train_and_val_data(). Use as_type='df' to get a pandas DataFrame.

```python
from matbench.bench import MatbenchBenchmark

mb = MatbenchBenchmark(autoload=False)
task = mb.matbench_dielectric
task.load()

for fold in task.folds:  # [0, 1, 2, 3, 4]
    train_inputs, train_outputs = task.get_train_and_val_data(fold)
    # train_inputs: pd.Series of pymatgen.Structure objects (for structure tasks)
    #               or composition strings (for composition tasks)
    # train_outputs: pd.Series of float (regression) or bool (classification)

    print(f"Fold {fold}: {len(train_inputs)} training samples")
    # Fold 0: 3811 training samples
    # Fold 1: 3811 training samples
    # ...

    # As DataFrame instead of tuple
    train_df = task.get_train_and_val_data(fold, as_type="df")
    print(train_df.columns.tolist())  # ['structure', 'n']
```

--------------------------------

### Initialize and Run Matbench Benchmark

Source: https://github.com/materialsproject/matbench/blob/main/benchmarks/matbench_v0.1_CrabNet_v1.2.1/notebook.ipynb

Initializes the Matbench benchmark for the experimental gap task, loads data, defines model parameters, trains the model, and records predictions for each fold. Includes memory deallocation for CUDA.

```python
mb = MatbenchBenchmark(autoload=False, subset=["matbench_expt_gap"])

default_maes = []
task = list(mb.tasks)[0]
task.load()
for i, fold in enumerate(task.folds):
    train_inputs, train_outputs = task.get_train_and_val_data(fold)

    train_val_df = pd.DataFrame(
        {"formula": train_inputs.values, "target": train_outputs.values}
    )
    if dummy:
        train_val_df = train_val_df[:100]

    test_inputs, test_outputs = task.get_test_data(fold, include_target=True)

    test_df = pd.DataFrame({"formula": test_inputs, "target": test_outputs})

    default_params = dict(
        fudge=0.02,
        d_model=512,
        out_dims=3,
        N=3,
        heads=4,
        out_hidden=[1024, 512, 256, 128],
        emb_scaler=1.0,
        pos_scaler=1.0,
        pos_scaler_log=1.0,
        bias=False,
        dim_feedforward=2048,
        dropout=0.1,
        elem_prop="mat2vec",
        pe_resolution=5000,
        ple_resolution=5000,
        epochs=40,
        epochs_step=10,
        criterion=None,
        lr=1e-3,
        betas=(0.9, 0.999),
        eps=1e-6,
        weight_decay=0,
        adam=False,
        min_trust=None,
        alpha=0.5,
        k=6,
        base_lr=1e-4,
        max_lr=6e-3,
    )

    default_model = get_model(
        mat_prop="expt_gap",
        train_df=train_val_df,
        learningcurve=False,
        force_cpu=False,
        **default_params,
    )

    default_true, default_pred, default_formulas, default_sigma = default_model.predict(
        test_df
    )

    default_mae = mean_absolute_error(default_true, default_pred)
    default_maes.append(default_mae)

    task.record(fold, default_pred, params=default_params)

    # deallocate CUDA memory https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/28
    del default_model
    gc.collect()
    torch.cuda.empty_cache()
```

--------------------------------

### Initialize MatbenchBenchmark from Preset

Source: https://github.com/materialsproject/matbench/blob/main/docs/Reference/MatbenchBenchmark/index.html

Create a MatbenchBenchmark instance using predefined presets for a given benchmark. Presets allow for easy selection of specific problem types, such as structure or composition.

```python
cls.from_preset(benchmark='matbench_v0.1', preset_name='structure', autoload=False)
```

--------------------------------

### Hyperparameter Optimization Setup for CrabNet

Source: https://github.com/materialsproject/matbench/blob/main/benchmarks/matbench_v0.1_Ax_CrabNet_v1.2.1/notebook.ipynb

Configures the hyperparameter search space, experiment parameters, and evaluation function for CrabNet. This setup is used with Ax for Bayesian optimization.

```python
search_space = {
            "name": "weight_decay", "type": "range", "bounds": [0.0, 1.0]
        },
        # {"name": "adam", "type": "choice", "values": [False, True]}, # issues with onehot
        # {"name": "min_trust", "type": "range", "bounds": [0.0, 1.0]}, #issues with onehot
        {"name": "alpha", "type": "range", "bounds": [0.0, 1.0]},
        {"name": "k", "type": "range", "bounds": [2, 10]},
    ],
    experiment_name="crabnet-hyperparameter",
    evaluation_function=crabnet_mae,
    objective_name="mae",
    minimize=True,
    parameter_constraints=["betas1 <= betas2", "emb_scaler + pos_scaler <= 1"],
    total_trials=total_trials,
)
```

--------------------------------

### Create Matbench Benchmark

Source: https://github.com/materialsproject/matbench/blob/main/benchmarks/matbench_v0.1_lattice_xgboost/notebook.ipynb

Initializes a Matbench benchmark instance, specifying the task subset to be loaded. Use this to begin a benchmarking process.

```python
# Create a benchmark
mb = MatbenchBenchmark(autoload=False, subset=["matbench_mp_e_form"])

```

--------------------------------

### Access Benchmark Info

Source: https://github.com/materialsproject/matbench/blob/main/docs/Reference/MatbenchBenchmark/index.html

Get a formatted string containing information about the benchmark and its current state.

```python
info
```

--------------------------------

### Initialize Matbench Benchmark

Source: https://github.com/materialsproject/matbench/blob/main/benchmarks/matbench_v0.1_automatminer_expressv2020/notebook.ipynb

Create an instance of the MatbenchBenchmark class. Set autoload to False to manually load data.

```python
# Create a benchmark
mb = MatbenchBenchmark(autoload=False)
```

--------------------------------

### coNGN Software Requirements

Source: https://github.com/materialsproject/matbench/blob/main/docs_src/Full Benchmark Data/matbench_v0.1_coNGN.md

Specifies the Python packages and versions required to run the coNGN benchmark. Ensure these are installed before execution.

```python
{
'python': ['git+https://github.com/robinruff/graphlist@dcbf79e',
            'kgcnn==3.0.0']
}
```

--------------------------------

### Full Matbench Benchmark Loop

Source: https://context7.com/materialsproject/matbench/llms.txt

This code demonstrates the complete end-to-end workflow for running a Matbench v0.1 benchmark, including loading tasks, training/predicting, recording results, adding metadata, and saving the benchmark.

```python
from matbench.bench import MatbenchBenchmark

mb = MatbenchBenchmark(autoload=False)

for task in mb.tasks:
    task.load()
    for fold in task.folds:
        train_inputs, train_outputs = task.get_train_and_val_data(fold)
        test_inputs = task.get_test_data(fold, include_target=False)

        # Replace with your actual model
        my_model.fit(train_inputs, train_outputs)
        predictions = my_model.predict(test_inputs)

        # Record hyperparameters for this fold (optional but recommended)
        params = my_model.get_params()
        task.record(fold, predictions, params=params)

# Add algorithm-level metadata (shown on leaderboard)
mb.add_metadata({
    "algorithm_version": "v1.0",
    "notes": "Gradient boosted trees with Magpie features",
    "training_time_hrs": 2.5,
})

# Validate and save
print(mb.is_valid)   # True
print(mb.is_complete)  # True
mb.to_file("my_algorithm_results.json.gz")
# Saved file is submitted to the Matbench leaderboard via GitHub PR
```

--------------------------------

### Get Fold Data as DataFrame

Source: https://github.com/materialsproject/matbench/blob/main/docs/Reference/MatbenchTask/index.html

Retrieves data for specified IDs from the task's DataFrame, returning a subset of the DataFrame.

```python
task._get_data_from_df(ids=task.folds_keys, as_type="df")
```

--------------------------------

### Instantiate MatbenchBenchmark from Preset

Source: https://github.com/materialsproject/matbench/blob/main/docs/Reference/MatbenchBenchmark/index.html

Use `from_preset` to create a MatbenchBenchmark object based on predefined configurations like 'structure', 'composition', 'regression', 'classification', or 'all' for 'matbench_v0.1'. Set `autoload=True` to load all datasets immediately, but be aware this can be time-consuming.

```python
from matbench.bench import MatbenchBenchmark

# Example: Instantiate with structure preset
benchmark = MatbenchBenchmark.from_preset("matbench_v0.1", "structure")

# Example: Instantiate with all presets and autoload
benchmark_all = MatbenchBenchmark.from_preset("matbench_v0.1", "all", autoload=True)
```

--------------------------------

### Save Experiment and Get Test Data

Source: https://github.com/materialsproject/matbench/blob/main/benchmarks/matbench_v0.1_Ax_CrabNet_v1.2.1/notebook.ipynb

Saves the hyperparameter optimization experiment configuration and retrieves test data for evaluation.

```python
experiment_fpath = join(experiment_dir, "experiment" + str(i) + ".json")
save_experiment(experiment, experiment_fpath)

# TODO: save plot, save experiment

test_inputs, test_outputs = task.get_test_data(fold, include_target=True)
```

--------------------------------

### Get Training and Validation Data

Source: https://github.com/materialsproject/matbench/blob/main/docs/Reference/MatbenchTask/index.html

Retrieves the training and validation data for a specific fold. Data can be returned as a pandas DataFrame or a tuple.

```python
def get_train_and_val_data(self, fold_number, as_type="tuple"):
    """
    The training + validation data. All model tuning and
    hyperparameter selection must be done on this data, NOT test data.

    Args:
        fold_number (int): Index of the fold to retrieve test data.

    Returns:
        (pd.Dataframe) or (tuple): Returns either a dataframe of
            training data or a 2-tuple of training data.
    """
    self._check_is_loaded()
    fold_key = self.folds_map[fold_number]
    ids = self.validation[fold_key].train
    return self._get_data_from_df(ids, as_type)
```

--------------------------------

### Get Fold Data as Tuple

Source: https://github.com/materialsproject/matbench/blob/main/docs/Reference/MatbenchTask/index.html

Retrieves data for specified IDs from the task's DataFrame, returning inputs and outputs as a 2-tuple.

```python
task._get_data_from_df(ids=task.folds_keys, as_type="tuple")
```

--------------------------------

### Initialize Matbench Benchmark

Source: https://github.com/materialsproject/matbench/blob/main/docs/Reference/MatbenchBenchmark/index.html

Initializes the Matbench benchmark. Supports 'matbench_v0.1' and allows loading datasets into memory or lazily.

```python
def __init__(self, benchmark=MBV01_KEY, autoload=False, subset=None):
    """
    Args:
        benchmark (str): The name of the benchmark. Only supported benchmark
            currently is "matbench_v0.1", though more will be added in the
            future.
        autoload (bool): If True, automatically load the dataset into memory
            For a full benchmark, this can take some time. If False, you'll
            need to load each task with .load before you can access the raw
            data.
        subset ([str]): A list of task names to use as a subset of a full
            benchmark. Only the named tasks will be contained in the class.
            Must correspond to the metadata file defined by the benchmark
            name.
    """
    if benchmark == MBV01_KEY:
        self.benchmark_name = MBV01_KEY
        self.metadata = mbv01_metadata
    else:
        raise ValueError(
            f"Only '{MBV01_KEY}' available. No other benchmarks defined!"
        )

    if subset:
        not_datasets = [k for k in subset if k not in self.metadata]
        if not_datasets:
            raise KeyError(
                f"Some tasks in {subset} are not benchmark="
                f
```

--------------------------------

### Initialize MatbenchBenchmark

Source: https://context7.com/materialsproject/matbench/llms.txt

Create a MatbenchBenchmark instance to manage one or more tasks. Use autoload=False to defer dataset loading and subset to select specific tasks.

```python
from matbench.bench import MatbenchBenchmark

# Full benchmark, datasets loaded lazily (recommended for large runs)
mb = MatbenchBenchmark(autoload=False)

# Subset of 3 tasks only
mb_subset = MatbenchBenchmark(
    autoload=False,
    subset=["matbench_dielectric", "matbench_steels", "matbench_glass"]
)

print(mb.benchmark_name)   # matbench_v0.1
print(len(list(mb.tasks))) # 13
print(mb.is_complete)      # True
print(mb.is_recorded)      # False (no predictions yet)
```

--------------------------------

### Get MatbenchBenchmark Info

Source: https://github.com/materialsproject/matbench/blob/main/docs/Reference/MatbenchBenchmark/index.html

The `get_info` method logs detailed information about the benchmark to the configured logging handlers. This method does not return any value.

```python
benchmark.get_info()
```

--------------------------------

### Create experiment directories

Source: https://github.com/materialsproject/matbench/blob/main/benchmarks/matbench_v0.1_CrabNet_v1.2.1/notebook.ipynb

Defines and creates necessary directories for experiments, figures, and results. Ensures these directories exist before saving output files.

```python
# create dir https://stackoverflow.com/a/273227/13697228
experiment_dir = join("experiments", "default")
figure_dir = join("figures", "default")
result_dir = join("results", "default")
Path(experiment_dir).mkdir(parents=True, exist_ok=True)
Path(figure_dir).mkdir(parents=True, exist_ok=True)
Path(result_dir).mkdir(parents=True, exist_ok=True)
```

--------------------------------

### Matbench Software Requirements

Source: https://github.com/materialsproject/matbench/blob/main/docs_src/Full Benchmark Data/matbench_v0.1_rf.md

Specifies the Python package versions required to run the Matbench v0.1 benchmarks. Ensure these dependencies are installed before execution.

```python
{
'python': ['scikit-learn==0.24.1',
            'numpy==1.20.1',
            'matbench==0.1.0',
            'automatminer==v1.0.3.20191111']
}
```

--------------------------------

### Initialize MatbenchBenchmark

Source: https://github.com/materialsproject/matbench/blob/main/docs/Reference/MatbenchBenchmark/index.html

Instantiate MatbenchBenchmark with a specific benchmark and optional subset of tasks. Set autoload to True to load all datasets into memory upon initialization.

```python
mb = MatbenchBenchmark()
mb.matbench_dielectric
```

```python
mb = MatbenchBenchmark(benchmark='matbench_v0.1', autoload=False, subset=['matbench_dielectric', 'matbench_expt_gap'])

```

--------------------------------

### Task Properties

Source: https://github.com/materialsproject/matbench/blob/main/docs/Reference/MatbenchTask/index.html

Access properties of a Matbench task to get comprehensive score metrics, check fold recording status, and identify polymorphs.

```APIDOC
## Task Properties

### `scores` property

**Description:**
Comprehensive score metrics for this task. Gets means, maxes, mins, and more distribution stats (across folds) for all scoring metrics defined for this task. There will be different scores for classification problems and regression problems.

**Returns:**
- `dict`: A dictionary of all the scores for this task.

### `is_recorded` property

**Description:**
Determine what folds in the task are recorded.

**Returns:**
- `{int: bool}`: Keys are fold numbers, values are whether the fold is recorded or not.

### `all_folds_recorded` property

**Description:**
Determine if all folds are recorded.

**Returns:**
- `bool`: True if all folds are recorded, False otherwise.

### `has_polymorphs` property

**Description:**
Determine if a task's raw data contains polymorphs.

**Returns:**
- `bool`: If true, contains polymorphs.
```

--------------------------------

### Task Initialization

Source: https://github.com/materialsproject/matbench/blob/main/docs/Reference/MatbenchTask/index.html

Initialize a Matbench task with a dataset name and optional parameters for loading and benchmarking.

```APIDOC
## Task Initialization

### `__init__(dataset_name, autoload=True, benchmark=MBV01_KEY)`

**Description:**
Initializes a Matbench task.

**Parameters:**
- `dataset_name` (str): Name of the task. Must belong to the benchmark given in the 'benchmark' argument.
- `autoload` (bool): Whether to automatically load the task data. Defaults to True.
- `benchmark` (str): The benchmark key to associate the task with. Defaults to `MBV01_KEY`.
```

--------------------------------

### GPTChem Software Requirements

Source: https://github.com/materialsproject/matbench/blob/main/docs_src/Full Benchmark Data/matbench_v0.1_gptchem.md

Specifies the Python packages and their versions required to run the GPTChem benchmark. This includes the gptchem package installed from a Git repository.

```python
{
    "python": ["git+https://github.com/kjappelbaum/gptchem.git",
               "matbench==0.1.0"]
}
```

--------------------------------

### Initialize MatbenchBenchmark from Preset

Source: https://context7.com/materialsproject/matbench/llms.txt

Use MatbenchBenchmark.from_preset to create benchmarks filtered by input type (structure/composition) or task type (regression/classification).

```python
from matbench.bench import MatbenchBenchmark

# Only structure-based tasks (9 tasks)
mb_struct = MatbenchBenchmark.from_preset("matbench_v0.1", "structure")

# Only composition-based tasks (4 tasks)
mb_comp = MatbenchBenchmark.from_preset("matbench_v0.1", "composition")

# Only regression tasks (10 tasks)
mb_reg = MatbenchBenchmark.from_preset("matbench_v0.1", "regression")

# Only classification tasks (3 tasks)
mb_clf = MatbenchBenchmark.from_preset("matbench_v0.1", "classification")

# All 13 tasks (equivalent to default constructor)
mb_all = MatbenchBenchmark.from_preset("matbench_v0.1", "all", autoload=False)

print(list(mb_comp.tasks_map.keys()))
# ['matbench_steels', 'matbench_expt_gap', 'matbench_expt_is_metal', 'matbench_glass']
```

--------------------------------

### Software Requirements for TPOT-Mat Benchmark

Source: https://github.com/materialsproject/matbench/blob/main/docs_src/Full Benchmark Data/matbench_v0.1_TPOT.md

Lists the Python packages and their specific versions required to run the TPOT-Mat benchmark. Ensure these dependencies are installed before execution.

```python
{'python': ['numpy==1.23.5',
            'pandas==1.5.1',
            'jupyter==1.0.0',
            'tpot==0.11.7',
            'scikit-learn==1.2.2',
            'matbench',
            'joblib']}
```

--------------------------------

### Matbench Benchmark Initialization

Source: https://github.com/materialsproject/matbench/blob/main/docs/Reference/MatbenchBenchmark/index.html

Initializes a Matbench benchmark instance. You can specify the benchmark name, whether to autoload datasets, and a subset of tasks.

```APIDOC
## Matbench Benchmark Initialization

### Description
Initializes a Matbench benchmark instance. You can specify the benchmark name, whether to autoload datasets, and a subset of tasks.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Method Signature
`__init__(self, benchmark='matbench_v0.1', autoload=False, subset=None)`

### Parameters
- **benchmark** (str) - Required - The name of the benchmark. Only supported benchmark currently is "matbench_v0.1", though more will be added in the future.
- **autoload** (bool) - Optional - If True, automatically load the dataset into memory. For a full benchmark, this can take some time. If False, you'll need to load each task with `.load` before you can access the raw data. Defaults to `False`.
- **subset** ([str]) - Optional - A list of task names to use as a subset of a full benchmark. Only the named tasks will be contained in the class. Must correspond to the metadata file defined by the benchmark name. Defaults to `None`.

### Returns
None
```