### Evaluate Proper Lp Calibration Errors

Source: https://github.com/dholzmueller/probmetrics/blob/main/README.md

Demonstrates the instantiation of ProperLpLoss for evaluating calibration errors for any p-norm. It shows examples for L1 and L2 norms.

```python
from probmetrics.metrics import (
  ProperLpLoss,
  BrierLoss,
  OverConfidenceLoss,
  UnderConfidenceLoss,
  TopClassLoss
)

# Evaluate proper Lp calibration errors for any p
lp_loss_l1 = ProperLpLoss(p=1)  # Evaluate E[ \| Y - E[Y|f(X)] \|_1 ] 
lp_loss_l2 = ProperLpLoss(p=2)  # Evaluate E[ \| Y - E[Y|f(X)] \|_2 ] 
```

--------------------------------

### Install Probmetrics Library

Source: https://github.com/dholzmueller/probmetrics/blob/main/README.md

Instructions for installing the Probmetrics package via pip, including optional dependencies for extended functionality.

```bash
pip install probmetrics
pip install probmetrics[extra,dev,dirichletcal]
```

--------------------------------

### Metrics.from_names: Create Metric Collection

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Shows how to create a collection of metrics using the Metrics.from_names factory method. This allows for efficient computation of multiple metrics simultaneously, including standard classification metrics and calibration-specific measures like ECE and RMSCE. The example demonstrates creating a metric set, computing them from true labels and probabilities, and retrieving available metric names.

```python
from probmetrics.metrics import Metrics
import torch

# Create a collection of metrics
metrics = Metrics.from_names([
    'logloss',
    'brier',
    'accuracy',
    'class-error',
    'ece-15',           # Expected Calibration Error with 15 bins
    'rmsce-15',         # Root Mean Squared Calibration Error
    'mce-15',           # Maximum Calibration Error
    'auroc-ovr'         # AUROC one-vs-rest
])

# Compute metrics from labels and probabilities
y_true = torch.tensor([0, 1, 2, 1, 0])
y_probs = torch.tensor([
    [0.8, 0.1, 0.1],
    [0.2, 0.7, 0.1],
    [0.1, 0.2, 0.7],
    [0.3, 0.5, 0.2],
    [0.6, 0.3, 0.1]
])

results = metrics.compute_all_from_labels_probs(y_true, y_probs)
for name, value in results.items():
    print(f"{name}: {value.item():.4f}")

# Get all available metric names
available_metrics = Metrics.get_available_names(metric_type='class')
print(f"Available metrics: {available_metrics[:10]}...")
```

--------------------------------

### MetricsWithCalibration: Refinement and Calibration Error

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Illustrates how to use MetricsWithCalibration to compute refinement error and calibration error using cross-validation. This class measures the improvement gained from post-hoc calibration. The example shows defining base metrics, setting up a calibrator and cross-validation splitter, and computing these errors on provided true labels and probabilities.

```python
from probmetrics.metrics import Metrics, MetricsWithCalibration, LogLoss, BrierLoss, CombinedMetrics
from probmetrics.calibrators import get_calibrator
from probmetrics.splitters import CVSplitter
import torch

# Define base metrics for refinement/calibration error
base_metrics = CombinedMetrics([LogLoss(), BrierLoss()])

# Create calibration-aware metrics with cross-validation
cal_metrics = MetricsWithCalibration(
    metrics=base_metrics,
    calibrator=get_calibrator('temp-scaling', calibrate_with_mixture=True),
    val_splitter=CVSplitter(n_cv=5),
    cal_name='ts-mix',
    random_state=42
)

# Compute refinement and calibration error
y_true = torch.tensor([0, 1, 1, 0, 1, 0, 1, 0, 0, 1])
y_probs = torch.tensor([
    [0.9, 0.1], [0.3, 0.7], [0.2, 0.8], [0.8, 0.2], [0.4, 0.6],
    [0.7, 0.3], [0.3, 0.7], [0.6, 0.4], [0.9, 0.1], [0.2, 0.8]
])

from probmetrics.distributions import CategoricalDirac, CategoricalProbs
results = cal_metrics.compute_all(
    CategoricalDirac(y_true, n_classes=2),
    CategoricalProbs(y_probs)
)

for name, value in results.items():
    print(f"{name}: {value.item():.4f}")
# Output: calib-err_logloss_ts-mix_cv-5, refinement_logloss_ts-mix_cv-5, etc.
```

--------------------------------

### Get List of Available Classification Metric Names

Source: https://github.com/dholzmueller/probmetrics/blob/main/README.md

Shows how to retrieve a list of all available metric names for classification tasks using the Metrics.get_available_names method with the MetricType.CLASS enum.

```python
from probmetrics.metrics import Metrics, MetricType
Metrics.get_available_names(metric_type=MetricType.CLASS)
```

--------------------------------

### LogisticCalibrator: Unified Classification Calibration

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Demonstrates the use of LogisticCalibrator, which unifies calibration strategies. It automatically selects quadratic scaling for binary classification and SMS for multiclass problems. The example shows initialization with specific binary and multiclass types, fitting on both binary and multiclass data, and predicting calibrated probabilities.

```python
from probmetrics.calibrators import LogisticCalibrator
import numpy as np

# Binary classification example
binary_calibrator = LogisticCalibrator(
    binary_type='quadratic',    # Options: 'linear', 'affine', 'quadratic'
    multiclass_type='sms'       # Options: 'svs', 'sms'
)

binary_probas = np.array([[0.3, 0.7], [0.8, 0.2], [0.5, 0.5]])
binary_labels = np.array([1, 0, 1])

binary_calibrator.fit(binary_probas, binary_labels)
calibrated_binary = binary_calibrator.predict_proba(binary_probas)

# Multiclass classification (automatically uses SMS)
multiclass_calibrator = LogisticCalibrator()
multiclass_probas = np.array([
    [0.5, 0.3, 0.2],
    [0.1, 0.8, 0.1],
    [0.2, 0.2, 0.6]
])
multiclass_labels = np.array([0, 1, 2])

multiclass_calibrator.fit(multiclass_probas, multiclass_labels)
calibrated_multi = multiclass_calibrator.predict_proba(multiclass_probas)
```

--------------------------------

### Initialize Probability Distributions

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Demonstrates how to create probability distributions from logits, raw probabilities, and ground truth labels using the library's distribution classes.

```python
import torch
from probmetrics.distributions import CategoricalLogits, CategoricalProbs, CategoricalDirac

logits = torch.tensor([[-1.0, 2.0, 0.5], [1.5, -0.5, 0.0]])
dist_logits = CategoricalLogits(logits)

binary_probs = torch.tensor([0.3, 0.7, 0.9])
binary_dist = CategoricalProbs(binary_probs)

labels = torch.tensor([0, 2, 1])
dirac_dist = CategoricalDirac(labels, n_classes=3)
```

--------------------------------

### Implement Warm-Start CatBoost Calibration

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Shows how to use the WS_CatboostClassifier for Lp calibration error estimation, including integration with MetricsWithCalibration and CVSplitter.

```python
from probmetrics.classifiers import WS_CatboostClassifier
from probmetrics.metrics import MetricsWithCalibration, ProperLpLoss, CombinedMetrics
from probmetrics.splitters import CVSplitter
import numpy as np

ws_catboost = WS_CatboostClassifier(iterations=10, use_init_logits=True, random_state=42)
lp_losses = CombinedMetrics([ProperLpLoss(p=1), ProperLpLoss(p=2)])
metrics = MetricsWithCalibration(lp_losses, calibrator=ws_catboost, val_splitter=CVSplitter(n_cv=5))

probas = np.random.rand(100, 3)
labels = np.random.randint(0, 3, 100)
ws_catboost.fit(probas, labels)
```

--------------------------------

### Initialize Venn-Abers Calibrator

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Demonstrates the initialization of the Venn-Abers calibrator using the factory method or direct class instantiation.

```python
from probmetrics.calibrators import get_calibrator, VennAbersCalibrator

ivap = get_calibrator('ivap')
ivap_ovo = VennAbersCalibrator(use_ovo=True)
```

--------------------------------

### Initialize a Calibrator

Source: https://github.com/dholzmueller/probmetrics/blob/main/README.md

How to instantiate a calibration model using the get_calibrator factory function.

```python
from probmetrics.calibrators import get_calibrator

calib = get_calibrator('logistic')
```

--------------------------------

### Instantiate Various Classification Metrics by Name

Source: https://github.com/dholzmueller/probmetrics/blob/main/README.md

Illustrates how to instantiate a wide range of classification metrics using their string names with the Metrics class. This includes standard metrics like logloss and accuracy, as well as specialized calibration and refinement metrics.

```python
from probmetrics.metrics import Metrics

metrics = Metrics.from_names([
    'logloss',
    'brier',  # for binary, this is 2x the brier from sklearn
    'accuracy', 'class-error',
    'auroc-ovr', # one-vs-rest
    'auroc-ovo-sklearn', # one-vs-one (can be slow!) 
    # calibration metrics
    'ece-15', 'rmsce-15', 'mce-15', 'smece'
    'refinement_logloss_ts-mix_all', 
    'calib-err_logloss_ts-mix_all',
    'refinement_brier_ts-mix_all', 
    'calib-err_brier_ts-mix_all',
    'calib-err_proper-L1-binary-as-1d_WS_CatboostClassifier_all',
    'calib-err_proper-L2-binary-as-1d_WS_CatboostClassifier_all',
    'calib-err_proper-Linf-binary-as-1d_WS_CatboostClassifier_all',
])
```

--------------------------------

### Create Calibrator Instances with get_calibrator

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Demonstrates the factory function to instantiate various calibration methods. It follows the scikit-learn fit/predict API for seamless integration.

```python
from probmetrics.calibrators import get_calibrator
import numpy as np

# Create a temperature scaling calibrator with Laplace smoothing
calibrator = get_calibrator('ts-mix')

# Sample uncalibrated probabilities (n_samples, n_classes)
probas = np.array([
    [0.1, 0.9],
    [0.3, 0.7],
    [0.8, 0.2],
    [0.6, 0.4]
])
labels = np.array([1, 1, 0, 0])

# Fit the calibrator
calibrator.fit(probas, labels)

# Get calibrated probabilities
calibrated_probas = calibrator.predict_proba(probas)
print(calibrated_probas)
```

--------------------------------

### Instantiate Over- and Under-Confidence Loss Metrics

Source: https://github.com/dholzmueller/probmetrics/blob/main/README.md

Shows how to instantiate OverConfidenceLoss and UnderConfidenceLoss metrics, which are designed for binary classification. These can be initialized using string names or by passing existing metric objects.

```python
from probmetrics.metrics import (
  ProperLpLoss,
  BrierLoss,
  OverConfidenceLoss,
  UnderConfidenceLoss,
  TopClassLoss
)

# Evaluate over-confidence and under-confidence 
# (Initialize via string name or by passing a metric object)
over_brier = OverConfidenceLoss.from_name("brier")
under_L1 = UnderConfidenceLoss.from_name("proper-L1")
```

--------------------------------

### Instantiate Top-Class Error Metrics

Source: https://github.com/dholzmueller/probmetrics/blob/main/README.md

Demonstrates the instantiation of TopClassLoss for evaluating top-class errors. It shows how to wrap existing loss metrics like BrierLoss or proper-L1 using either string names or metric objects.

```python
from probmetrics.metrics import (
  ProperLpLoss,
  BrierLoss,
  OverConfidenceLoss,
  UnderConfidenceLoss,
  TopClassLoss
)

# Evaluate top-class error with any accompanying loss
topclass_brier = TopClassLoss(BrierLoss(binary_as_multiclass=False))
topclass_L1 = TopClassLoss.from_name("proper-L1")
```

--------------------------------

### PyTorch Integration for Temperature Scaling

Source: https://github.com/dholzmueller/probmetrics/blob/main/README.md

Demonstrates using CategoricalProbs with PyTorch tensors for calibration, suitable for GPU acceleration. It shows fitting a model and predicting calibrated probabilities.

```python
from probmetrics.distributions import CategoricalProbs
import torch

probas = torch.as_tensor([[0.1, 0.9]])
labels = torch.as_tensor([1])

# if you have logits, you can use CategoricalLogits instead
calib.fit_torch(CategoricalProbs(probas), labels)
result = calib.predict_proba_torch(CategoricalProbs(probas))
calibrated_probas = result.get_probs()
```

--------------------------------

### Compose Advanced Calibration Metric Wrappers

Source: https://github.com/dholzmueller/probmetrics/blob/main/README.md

Illustrates how to compose multiple metric wrappers, such as combining TopClassLoss with UnderConfidenceLoss or OverConfidenceLoss with BrierLoss, to create complex evaluation metrics.

```python
from probmetrics.metrics import (
  ProperLpLoss,
  BrierLoss,
  OverConfidenceLoss,
  UnderConfidenceLoss,
  TopClassLoss
)

# Compose wrappers (e.g., top-class with underconfidence for proper-L1)
under_topclass_l1 = TopClassLoss(UnderConfidenceLoss.from_name("proper-L1"))
over_topclass_brier = TopClassLoss(OverConfidenceLoss(BrierLoss()))
```

--------------------------------

### Fit and Predict with Calibrator

Source: https://github.com/dholzmueller/probmetrics/blob/main/README.md

Demonstrates the standard workflow for fitting a calibrator to probability estimates and labels, then generating calibrated probabilities.

```python
import numpy as np

probas = np.asarray([[0.1, 0.9]])
labels = np.asarray([1])
calib.fit(probas, labels)
calibrated_probas = calib.predict_proba(probas)
```

--------------------------------

### Compute Multiple Refinement and Calibration Metrics

Source: https://github.com/dholzmueller/probmetrics/blob/main/README.md

Shows how to efficiently compute multiple metrics including logloss, refinement logloss, and calibration error using the Metrics class in PyTorch. This is more efficient than individual computations.

```python
import torch
from probmetrics.metrics import Metrics

# compute multiple metrics at once 
# this is more efficient than computing them individually
metrics = Metrics.from_names(['logloss', 
                              'refinement_logloss_ts-mix_all',
                              'calib-err_logloss_ts-mix_all'])
y_true = torch.tensor(...)
y_logits = torch.tensor(...)
results = metrics.compute_all_from_labels_logits(y_true, y_logits)
print(results['refinement_logloss_ts-mix_all'].item())
```

--------------------------------

### ProbMetrics Calibration

Source: https://github.com/dholzmueller/probmetrics/blob/main/README.md

This section demonstrates how to initialize and use ProbMetrics for evaluating calibration errors, including defining metrics, setting up a calibrator, and computing results.

```APIDOC
## ProbMetrics Calibration API

### Description
This API allows for the evaluation of calibration errors in probabilistic predictions using various metrics and a customizable recalibration process.

### Method
N/A (This is a library usage example, not a direct API endpoint)

### Endpoint
N/A

### Parameters
N/A

### Request Example
```python
from probmetrics.metrics import MetricsWithCalibration, CombinedMetrics, ProperLpLoss, OverConfidenceLoss, UnderConfidenceLoss, BrierLoss
from probmetrics.classifiers import WS_CatboostClassifier, WS_LGBMClassifier
from probmetrics.splitters import CVSplitter
import torch

# Define individual loss metrics
loss_l1 = ProperLpLoss(p=1)
loss_l2 = ProperLpLoss(p=2)

# Define combined metrics
combined_losses = CombinedMetrics([
    ProperLpLoss(p=1),
    OverConfidenceLoss.from_name("brier"),
    OverConfidenceLoss.from_name("proper-L1"),
    UnderConfidenceLoss.from_name("proper-L1"),
    UnderConfidenceLoss(BrierLoss()),
    BrierLoss()
])

# Initialize MetricsWithCalibration with a specific loss and calibrator
metrics_single = MetricsWithCalibration(
    loss=loss_l2,
    calibrator=WS_CatboostClassifier(), # Recommended calibrator
    val_splitter=CVSplitter(n_cv=5) # Cross-validation splitter
)

# Initialize MetricsWithCalibration with combined losses and a different calibrator
metrics_combined = MetricsWithCalibration(
    loss=combined_losses,
    calibrator=WS_LGBMClassifier(),
    val_splitter=CVSplitter(n_cv=5)
)

# Example true labels and probabilities (replace with your actual data)
y_true = torch.tensor([0, 1, 0, 1])
y_prob = torch.tensor([[0.1, 0.9], [0.8, 0.2], [0.6, 0.4], [0.3, 0.7]])

# Compute all metrics
results = metrics_single.compute_all_from_labels_probs(y_true, y_prob)
print(results)

results_combined = metrics_combined.compute_all_from_labels_probs(y_true, y_prob)
print(results_combined)
```

### Response
#### Success Response (200)
- **results** (dict) - A dictionary containing the computed calibration metrics.

#### Response Example
```json
{
  "metric_name_1": value1,
  "metric_name_2": value2
}
```

### Error Handling
- If `y_true` or `y_prob` are not in the expected format, a `TypeError` or `ValueError` may be raised.
- Issues with the calibrator or splitter will raise their respective exceptions.
```

--------------------------------

### Perform Temperature Scaling with TemperatureScalingCalibrator

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Shows how to use the TemperatureScalingCalibrator with PyTorch tensors, utilizing bisection search for optimization.

```python
from probmetrics.calibrators import TemperatureScalingCalibrator
from probmetrics.distributions import CategoricalProbs
import torch

# Create temperature scaling calibrator
ts_calibrator = TemperatureScalingCalibrator(
    opt='bisection',
    max_bisection_steps=30,
    use_inv_temp=True
)

# Prepare data as PyTorch tensors
probas = torch.tensor([
    [0.1, 0.7, 0.2],
    [0.3, 0.5, 0.2],
    [0.8, 0.1, 0.1]
])
labels = torch.tensor([1, 1, 0])

# Fit using PyTorch interface
ts_calibrator.fit_torch(CategoricalProbs(probas), labels)

# Apply calibration
result = ts_calibrator.predict_proba_torch(CategoricalProbs(probas))
calibrated_probs = result.get_probs()
print(f"Inverse temperature: {ts_calibrator.invtemp_}")
print(f"Calibrated probabilities:\n{calibrated_probs}")
```

--------------------------------

### Evaluate Calibration Errors with MetricsWithCalibration

Source: https://github.com/dholzmueller/probmetrics/blob/main/README.md

Demonstrates how to evaluate calibration errors by wrapping a loss function with a calibrator and a cross-validation splitter. It supports both single loss functions and combined metrics for efficient computation.

```python
from probmetrics.metrics import MetricsWithCalibration, CombinedMetrics
from probmetrics.classifiers import WS_CatboostClassifier, WS_LGBMClassifier
from probmetrics.splitters import CVSplitter

loss = ProperLpLoss(p=2)
metrics = MetricsWithCalibration(loss, calibrator=WS_CatboostClassifier(), val_splitter=CVSplitter(n_cv=5))

combined_losses = CombinedMetrics([ProperLpLoss(p=1), BrierLoss()])
metrics = MetricsWithCalibration(combined_losses, calibrator=WS_LGBMClassifier(), val_splitter=CVSplitter(n_cv=5))

results = metrics.compute_all_from_labels_probs(y_true, y_prob)
```

--------------------------------

### Fit and Predict Probabilities with SMS Calibrator

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Demonstrates fitting an SMS calibrator with probability and label data, and then predicting calibrated probabilities. This is a fundamental step for probability calibration.

```python
sms_calibrator.fit(probas, labels)
calibrated = sms_calibrator.predict_proba(probas)
print(f"Calibrated probabilities:\n{calibrated}")
```

--------------------------------

### Binary Classification Calibration with IVAP

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Demonstrates how to use the IVAP calibrator for binary classification tasks. It takes raw probabilities and true labels as input, fits the calibrator, and then predicts calibrated probabilities.

```python
import numpy as np
import probmetrics as pm

ivap = pm.get_calibrator('ivap')

binary_probs = np.array([
    [0.8, 0.2],
    [0.3, 0.7],
    [0.5, 0.5],
    [0.9, 0.1]
])
binary_labels = np.array([0, 1, 1, 0])

ivap.fit(binary_probs, binary_labels)
calibrated = ivap.predict_proba(binary_probs)
print(f"IVAP calibrated:\n{calibrated}")
```

--------------------------------

### Define Calibration Metrics

Source: https://github.com/dholzmueller/probmetrics/blob/main/README.md

Initializes a list of calibration metrics using predefined names. These metrics allow for estimating L1 and L2 calibration errors, including top-class specific variations.

```python
metrics = Metrics.from_names([
    'proper-L1-binary-as-1d',
    'proper-L2',
    "topclass-proper-L1-binary-as-1d",
    "topclass-under-proper-L1-binary-as-1d",
    "topclass-over-proper-L1-binary-as-1d"
])
```

--------------------------------

### Analyze Top-Class Calibration with TopClassLoss

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Wraps existing metrics to focus exclusively on the most confident class. Useful for multiclass calibration where only the top prediction matters.

```python
from probmetrics.metrics import TopClassLoss, ProperLpLoss, BrierLoss
from probmetrics.distributions import CategoricalDirac, CategoricalProbs
import torch

# Create top-class L1 calibration metric
topclass_l1 = TopClassLoss(ProperLpLoss(p=1))

# Alternative: create from string name
topclass_l1_alt = TopClassLoss.from_name("proper-L1-binary-as-1d")

# Top-class Brier loss
topclass_brier = TopClassLoss(BrierLoss(binary_as_multiclass=False))

# Multiclass predictions
y_true = torch.tensor([0, 1, 2, 1])
y_probs = torch.tensor([
    [0.7, 0.2, 0.1],  # Top class: 0, correct
    [0.3, 0.5, 0.2],  # Top class: 1, correct
    [0.4, 0.4, 0.2],  # Top class: 0, incorrect
    [0.2, 0.6, 0.2]   # Top class: 1, correct
])

# f_x represents original predictions for calibration reference
f_x = CategoricalProbs(y_probs)

result = topclass_l1.compute(
    CategoricalDirac(y_true, n_classes=3),
    CategoricalProbs(y_probs),
    f_x=f_x
)
print(f"Top-class L1 error: {result.item():.4f}")
```

--------------------------------

### Metrics.from_names API

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Factory method to create and compute a collection of classification and calibration metrics.

```APIDOC
## Metrics.from_names

### Description
Creates a metrics collection from a list of strings and computes them efficiently.

### Parameters
- **names** (list) - List of metric names (e.g., 'logloss', 'ece-15', 'auroc-ovr').

### Request Example
metrics = Metrics.from_names(['logloss', 'brier', 'ece-15'])
results = metrics.compute_all_from_labels_probs(y_true, y_probs)
```

--------------------------------

### SVSCalibrator: Structured Vector Scaling Calibration

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Initializes and fits the SVSCalibrator for multiclass probability calibration. It uses a structured vector scaling method for improved computational efficiency over SMS. The code shows how to set parameters like penalty and lambda values, fit the calibrator, and retrieve learned parameters like 'v' (diagonal scaling) and 'b' (bias).

```python
from probmetrics.calibrators import SVSCalibrator
import numpy as np

# Create SVS calibrator
svs_calibrator = SVSCalibrator(
    penalty='ridge',
    lambda_intercept=1.0,
    lambda_diagonal=1.0,
    opt='bfgs'
)

# Fit on validation data
probas = np.array([
    [0.6, 0.3, 0.1],
    [0.2, 0.7, 0.1],
    [0.1, 0.2, 0.7]
])
labels = np.array([0, 1, 2])

svs_calibrator.fit(probas, labels)
calibrated = svs_calibrator.predict_proba(probas)
print(f"Diagonal scaling (v): {svs_calibrator.v}")
print(f"Bias (b): {svs_calibrator.b}")
```

--------------------------------

### TemperatureScalingCalibrator

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Implementation of temperature scaling for probability calibration, supporting bisection search and PyTorch tensors.

```APIDOC
## POST /calibrators/temperature-scaling

### Description
Performs temperature scaling on logits to minimize cross-entropy loss. Optimized via bisection search or L-BFGS.

### Method
POST

### Parameters
#### Request Body
- **opt** (string) - Optional - Optimization method: 'bisection', 'lbfgs', 'lbfgs_line_search'.
- **max_bisection_steps** (int) - Optional - Number of iterations for bisection.
- **use_inv_temp** (boolean) - Optional - Whether to optimize the inverse temperature parameter.

### Request Example
{
  "opt": "bisection",
  "max_bisection_steps": 30
}

### Response
#### Success Response (200)
- **invtemp_** (float) - The optimized inverse temperature value.
```

--------------------------------

### Represent Categorical Distributions with CategoricalProbs and CategoricalLogits

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Utility classes to handle probability vectors and raw logits. Provides methods to extract modes, class counts, and convert between representations.

```python
from probmetrics.distributions import CategoricalProbs, CategoricalLogits, CategoricalDirac
import torch

# Create from probabilities
probs = torch.tensor([
    [0.2, 0.5, 0.3],
    [0.8, 0.1, 0.1]
])
dist_probs = CategoricalProbs(probs)

print(f"Probabilities:\n{dist_probs.get_probs()}")
print(f"Logits:\n{dist_probs.get_logits()}")
print(f"Modes: {dist_probs.get_modes()}")  # Most likely class
print(f"Number of classes: {dist_probs.get_n_classes()}")
print(f"Number of samples: {dist_probs.get_n_samples()}")
```

--------------------------------

### Multiclass Classification Calibration with IVAP-OVR

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Illustrates the use of the IVAP one-vs-rest (OVR) wrapper for multiclass classification calibration. This approach is suitable when dealing with more than two classes.

```python
import numpy as np
import probmetrics as pm

ivap_ovr = pm.get_calibrator('ivap-ovr')
multiclass_probs = np.array([
    [0.7, 0.2, 0.1],
    [0.2, 0.6, 0.2],
    [0.1, 0.3, 0.6]
])
multiclass_labels = np.array([0, 1, 2])

ivap_ovr.fit(multiclass_probs, multiclass_labels)
calibrated_mc = ivap_ovr.predict_proba(multiclass_probs)
print(f"IVAP-OVR calibrated:
{calibrated_mc}")
```

--------------------------------

### POST /calibrator/fit

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Fits a calibration model to the provided probability estimates and ground truth labels.

```APIDOC
## POST /calibrator/fit

### Description
Trains a calibration model using provided probability distributions and target labels. This method aligns predicted probabilities with observed frequencies.

### Method
POST

### Endpoint
/calibrator/fit

### Parameters
#### Request Body
- **probs** (array) - Required - A 2D array of shape (n_samples, n_classes) containing raw probability estimates.
- **labels** (array) - Required - A 1D array of shape (n_samples,) containing ground truth class labels.

### Request Example
{
  "probs": [[0.8, 0.2], [0.3, 0.7]],
  "labels": [0, 1]
}

### Response
#### Success Response (200)
- **status** (string) - Confirmation of successful model fitting.

#### Response Example
{
  "status": "success"
}
```

--------------------------------

### POST /calibrator/predict_proba

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Applies a fitted calibration model to transform raw probability estimates into calibrated probabilities.

```APIDOC
## POST /calibrator/predict_proba

### Description
Transforms raw model outputs into calibrated probabilities using the parameters learned during the fit phase.

### Method
POST

### Endpoint
/calibrator/predict_proba

### Parameters
#### Request Body
- **probs** (array) - Required - A 2D array of shape (n_samples, n_classes) to be calibrated.

### Request Example
{
  "probs": [[0.9, 0.1]]
}

### Response
#### Success Response (200)
- **calibrated_probs** (array) - The adjusted probability distribution.

#### Response Example
{
  "calibrated_probs": [[0.85, 0.15]]
}
```

--------------------------------

### Factory: get_calibrator

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

A factory function to instantiate various calibration models using a common scikit-learn compatible interface.

```APIDOC
## GET /calibrators/factory

### Description
Creates a calibrator instance by name. Supports methods like 'ts-mix', 'sms', 'svs', 'isotonic', and 'ivap'.

### Method
GET (Factory Pattern)

### Parameters
#### Query Parameters
- **method_name** (string) - Required - The identifier for the calibration method (e.g., 'ts-mix', 'sms', 'svs', 'isotonic').

### Request Example
get_calibrator('ts-mix')

### Response
#### Success Response (200)
- **calibrator** (Object) - A fitted or unfitted calibrator object following the scikit-learn fit/predict API.
```

--------------------------------

### Configure SMSCalibrator for Multiclass Classification

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Configures the Structured Matrix Scaling calibrator with specific regularization parameters and optimization settings.

```python
from probmetrics.calibrators import SMSCalibrator
import numpy as np

# Create SMS calibrator with ridge regularization
sms_calibrator = SMSCalibrator(
    penalty='ridge',
    rho=1.0,
    tau=1.0,
    lambda_intercept=1.0,
    lambda_diagonal=1.0,
    lambda_off_diagonal=1.0,
    opt='bfgs',
    max_iter=500,
    tol=1e-5
)

# Multiclass probabilities
probas = np.array([
    [0.7, 0.2, 0.1],
    [0.3, 0.5, 0.2],
    [0.1, 0.3, 0.6],
    [0.4, 0.4, 0.2]
])
labels = np.array([0, 1, 2, 0])
```

--------------------------------

### Estimate Calibration Error with ProperLpLoss

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Computes the variational form of Lp calibration error. This snippet demonstrates combining multiple Lp losses and using a calibrator for rigorous assessment.

```python
from probmetrics.metrics import ProperLpLoss, MetricsWithCalibration, CombinedMetrics
from probmetrics.classifiers import WS_CatboostClassifier
from probmetrics.splitters import CVSplitter
from probmetrics.distributions import CategoricalDirac, CategoricalProbs
import torch

# Create L1 and L2 proper losses
l1_loss = ProperLpLoss(p=1, binary_as_multiclass=False)
l2_loss = ProperLpLoss(p=2, binary_as_multiclass=False)
linf_loss = ProperLpLoss(p=float('inf'), binary_as_multiclass=False)

# Combine losses for efficient computation
combined_losses = CombinedMetrics([l1_loss, l2_loss, linf_loss])

# Use with calibration for proper calibration error estimation
metrics_with_cal = MetricsWithCalibration(
    combined_losses,
    calibrator=WS_CatboostClassifier(),
    val_splitter=CVSplitter(n_cv=5)
)

y_true = torch.tensor([0, 1, 1, 0, 1, 0, 1, 0, 0, 1])
y_probs = torch.tensor([
    [0.9, 0.1], [0.3, 0.7], [0.2, 0.8], [0.8, 0.2], [0.4, 0.6],
    [0.7, 0.3], [0.3, 0.7], [0.6, 0.4], [0.9, 0.1], [0.2, 0.8]
])

results = metrics_with_cal.compute_all(
    CategoricalDirac(y_true, n_classes=2),
    CategoricalProbs(y_probs)
)
print(results)
```

--------------------------------

### SVSCalibrator API

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Structured Vector Scaling (SVS) calibrator for multiclass problems using diagonal scaling.

```APIDOC
## SVSCalibrator

### Description
A faster alternative to SMS for multiclass calibration using diagonal scaling with a structured softmax scheme.

### Parameters
- **penalty** (str) - 'ridge' or 'lasso' regularization.
- **lambda_intercept** (float) - Regularization strength for intercept.
- **lambda_diagonal** (float) - Regularization strength for diagonal scaling.
- **opt** (str) - Optimization algorithm (e.g., 'bfgs').

### Request Example
svs_calibrator = SVSCalibrator(penalty='ridge', lambda_intercept=1.0, lambda_diagonal=1.0, opt='bfgs')
svs_calibrator.fit(probas, labels)
calibrated = svs_calibrator.predict_proba(probas)
```

--------------------------------

### MetricsWithCalibration API

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Computes refinement and calibration error using cross-validation.

```APIDOC
## MetricsWithCalibration

### Description
Evaluates how much a model's predictions can be improved through post-hoc calibration using cross-validation.

### Parameters
- **metrics** (object) - Base metrics to evaluate.
- **calibrator** (object) - Calibration method to apply.
- **val_splitter** (object) - Cross-validation strategy.

### Request Example
cal_metrics = MetricsWithCalibration(metrics=base_metrics, calibrator=calibrator, val_splitter=splitter)
results = cal_metrics.compute_all(y_true, y_probs)
```

--------------------------------

### Calculate Calibration Errors

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Computes ECE, RMSCE, and MCE metrics using the CalibrationError class with configurable binning and norms.

```python
from probmetrics.metrics import CalibrationError, Metrics
from probmetrics.distributions import CategoricalDirac, CategoricalProbs
import torch

ece = CalibrationError(n_bins=15, norm='l1')
rmsce = CalibrationError(n_bins=15, norm='l2')
mce = CalibrationError(n_bins=15, norm='max')

y_true = torch.tensor([0, 1, 1, 0])
y_probs = torch.tensor([[0.9, 0.1], [0.2, 0.8], [0.3, 0.7], [0.8, 0.2]])

val = ece.compute(CategoricalDirac(y_true, n_classes=2), CategoricalProbs(y_probs))
```

--------------------------------

### Perform Cross-Validation Splitting

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Utilizes CVSplitter for stratified K-fold cross-validation and AllSplitter for research-based full-dataset evaluation.

```python
from probmetrics.splitters import CVSplitter, AllSplitter
from probmetrics.distributions import CategoricalDirac
import torch

cv_splitter = CVSplitter(n_cv=5)
labels = torch.tensor([0, 0, 1, 1, 2, 2, 0, 1, 2, 0])
y_dist = CategoricalDirac(labels, n_classes=3)

splits = list(cv_splitter.get_splits(y_dist, random_state=42))
all_splitter = AllSplitter()
all_splits = list(all_splitter.get_splits(y_dist))
```

--------------------------------

### LogisticCalibrator API

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Unified interface for selecting the optimal calibration method based on classification type.

```APIDOC
## LogisticCalibrator

### Description
Automatically selects the best calibration method: quadratic scaling for binary and SMS for multiclass.

### Parameters
- **binary_type** (str) - 'linear', 'affine', or 'quadratic'.
- **multiclass_type** (str) - 'svs' or 'sms'.

### Request Example
calibrator = LogisticCalibrator(binary_type='quadratic', multiclass_type='sms')
calibrator.fit(probas, labels)
calibrated = calibrator.predict_proba(probas)
```

--------------------------------

### SMSCalibrator

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Structured Matrix Scaling (SMS) for multiclass classification, applying regularized matrix transformations to logits.

```APIDOC
## POST /calibrators/sms

### Description
Applies a regularized matrix transformation to logits for multiclass calibration.

### Method
POST

### Parameters
#### Request Body
- **penalty** (string) - Optional - Regularization type: 'ridge', 'lasso', 'mcp', 'group_lasso'.
- **lambda_diagonal** (float) - Optional - Strength for diagonal elements.
- **lambda_off_diagonal** (float) - Optional - Strength for off-diagonal elements.
- **opt** (string) - Optional - Optimizer: 'bfgs' or 'saga'.

### Request Example
{
  "penalty": "ridge",
  "lambda_diagonal": 1.0,
  "opt": "bfgs"
}

### Response
#### Success Response (200)
- **status** (string) - Calibration completion status.
```

--------------------------------

### Isolate Confidence Errors with OverConfidenceLoss and UnderConfidenceLoss

Source: https://context7.com/dholzmueller/probmetrics/llms.txt

Measures specific types of miscalibration by isolating over-confident or under-confident predictions. Can be combined with TopClassLoss for multiclass scenarios.

```python
from probmetrics.metrics import OverConfidenceLoss, UnderConfidenceLoss, ProperLpLoss, TopClassLoss
from probmetrics.distributions import CategoricalDirac, CategoricalProbs
import torch

# Create over-confidence and under-confidence metrics
over_l1 = OverConfidenceLoss(ProperLpLoss(p=1, binary_as_multiclass=False))
under_l1 = UnderConfidenceLoss(ProperLpLoss(p=1, binary_as_multiclass=False))

# Alternative: from string names
over_brier = OverConfidenceLoss.from_name("brier")
under_brier = UnderConfidenceLoss.from_name("brier")

# For multiclass, combine with TopClassLoss
topclass_over_l1 = TopClassLoss(OverConfidenceLoss(ProperLpLoss(p=1)))
topclass_under_l1 = TopClassLoss(UnderConfidenceLoss(ProperLpLoss(p=1)))

# Binary example
y_true = torch.tensor([0, 1, 1, 0])
y_probs = torch.tensor([
    [0.8, 0.2],  # Confident and correct
    [0.3, 0.7],  # Under-confident
    [0.1, 0.9],  # Over-confident
    [0.6, 0.4]   # Slightly wrong
])

f_x = CategoricalProbs(y_probs)

over_result = over_l1.compute(
    CategoricalDirac(y_true, n_classes=2),
    CategoricalProbs(y_probs),
    f_x=f_x
)
print(f"Over-confidence error: {over_result.item():.4f}")
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.