### Cross-Validation Setup and Execution in Python

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/notebooks/plotting_diagnostics_demo.ipynb

This snippet demonstrates the setup and execution of cross-validation using scikit-learn's cross_validate function. It initializes a KFold splitter and defines scoring metrics. The code then performs cross-validation on both a 'full' and a 'simple' model, measuring the time taken for each.

```python
splitter = KFold(n_splits=5, shuffle=True, random_state=42)
scoring = ("deviance", "nll", "gini")

# ── CV: full model ───────────────────────────────────────────
cv_full = cross_validate(
    model_full_cv,
    X_train,
    y_train,
    cv=splitter,
    sample_weight=exp_train,
    fit_mode="fit_reml",
    scoring=scoring,
    return_oof=True,
)
print(f"Full model CV: {time.perf_counter() - t0:.1f}s")

t1 = time.perf_counter()
# ── CV: simple model ─────────────────────────────────────────
cv_simple = cross_validate(
    model_simple_cv,
    X_train,
    y_train,
    cv=splitter,
    sample_weight=exp_train,
    fit_mode="fit_reml",
    scoring=scoring,
    return_oof=True,
)
print(f"Simple model CV: {time.perf_counter() - t1:.1f}s")
print(f"Total CV time: {time.perf_counter() - t0:.1f}s")
```

--------------------------------

### Install SuperGLM via pip

Source: https://github.com/strudeldoodles/superglm/blob/master/README.md

Commands to install the SuperGLM package directly from the GitHub repository, including an option for all dependencies.

```bash
pip install git+https://github.com/StrudelDoodleS/superglm.git
```

```bash
pip install "superglm[all] @ git+https://github.com/StrudelDoodleS/superglm.git"
```

--------------------------------

### Execute Different Fitting Modes

Source: https://github.com/strudeldoodles/superglm/blob/master/README.md

Examples of standard penalised fitting, exact REML, and discrete/fREML-style REML for large datasets.

```python
# Standard penalised fit
model = SuperGLM(
    family="poisson",
    penalty="group_elastic_net",
    selection_penalty=0.01,
    spline_penalty=0.1,
    features=features,
)
model.fit(df, y, sample_weight=exposure)
```

```python
# Exact REML
model = SuperGLM(family="poisson", selection_penalty=0, features=features)
model.fit_reml(df, y, sample_weight=exposure, max_reml_iter=30)
```

```python
# Discrete / fREML-style REML
model = SuperGLM(
    family="poisson",
    selection_penalty=0,
    discrete=True,
    n_bins=256,
    features=features,
)
model.fit_reml(df, y, sample_weight=exposure, max_reml_iter=30)
```

--------------------------------

### Configure Ordered Categorical Features in SuperGLM

Source: https://context7.com/strudeldoodles/superglm/llms.txt

Demonstrates the use of `OrderedCategorical` features in SuperGLM for variables with a natural order, allowing for smooth transitions between levels. Examples include using spline bases over ordered categories and specifying explicit numeric values for each category.

```python
from superglm import OrderedCategorical

# Spline basis over ordered categories
ordered_cat = OrderedCategorical(
    order=["A", "B", "C", "D", "E", "F"],
    basis="spline",
    n_knots=3,
)

# With explicit numeric values
ordered_explicit = OrderedCategorical(
    values={"A": 0.0, "B": 0.2, "C": 0.4, "D": 0.6, "E": 0.8, "F": 1.0},
    basis="spline",
)

model = SuperGLM(
    family="poisson",
    features={
        "DamageGrade": OrderedCategorical(order=["Low", "Med", "High"], basis="spline"),
        "DrivAge": Spline(k=10),
    },
)
```

--------------------------------

### Comparing SuperGLM Model Performance Metrics in Python

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/notebooks/plotting_diagnostics_demo.ipynb

This Python snippet generates a summary table comparing the 'full' and 'simple' SuperGLM models across key metrics (deviance, nll, gini). It calculates the mean and median for each metric for both models, as well as the difference between them. The output includes an interpretation guide for understanding the comparison results.

```python
# ── Summary + paired comparison table ─────────────────────────
metrics = ["deviance", "nll", "gini"]
rows = []
for metric in metrics:
    f = cv_full.fold_scores[metric]
    s = cv_simple.fold_scores[metric]
    delta = f - s  # full minus simple

    rows.append(
        {
            "metric": metric,
            "better_when": "higher" if metric == "gini" else "lower",
            "full_mean": f"{f.mean():.4f}",
            "simple_mean": f"{s.mean():.4f}",
            "full_minus_simple_mean": f"{delta.mean():+.4f}",
            "full_median": f"{f.median():.4f}",
            "simple_median": f"{s.median():.4f}",
            "full_minus_simple_median": f"{delta.median():+.4f}",
            "delta_std": f"{delta.std():.4f}",
        }
    )

comparison = pd.DataFrame(rows)
print("=== Model comparison (full minus simple) ===")
print(comparison.to_string(index=False))
print()
print("Interpretation:")
print("  deviance/nll: lower is better, so negative full_minus_simple favors full")
print("  gini: higher is better, so positive full_minus_simple favors full")
print("  Very small deltas mean the global CV picture is basically a tie.")
```

--------------------------------

### Regularization Path Fitting with SuperGLM

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/guide/fitting.md

Fits a sequence of models along a regularization path from high to low regularization using warm starts. It can automatically generate a lambda sequence or accept a custom one.

```python
from superglm import PathResult

model = SuperGLM(
    family=Poisson(),
    penalty=GroupLasso(),
    features={
        "DrivAge": Spline(k=14),
        "Area": Categorical(base="most_exposed"),
    },
)
result = model.fit_path(df, y, sample_weight=exposure, n_lambda=50, lambda_ratio=1e-3)

result.lambda_seq       # (50,) decreasing lambda values
result.coef_path        # (50, p) coefficients at each lambda
result.deviance_path    # (50,) deviance at each lambda
result.n_iter_path      # (50,) PIRLS iterations per lambda
```

```python
result = model.fit_path(df, y, sample_weight=exposure, lambda_seq=[1.0, 0.1, 0.01])
```

--------------------------------

### SuperGLM Distribution Families and Link Functions

Source: https://context7.com/strudeldoodles/superglm/llms.txt

Details the supported distribution families and link functions in SuperGLM for various insurance modeling tasks. Includes examples for Poisson, Gamma, Tweedie, Negative Binomial, Binomial, and Gaussian families, along with custom link functions.

```python
from superglm import (
    SuperGLM, Poisson, Gamma, Tweedie, NegativeBinomial, Binomial, Gaussian,
    LogLink, LogitLink, IdentityLink, InverseLink, ProbitLink
)

# Poisson - Claim frequency modeling
model_freq = SuperGLM(family="poisson", splines=["DrivAge"])
model_freq.fit(df, claim_counts, sample_weight=exposure)

# Gamma - Claim severity modeling
model_sev = SuperGLM(family="gamma", splines=["DrivAge"])
model_sev.fit(df, claim_amounts, sample_weight=claim_counts)

# Tweedie - Pure premium (frequency x severity)
model_pp = SuperGLM(family=Tweedie(p=1.5), splines=["DrivAge"])
model_pp.fit(df, losses, sample_weight=exposure)

# Profile-estimate Tweedie power
result = model_pp.estimate_p(df, losses, sample_weight=exposure, p_range=(1.1, 1.9))
print(f"Estimated p: {result.p_hat:.3f}")
print(f"95% CI: {result.ci()}")

# Negative Binomial - Overdispersed frequency
model_nb = SuperGLM(family=NegativeBinomial(theta=1.0), splines=["DrivAge"])
model_nb.fit(df, claim_counts, sample_weight=exposure)

# Profile-estimate theta
theta_result = model_nb.estimate_theta(df, claim_counts, sample_weight=exposure)
print(f"Estimated theta: {theta_result.theta_hat:.3f}")

# Binomial - Binary classification
model_bin = SuperGLM(family="binomial", splines=["DrivAge"])
model_bin.fit(df, binary_target)
probabilities = model_bin.predict(df)  # Returns P(Y=1)

# Custom link function
model_probit = SuperGLM(family="binomial", link=ProbitLink(), splines=["DrivAge"])
```

--------------------------------

### POST /fit_path

Source: https://github.com/strudeldoodles/superglm/blob/master/README.md

Fits a sequence of models across a range of regularisation values using warm starts.

```APIDOC
## POST /fit_path

### Description
Computes the regularisation path by fitting the model across a sequence of lambda values.

### Parameters
- **df** (DataFrame) - Required - Input features.
- **y** (Series) - Required - Target variable.
- **n_lambda** (int) - Optional - Number of lambda values in the sequence.
- **lambda_ratio** (float) - Optional - Ratio between the smallest and largest lambda.
- **lambda_seq** (list) - Optional - Custom sequence of lambda values.

### Response
- **lambda_seq** (array) - The sequence of lambda values used.
- **coef_path** (array) - Coefficients at each lambda step.
- **deviance_path** (array) - Deviance values per lambda.
```

--------------------------------

### Initialize and Fit SuperGLM Models

Source: https://github.com/strudeldoodles/superglm/blob/master/README.md

Demonstrates how to initialize a SuperGLM model using auto-detection or explicit feature configuration, and how to fit the model to data.

```python
from superglm import SuperGLM

model = SuperGLM(
    family="poisson",
    penalty="group_lasso",
    selection_penalty=0.01,
    splines=["DrivAge", "VehAge", "BonusMalus"],
    n_knots=10,
)
model.fit(df, y, sample_weight=exposure)
predictions = model.predict(df)
```

```python
from superglm import SuperGLM, Spline, Categorical, Numeric

model = SuperGLM(
    family="poisson",
    penalty="group_lasso",
    selection_penalty=0.01,
    features={
        "DrivAge": Spline(kind="bs", k=14),
        "VehAge": Spline(kind="cr", k=10),
        "BonusMalus": Spline(kind="ns", k=10),
        "Area": Categorical(base="most_exposed"),
        "LogDensity": Numeric(),
    },
)
model.fit(df, y, sample_weight=exposure)
```

--------------------------------

### Initialize GLM Families using Convenience Constructors

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/api/families.md

Demonstrates how to use the superglm.families module to instantiate common GLM distribution objects. This includes both simple distributions and parameterized ones like NegativeBinomial and Tweedie.

```python
from superglm import families

families.poisson()              # Poisson()
families.gaussian()             # Gaussian()
families.gamma()                # Gamma()
families.binomial()             # Binomial()
families.nb2(theta=1.0)         # NegativeBinomial(theta=1.0)
families.nb2(theta="auto")      # NegativeBinomial(theta="auto")
families.tweedie(p=1.5)         # Tweedie(p=1.5)
```

--------------------------------

### SuperGLM Model Fitting

Source: https://github.com/strudeldoodles/superglm/blob/master/README.md

Demonstrates how to initialize a SuperGLM model with specific families, penalties, and feature constraints.

```APIDOC
## POST /model/fit

### Description
Fits a Generalized Linear Model using the specified family and penalty structure. Supports feature-level constraints like monotonicity.

### Method
POST

### Parameters
#### Request Body
- **family** (string/object) - Required - The distribution family (e.g., 'poisson', 'binomial', 'Tweedie(p=1.5)').
- **penalty** (object) - Optional - The regularization penalty (e.g., GroupLasso).
- **features** (object) - Optional - Dictionary mapping feature names to constraint types (e.g., Spline, Categorical).

### Request Example
{
  "family": "Tweedie(p=1.5)",
  "penalty": "GroupLasso(lambda1=0.01)",
  "features": {
    "BonusMalus": "Spline(kind='bs', k=14, monotone='increasing')"
  }
}

### Response
#### Success Response (200)
- **model** (object) - The fitted model instance.
```

--------------------------------

### Regularization Path Fitting

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/guide/fitting.md

Fits a sequence of models across a range of regularization parameters using warm starts.

```APIDOC
## POST /fit_path

### Description
Computes a sequence of models from high to low regularization, useful for cross-validation and model selection.

### Method
POST

### Parameters
#### Request Body
- **n_lambda** (integer) - Optional - Number of lambda values in the sequence.
- **lambda_ratio** (float) - Optional - Ratio between the smallest and largest lambda.
- **lambda_seq** (array) - Optional - Custom sequence of lambda values.

### Response
#### Success Response (200)
- **lambda_seq** (array) - The sequence of lambda values used.
- **coef_path** (array) - The coefficient matrix for the path.
- **deviance_path** (array) - The deviance values for the path.
```

--------------------------------

### Perform Binary Classification with SuperGLM

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/guide/families.md

Demonstrates how to initialize a SuperGLM model with a binomial family and custom link functions. It also shows the use of the sklearn-compatible SuperGLMClassifier for generating hard labels and class probabilities.

```python
from superglm import SuperGLM, ProbitLink, CloglogLink, SuperGLMClassifier

# Basic binomial model
model = SuperGLM(family="binomial", selection_penalty=0)
model.fit(df, y)
probabilities = model.predict(df)

# Custom link functions
model_probit = SuperGLM(family="binomial", link=ProbitLink(), selection_penalty=0)
model_cloglog = SuperGLM(family="binomial", link=CloglogLink(), selection_penalty=0)

# Sklearn-compatible classifier
clf = SuperGLMClassifier(selection_penalty=0, spline_features=["age"])
clf.fit(df, y)
labels = clf.predict(df)
probs = clf.predict_proba(df)
```

--------------------------------

### Generate Loss Ratio Chart (Python)

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/notebooks/plotting_diagnostics_demo.ipynb

Generates a chart comparing observed vs. predicted loss ratios per quantile bin, with an optional exposure share overlay. It can bin by predicted value or a specific feature. Dependencies include matplotlib for plotting.

```python
t0 = time.perf_counter()
result = loss_ratio_chart(
    y_obs=y,
    y_pred=mu,
    exposure=exposure,
    n_bins=20,
)
plt.gcf().set_size_inches(10, 5)
plt.show()
print(f"[{time.perf_counter() - t0:.2f}s]")
```

```python
t0 = time.perf_counter()
result = loss_ratio_chart(
    y_obs=y,
    y_pred=mu,
    exposure=exposure,
    n_bins=15,
    feature_values=X["DrivAge"].to_numpy(),
    feature_name="DrivAge",
)
plt.gcf().set_size_inches(10, 5)
plt.show()
print(f"[{time.perf_counter() - t0:.2f}s]")
```

--------------------------------

### Fit Tweedie Models

Source: https://github.com/strudeldoodles/superglm/blob/master/README.md

Shows how to fit a model with a fixed Tweedie power or estimate the power parameter using profile likelihood.

```python
from superglm import Tweedie
model = SuperGLM(family=Tweedie(p=1.5), penalty=GroupLasso())
model = SuperGLM(family=Tweedie(p=1.5), penalty=GroupLasso(lambda1=0.01))
result = model.estimate_p(df, y, sample_weight=exposure, p_range=(1.1, 1.9))
print(result.p_hat)
result.profile_plot()
```

--------------------------------

### Create Polynomial Features with SuperGLM

Source: https://context7.com/strudeldoodles/superglm/llms.txt

Explains the use of `Polynomial` features in SuperGLM, which employ an orthogonal polynomial (Legendre basis). This method is noted for its stability across refits and suitability for modeling simple monotone or quadratic relationships. An example model configuration is provided.

```python
from superglm import Polynomial

model = SuperGLM(
    family="poisson",
    penalty="group_lasso",
    selection_penalty=0.01,
    features={
        "VehAge": Polynomial(degree=2),   # Quadratic
        "BonusMalus": Polynomial(degree=3),  # Cubic (default)
        "Area": Categorical(),
    },
)
model.fit(df, y, sample_weight=exposure)
```

--------------------------------

### Initialize SuperGLM with Auto-detection

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/getting-started/quickstart.md

Demonstrates how to initialize a SuperGLM model by specifying spline columns while allowing the model to auto-detect other features. It fits the model using a Poisson family and group lasso penalty.

```python
from superglm import SuperGLM

model = SuperGLM(
    family="poisson",
    penalty="group_lasso",
    selection_penalty=0.01,
    splines=["DrivAge", "VehAge", "BonusMalus"],
    n_knots=10,
)
model.fit(df, y, sample_weight=exposure)
predictions = model.predict(df)
```

--------------------------------

### Serialize and Load SuperGLM Model with Pickle

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/guide/deployment.md

Demonstrates how to fit a SuperGLM model using the native API, serialize it to disk using pickle, and reload it to perform predictions and term inference without refitting.

```python
import pickle
from pathlib import Path
import numpy as np
from superglm import Categorical, Numeric, Spline, SuperGLM

model = SuperGLM(
    family="poisson",
    selection_penalty=0.0,
    discrete=True,
    features={
        "age": Spline(kind="bs", k=12, knot_strategy="quantile_tempered", knot_alpha=0.2),
        "density": Numeric(),
        "region": Categorical(base="most_exposed"),
    },
)

model.fit_reml(train_df, claim_count, offset=np.log(exposure), max_reml_iter=20)

with Path("pricing_model.pkl").open("wb") as f:
    pickle.dump(model, f)

with Path("pricing_model.pkl").open("rb") as f:
    loaded = pickle.load(f)

pred = loaded.predict(score_df, offset=np.log(score_exposure))
age_term = loaded.term_inference("age", with_se=False)
print(age_term.spline.interior_knots)
```

--------------------------------

### SuperGLM fit_path(): Regularization Path Computation in Python

Source: https://context7.com/strudeldoodles/superglm/llms.txt

Computes a sequence of models across a range of regularization strengths, utilizing warm starts for efficiency. This method is valuable for model selection and understanding how regularization impacts model complexity. It requires the SuperGLM library and allows specification of the number of lambda values and the lambda ratio.

```Python
from superglm import SuperGLM, PathResult

model = SuperGLM(
    family="poisson",
    penalty="group_lasso",
    splines=["DrivAge", "VehAge"],
)
result = model.fit_path(
    df, y,
    sample_weight=exposure,
    n_lambda=50,
    lambda_ratio=1e-3,
)

```

--------------------------------

### Load and Prepare French MTPL2 Frequency Dataset

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/notebooks/plotting_diagnostics_demo.ipynb

Loads the French MTPL2 frequency dataset from a Parquet file or fetches it from OpenML if the file is not found. This step prepares the data for subsequent SuperGLM model fitting and analysis.

```python
t0 = time.perf_counter()

DATA_PATH = Path("../../data/freMTPL2freq.parquet")

if DATA_PATH.exists():
    df = pd.read_parquet(DATA_PATH)
else:
    from sklearn.datasets import fetch_openml

    df = fetch_openml(data_id=41214, as_frame=True, parser="auto").frame
```

--------------------------------

### SuperGLM Regularization Path Fitting

Source: https://github.com/strudeldoodles/superglm/blob/master/README.md

Shows how to fit a sequence of SuperGLM models across a range of regularization strengths using `fit_path`. This method allows for warm starts and efficient computation of the regularization path. It accepts a DataFrame, target variable, sample weights, and either the number of lambda values or a custom lambda sequence. The result object contains coefficients, deviance, and iteration counts for each lambda.

```python
from superglm import PathResult, SuperGLM, Poisson, Spline, Categorical

model = SuperGLM(
    family=Poisson(),
    penalty=GroupLasso(),
    features={
        "DrivAge": Spline(k=14),
        "Area": Categorical(base="most_exposed"),
    },
)
result = model.fit_path(df, y, sample_weight=exposure, n_lambda=50, lambda_ratio=1e-3)

result.lambda_seq       # (50,) decreasing lambda values
result.coef_path        # (50, p) coefficients at each lambda
result.deviance_path    # (50,) deviance at each lambda
result.n_iter_path      # (50,) PIRLS iterations per lambda

# Or pass a custom lambda sequence:
result = model.fit_path(df, y, sample_weight=exposure, lambda_seq=[1.0, 0.1, 0.01])
```

--------------------------------

### Estimate Tweedie Power Parameter

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/notebooks/tweedie_profile_estimation.ipynb

Uses the SuperGLM estimate_p method to find the optimal power parameter via profile likelihood. It requires an initial guess and uses Brent's method to minimize the negative log-likelihood.

```python
model = SuperGLM(
    family=Tweedie(p=1.2),
    features={"x": Spline(n_knots=10)},
)

result = model.estimate_p(X, y, sample_weight=exposure, offset=offset)

print(f"Estimated p:   {result.p_hat:.4f}  (true: {TRUE_P})")
print(f"Estimated phi: {result.phi_hat:.4f}  (true: {TRUE_PHI})")
```

--------------------------------

### Estimate Tweedie Power Parameter

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/guide/families.md

Demonstrates fitting a Tweedie model and estimating the power parameter p via profile likelihood. Provides functionality for confidence intervals and deviance visualization.

```python
from superglm import SuperGLM, Tweedie

model = SuperGLM(family=Tweedie(p=1.5), selection_penalty=0.01)
result = model.estimate_p(df, y, sample_weight=exposure, p_range=(1.1, 1.9))
print(result.p_hat)

# Confidence interval and plotting
ci = result.ci(alpha=0.05)
result.profile_plot()
```

--------------------------------

### Apply Monotone Constraints

Source: https://github.com/strudeldoodles/superglm/blob/master/README.md

Demonstrates how to declare monotone constraints at the feature level using splines and apply post-fit repairs to ensure the model adheres to these constraints.

```python
features = {
    "BonusMalus": Spline(kind="bs", k=14, monotone="increasing"),
}
model.fit_reml(df, y, sample_weight=exposure)
model.apply_monotone_postfit(df, sample_weight=exposure)
```

--------------------------------

### Simulate and Fit SuperGLM with Varying Sample Sizes (Python)

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/notebooks/tweedie_profile_estimation.ipynb

This Python script simulates data using a Tweedie distribution with different sample sizes. It then fits a SuperGLM model to each simulated dataset, estimating the 'p' parameter and its confidence interval. The results, including parameter estimates, confidence interval width, and evaluation counts, are collected and presented in a DataFrame.

```python
sample_sizes = [5_000, 10_000, 20_000, 50_000]
results_table = []

for n_sub in sample_sizes:
    rng_sub = np.random.default_rng(123)
    x_sub = rng_sub.uniform(0, 1, n_sub)
    exp_sub = rng_sub.uniform(0.5, 2.0, n_sub)
    mu_rate_sub = np.exp(np.log(5.0) + 0.5 * np.sin(2 * np.pi * x_sub))
    mu_sub = mu_rate_sub * exp_sub
    y_sub = generate_tweedie_cpg(n_sub, mu=mu_sub, phi=TRUE_PHI / exp_sub, p=TRUE_P, rng=rng_sub)
    X_sub = pd.DataFrame({"x": x_sub})

    m = SuperGLM(family=Tweedie(p=1.5), features={"x": Spline(n_knots=8)})
    r = m.estimate_p(X_sub, y_sub, sample_weight=exp_sub, offset=np.log(exp_sub))
    ci = r.ci(alpha=0.05)
    results_table.append(
        {
            "n": n_sub,
            "p_hat": r.p_hat,
            "phi_hat": r.phi_hat,
            "ci_lo": ci[0],
            "ci_hi": ci[1],
            "ci_width": ci[1] - ci[0],
            "evals": r.n_evaluations,
        }
    )

df_results = pd.DataFrame(results_table)
df_results["error"] = df_results["p_hat"] - TRUE_P
df_results[["n", "p_hat", "error", "ci_lo", "ci_hi", "ci_width", "phi_hat", "evals"]]
```

--------------------------------

### SuperGLM Class Methods

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/api/model.md

Overview of the primary methods for initializing, training, and evaluating SuperGLM models.

```APIDOC
## SuperGLM Class Methods

### Description
The SuperGLM class provides a structured interface for Generalized Linear Models. It supports various fitting techniques, path estimation, and statistical inference.

### Methods
- **__init__**: Initializes the SuperGLM model.
- **fit**: Fits the model to the provided data.
- **fit_reml**: Fits the model using Restricted Maximum Likelihood.
- **fit_path**: Fits the model over a regularization path.
- **predict**: Generates predictions for new data.
- **relativities**: Calculates model relativities.
- **term_inference**: Performs statistical inference on model terms.
- **plot**: Generates diagnostic plots for the model.
- **metrics**: Returns performance metrics.
- **refit_unpenalised**: Refits the model without penalization.
- **drop1**: Performs drop-one-term analysis.
- **estimate_theta**: Estimates the dispersion parameter theta.
- **estimate_p**: Estimates the power parameter p.
```

--------------------------------

### Generate Lift Chart with SuperGLM

Source: https://context7.com/strudeldoodles/superglm/llms.txt

Creates a lift chart to compare observed versus predicted values across quantile bins. It requires observed values, predicted values, and exposure data. The output includes bin information and a figure that can be saved.

```python
from superglm import lift_chart

result = lift_chart(
    y_obs=y_holdout,
    y_pred=model.predict(df_holdout),
    exposure=exposure_holdout,
    n_bins=10,
)

print(result.bins[["bin", "observed", "predicted", "obs_pred_ratio"]])
result.figure.savefig("lift_chart.png")
```

--------------------------------

### Analyze Regularization Path

Source: https://context7.com/strudeldoodles/superglm/llms.txt

Prints key information about the regularization path of a fitted SuperGLM model, including the lambda sequence, coefficient path shape, deviance, and effective degrees of freedom. It then demonstrates how to make predictions using the model fitted at the last regularization point.

```python
print(f"Lambda sequence: {result.lambda_seq[:5]}...")
print(f"Coefficient path shape: {result.coef_path.shape}")
print(f"Deviance at each lambda: {result.deviance_path[:5]}...")
print(f"Effective DF path: {result.edf_path[:5]}...")

# Model is fitted at the last (least-regularized) point
predictions = model.predict(df)
```

--------------------------------

### Create Double Lift Chart

Source: https://context7.com/strudeldoodles/superglm/llms.txt

Compares model predictions against baseline or current rates on holdout data to assess performance.

```python
from superglm import double_lift_chart
result = double_lift_chart(y_obs=y_holdout, y_pred_model=model.predict(df_holdout), y_pred_current=current_predictions, exposure=exposure_holdout, n_bins=20, labels=("Actual", "New Model", "Current Rates"))
print(result.bins[["bin", "actual_index", "model_index", "current_index"]])
```

--------------------------------

### Apply Weights and Offsets for Count Models

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/getting-started/quickstart.md

Illustrates the two standard patterns for insurance modeling: using an offset to model claim rates from raw counts, or using sample weights to account for heteroscedasticity in rate-based targets.

```python
# Raw count target: offset absorbs exposure, model estimates a rate
model.fit(df, claim_counts, offset=np.log(exposure))

# Rate target (count / exposure): weight by exposure for heteroscedasticity
model.fit(df, claim_rate, sample_weight=exposure)
```

--------------------------------

### Direct IRLS Solver Implementation

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/guide/optimization.md

Illustrates the iterative process for solving the smooth penalized subproblem using block-diagonal summaries and linear system solvers.

```python
# ── Direct IRLS: smooth penalties only ─────────────────────────
η = α + Xβ
μ = g⁻¹(η)

for t in 1, 2, ...:
    W, z = working_quantities(η, μ, y)

    # Form augmented (p+1)×(p+1) system from cached summaries
    XᵀWX, XᵀW1, XᵀWz = block_xtwx_rhs(groups, W, Wz)

    M = [[ΣWᵢ,   XᵀW1  ],           # intercept row
         [XᵀW1,  XᵀWX + S]]          # coefficient rows
    rhs = [ΣWᵢzᵢ, XᵀWz]

    α, β = solve(M, rhs)              # one eigh or Cholesky

    η = α + Xβ
    μ = g⁻¹(η)
    if converged(deviance): break
```

--------------------------------

### Generate Lift Chart

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/notebooks/plotting_diagnostics_demo.ipynb

Creates a lift chart to compare observed versus predicted values across equal-exposure quantile bins. It utilizes the lift_chart function and displays the results along with a performance plot.

```python
t0 = time.perf_counter()
result = lift_chart(
    y_obs=y,
    y_pred=mu,
    sample_weight=np.ones_like(y),
    exposure=exposure,
    n_bins=20,
)
plt.gcf().set_size_inches(10, 5)
plt.show()
print(result.bins.to_string(index=False))
print(f"[{time.perf_counter() - t0:.2f}s]")
```

--------------------------------

### Generate Model Summary in Python

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/notebooks/mtpl2_frequency_walkthrough.ipynb

This snippet demonstrates how to invoke the summary method on a fitted SuperGLM model. It outputs a formatted table containing model diagnostics, parametric coefficients, and smooth term statistics.

```python
model.summary()
```

--------------------------------

### Extract Standardized and Leverage Metrics

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/notebooks/plotting_diagnostics_demo.ipynb

Prints key statistical metrics for a model, including standardized deviance and Pearson residuals, leverage, Cook's distance, effective degrees of freedom, and dispersion (phi).

```python
print(
    f"Std deviance residuals — mean: {m.std_deviance_residuals.mean():.4f}, "
    f"std: {m.std_deviance_residuals.std():.4f}"
)
print(
    f"Std Pearson residuals  — mean: {m.std_pearson_residuals.mean():.4f}, "
    f"std: {m.std_pearson_residuals.std():.4f}"
)
print(f"Leverage — mean: {m.leverage.mean():.6f}, max: {m.leverage.max():.6f}")
print(
    f"Cook's D — mean: {m.cooks_distance.mean():.6f}, "
    f"max: {m.cooks_distance.max():.6f}, "
    f">0.5: {(m.cooks_distance > 0.5).sum()}, "
    f">1.0: {(m.cooks_distance > 1.0).sum()}"
)
print(f"Effective df: {m.effective_df:.1f}")
print(f"Phi (dispersion): {m.phi:.4f}")
```

--------------------------------

### Configure Categorical Features in SuperGLM

Source: https://context7.com/strudeldoodles/superglm/llms.txt

Illustrates how to define and configure categorical features for SuperGLM models. It covers different reference level strategies ('most_exposed', 'first', explicit) and demonstrates how categorical features are handled under group lasso, including accessing reconstructed relativities.

```python
from superglm import Categorical

# Reference level strategies
cat_exposure = Categorical(base="most_exposed")  # Highest-exposure level (default)
cat_first = Categorical(base="first")            # Alphabetically first
cat_explicit = Categorical(base="B")             # Explicit level name

model = SuperGLM(
    family="poisson",
    penalty="group_lasso",
    selection_penalty=0.01,
    features={
        "Area": Categorical(base="most_exposed"),
        "VehicleType": Categorical(base="first"),
    },
)
model.fit(df_with_cats, y, sample_weight=exposure)

# Access reconstructed relativities
area_info = model.reconstruct_feature("Area")
print(f"Base level: {area_info['base_level']}")
print(f"Relativities: {area_info['relativities']}")
```

--------------------------------

### Ordinary Least Squares (OLS) Implementation

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/guide/optimization.md

This snippet demonstrates the direct solution for Ordinary Least Squares (OLS) using the normal equations. It involves a single matrix solve and is suitable for simple quadratic objectives without iteration.

```python
# ── OLS: one solve, no iteration ──────────────────────────────
β = solve(XᵀX, Xᵀy)            # normal equations: (XᵀX)β = Xᵀy
```

--------------------------------

### Define and Fit SuperGLM Model with Interactions (Python)

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/guide/interactions.md

This snippet demonstrates how to initialize a SuperGLM model with specified features and interactions, and then fit the model to the data. It highlights the use of the 'interactions' parameter to define relationships between features like 'age' and 'region'.

```python
model = SuperGLM(
    features={"age": Spline(k=14), "region": Categorical()},
    interactions=[("age", "region")],
    selection_penalty=0.01,
)
model.fit(df, y, sample_weight=exposure)
```

--------------------------------

### Load and Prepare MTPL2 Dataset

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/notebooks/mtpl2_frequency_walkthrough.ipynb

Loads the French MTPL2 dataset from a local parquet file or OpenML, performs standard actuarial cleaning, and prepares the feature matrix and target variables for modeling.

```python
from sklearn.datasets import fetch_openml

DATA_PATH = Path("../../data/freMTPL2freq.parquet")

if DATA_PATH.exists():
    df = pd.read_parquet(DATA_PATH)
else:
    df = fetch_openml(data_id=41214, as_frame=True, parser="auto").frame

# Standard cleaning
claim_count = df["ClaimNb"].astype(float).clip(upper=4)
exposure_series = df["Exposure"].astype(float).clip(lower=0.01)
df["ClaimNb"] = claim_count
df["Exposure"] = exposure_series
df["DrivAge"] = df["DrivAge"].astype(float).clip(18, 90)
df["VehAge"] = df["VehAge"].astype(float).clip(0, 20)
df["BonusMalus"] = df["BonusMalus"].astype(float).clip(50, 150)
df["Density"] = df["Density"].astype(float)
df["VehPower"] = df["VehPower"].astype(float)
df["LogDensity"] = np.log1p(df["Density"])

y = (df["ClaimNb"] / df["Exposure"]).to_numpy(dtype=np.float64)
exposure = df["Exposure"].to_numpy(dtype=np.float64)

features_to_use = ["DrivAge", "VehAge", "BonusMalus", "LogDensity", "Area", "VehPower"]
X = df[features_to_use].copy()
```

--------------------------------

### Override Link Functions

Source: https://github.com/strudeldoodles/superglm/blob/master/README.md

Demonstrates how to override the default link function for a given family, such as using an InverseLink for a Gamma family.

```python
from superglm import SuperGLM, InverseLink
model = SuperGLM(family="gamma", link=InverseLink())
```

--------------------------------

### Perform Train/Holdout Split (Python)

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/notebooks/plotting_diagnostics_demo.ipynb

Splits the dataset into training and holdout sets with a specified ratio (80% train, 20% holdout). This is a crucial step for model validation and evaluation. It uses numpy for random number generation and shuffling.

```python
from sklearn.model_selection import KFold

from superglm.model_selection import cross_validate
from superglm.validation import lorenz_curve as _lorenz

t0 = time.perf_counter()

# ── Train/holdout split (80/20, shuffled, fixed seed) ─────────
rng_split = np.random.default_rng(42)
n_total = len(y)
idx_all = np.arange(n_total)
rng_split.shuffle(idx_all)
n_train = int(0.8 * n_total)
train_idx, holdout_idx = idx_all[:n_train], idx_all[n_train:]

X_train, X_holdout = (
    X.iloc[train_idx].reset_index(drop=True),
    X.iloc[holdout_idx].reset_index(drop=True),
)
y_train, y_holdout = y[train_idx], y[holdout_idx]
exp_train, exp_holdout = exposure[train_idx], exposure[holdout_idx]

print(f"Training: {len(y_train):,} rows, Holdout: {len(y_holdout):,} rows")
print(f"[{time.perf_counter() - t0:.2f}s]")
```

--------------------------------

### Binary Classification with SuperGLM

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/guide/families.md

Demonstrates how to use the Binomial family for binary classification, including custom link functions and the sklearn-compatible SuperGLMClassifier.

```APIDOC
## Binomial Classification

### Description
Configures the model for binary outcomes (y in {0, 1}) using the Binomial family. Supports custom link functions like Probit or Cloglog, and provides a classifier interface for sklearn compatibility.

### Method
POST /fit

### Parameters
#### Request Body
- **family** (string) - Required - Set to "binomial"
- **link** (object) - Optional - Link function (e.g., ProbitLink(), CloglogLink())
- **selection_penalty** (float) - Optional - Regularization strength

### Request Example
{
  "family": "binomial",
  "link": "logit",
  "selection_penalty": 0
}

### Response
#### Success Response (200)
- **probabilities** (array) - Predicted probabilities for class 1
```

--------------------------------

### Create Spline Features with SuperGLM

Source: https://context7.com/strudeldoodles/superglm/llms.txt

Demonstrates the creation of various spline-based features using the SuperGLM library. It showcases the `Spline` factory function with different basis types (B-spline, natural spline, cubic regression spline), selection penalties, monotone constraints, knot placement strategies, and fixed knots.

```python
from superglm import Spline, BasisSpline, NaturalSpline, CubicRegressionSpline

# Recommended API - use Spline factory
spline_bs = Spline(kind="bs", k=14)              # B-spline (P-spline), 13 columns after identifiability
spline_ns = Spline(kind="ns", k=10)              # Natural spline, linear tails
spline_cr = Spline(kind="cr", k=10)              # Cubic regression spline

# Double-penalty selection (mgcv-style)
spline_select = Spline(kind="cr", k=12, select=True)  # Decomposes into linear + wiggly subgroups

# Monotone constraints (post-fit isotonic repair)
spline_mono = Spline(kind="bs", k=14, monotone="increasing")

# Knot placement strategies
spline_quantile = Spline(
    kind="bs", k=14,
    knot_strategy="quantile_tempered",  # Density-weighted knots
    knot_alpha=0.2,                      # Tempering parameter
)

# Fixed knots for reproducibility across refits
spline_fixed = Spline(
    kind="bs", k=14,
    knots=np.linspace(20, 70, 12),      # Explicit interior knots
    boundary=(18, 80),                   # Fixed boundary
)

# Example model with various spline types
model = SuperGLM(
    family="poisson",
    selection_penalty=0,
    features={
        "DrivAge": Spline(kind="cr", k=14, select=True),
        "VehAge": Spline(kind="bs", k=10),
        "BonusMalus": Spline(kind="ns", k=8),
    },
)
model.fit_reml(df, y, sample_weight=exposure)
```

--------------------------------

### SuperGLM Main Model Class: Building and Fitting GLMs in Python

Source: https://context7.com/strudeldoodles/superglm/llms.txt

Demonstrates the primary SuperGLM class for creating and fitting penalized GLMs with splines and group penalties. It shows both auto-detect mode for rapid prototyping and explicit mode for detailed feature control, supporting various distribution families and link functions. Requires numpy and pandas for data manipulation.

```Python
import numpy as np
import pandas as pd
from superglm import SuperGLM, Spline, Categorical, Numeric

# Create sample insurance data
np.random.seed(42)
n = 5000
df = pd.DataFrame({
    "DrivAge": np.random.uniform(18, 80, n),
    "VehAge": np.random.uniform(0, 20, n),
    "BonusMalus": np.random.uniform(50, 150, n),
    "Area": np.random.choice(["A", "B", "C", "D"], n),
    "LogDensity": np.random.uniform(0, 10, n),
})
exposure = np.random.uniform(0.5, 1.0, n)
y = np.random.poisson(0.1 * exposure)

# Auto-detect mode - quick prototyping
model = SuperGLM(
    family="poisson",
    penalty="group_lasso",
    selection_penalty=0.01,
    splines=["DrivAge", "VehAge", "BonusMalus"],  # These become splines
    n_knots=10,
)
model.fit(df, y, sample_weight=exposure)
predictions = model.predict(df)

# Explicit mode - full control over each feature
model_explicit = SuperGLM(
    family="poisson",
    penalty="group_lasso",
    selection_penalty=0.01,
    features={
        "DrivAge": Spline(kind="bs", k=14),       # B-spline with 14 basis functions
        "VehAge": Spline(kind="cr", k=10),        # Cubic regression spline
        "BonusMalus": Spline(kind="ns", k=10),    # Natural spline
        "Area": Categorical(base="most_exposed"), # Categorical with exposure-based reference
        "LogDensity": Numeric(),                  # Pass-through numeric
    },
)
model_explicit.fit(df, y, sample_weight=exposure)

print(f"Deviance: {model_explicit.result.deviance:.2f}")
print(f"Effective DF: {model_explicit.result.effective_df:.2f}")
```

--------------------------------

### Generate Model Summary Table

Source: https://context7.com/strudeldoodles/superglm/llms.txt

Displays coefficient estimates, standard errors, p-values, and smooth term tests in a formatted table similar to statsmodels.

```python
from superglm import SuperGLM, Spline, Categorical

model = SuperGLM(
    family="poisson",
    selection_penalty=0,
    features={
        "DrivAge": Spline(kind="cr", k=14),
        "VehAge": Spline(kind="cr", k=10),
        "Area": Categorical(),
    },
)
model.fit_reml(df, y, sample_weight=exposure)

print(model.summary())
```

--------------------------------

### Generate Holdout Double Lift Chart

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/notebooks/plotting_diagnostics_demo.ipynb

Calculates and visualizes a double-lift chart to compare two models against observed data. It bins the data, computes performance indices, and outputs a summary table with execution time.

```python
result_holdout = double_lift_chart(
    y_obs=y_holdout,
    y_pred_model=mu_holdout_full,
    y_pred_current=mu_holdout_simple,
    exposure=exp_holdout,
    n_bins=20,
    labels=("Actual", "Full (interaction)", "Main effects only"),
)
plt.gcf().set_size_inches(10, 6)
plt.suptitle("Holdout Double Lift Chart (untouched 20% sample)", fontsize=12)
plt.show()

print(
    result_holdout.bins[
        [
            "bin",
            "n_rows",
            "exposure_share",
            "actual_index",
            "model_index",
            "current_index",
            "sort_score_median",
        ]
    ].to_string(index=False)
)
print(f"[{time.perf_counter() - t0:.2f}s]")
```

--------------------------------

### Generate Double Lift Chart

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/guide/validation.md

Compares a new model against a baseline model by sorting observations by prediction ratios and binning by exposure to visualize incremental value.

```python
from superglm.validation import double_lift_chart

result = double_lift_chart(
    y_obs=y_holdout,
    y_pred_model=mu_new,
    y_pred_current=mu_baseline,
    exposure=exposure_holdout,
    n_bins=20,
)
```

--------------------------------

### PIRLS Solver Dispatch Logic

Source: https://github.com/strudeldoodles/superglm/blob/master/docs/guide/optimization.md

Demonstrates the conditional logic used to select between a direct solver for smooth penalties and a proximal solver for nonsmooth group lasso penalties.

```python
# ── PIRLS dispatch ─────────────────────────────────────────────
# At fit time, the solver is chosen automatically:

if selection_penalty == 0:
    # All penalties are quadratic (smoothing only)
    # → one dense (p+1)×(p+1) solve per IRLS iteration
    result, H⁻¹ = fit_irls_direct(X, y, W, S)

else:
    # Group lasso / sparse group lasso penalty is nonsmooth
    # → proximal Newton BCD inner loop per IRLS iteration
    result = fit_pirls(X, y, W, S, penalty)
```