### Install WarpGBM on Windows

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Install WarpGBM on Windows by cloning the repository, building a wheel, and then installing it via pip.

```bash
git clone https://github.com/jefferythewind/warpgbm.git
cd warpgbm
python setup.py bdist_wheel
pip install dist/warpgbm-*.whl
```

--------------------------------

### WarpGBM Development Environment Setup

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Provides commands for setting up a development environment for WarpGBM using Conda. It includes creating an environment, installing PyTorch with CUDA support, and installing WarpGBM in development mode.

```bash
# Create conda environment
conda create -n warpgbm_dev python=3.11
conda activate warpgbm_dev

# Install PyTorch with CUDA
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124

# Install WarpGBM in dev mode
cd /path/to/warpgbm
pip install -e .

# Install dev dependencies
pip install pytest scikit-learn pandas numerapi
```

--------------------------------

### Install WarpGBM

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Install the latest version from GitHub or the stable version from PyPI. Ensure PyTorch with CUDA support is installed.

```bash
pip install git+https://github.com/jefferythewind/warpgbm.git
```

```bash
pip install warpgbm
```

--------------------------------

### Quick Start: Regression

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Train a regression model in 5 lines of Python code. Requires X_train and y_train data.

```python
from warpgbm import WarpGBM
import numpy as np

model = WarpGBM(objective='regression', max_depth=5, n_estimators=100)
model.fit(X_train, y_train)
predictions = model.predict(X_test)
```

--------------------------------

### Example: Regression with Custom Parameters

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Demonstrates training a regression model with specific parameters like num_bins and learning rate. Uses scikit-learn for data generation.

```python
import numpy as np
from sklearn.datasets import make_regression
from warpgbm import WarpGBM

# Generate data
X, y = make_regression(n_samples=100_000, n_features=500, random_state=42)
X, y = X.astype(np.float32), y.astype(np.float32)

# Train
model = WarpGBM(
    objective='regression',
    max_depth=5, 
    n_estimators=100, 
    learning_rate=0.01,
    num_bins=32
)
model.fit(X, y)

# Predict
preds = model.predict(X)
print(f"Correlation: {np.corrcoef(preds, y)[0,1]:.4f}")
```

--------------------------------

### WarpGBM Testing Workflow Examples

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Demonstrates how to run tests for WarpGBM using pytest. Examples include running a specific test file, running multiple specific test files, and running all tests except slow ones.

```bash
# Run specific test
pytest tests/test_multiclass.py -v
pytest tests/test_feature_importance.py -v

# Run all tests except slow ones
pytest tests/test_fit_predict_corr.py tests/test_invariant.py tests/test_multiclass.py -v
```

--------------------------------

### WarpGBM Warm Start vs. Default Training

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Compares training behavior with and without the `warm_start` parameter. Without warm start, `fit` retrains from scratch. With warm start, `fit` adds trees to the existing forest.

```python
model = WarpGBM(n_estimators=100)
model.fit(X, y)  # Trains 100 trees
model.fit(X, y)  # Retrains from scratch, 100 new trees
```

```python
model = WarpGBM(n_estimators=50, warm_start=True)
model.fit(X, y)  # Trains 50 trees
model.n_estimators = 100
model.fit(X, y)  # Adds 50 more trees (total: 100)
```

--------------------------------

### Test Structure: Basic Unit Test

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

A standard structure for writing unit tests, including setup, execution, and assertion phases. This example demonstrates a basic test for feature name.

```python
def test_feature_name():
    # 1. Setup
    X, y = generate_data()
    
    # 2. Execute
    model = WarpGBM(objective='...', ...)
    model.fit(X, y)
    preds = model.predict(X)
    
    # 3. Assert
    assert accuracy > threshold
    assert preds.shape == expected_shape
    assert model.num_classes == expected_classes
```

--------------------------------

### Install WarpGBM on Linux/macOS

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Install WarpGBM using pip, which will compile CUDA extensions locally. Ensure PyTorch and CUDA are set up.

```bash
pip install git+https://github.com/jefferythewind/warpgbm.git
```

--------------------------------

### Incremental Training Example for WarpGBM

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Demonstrates incremental training using WarpGBM with `warm_start=True`. The model is initially trained with 50 trees, and then further trained to reach 100 trees.

```python
model = WarpGBM(n_estimators=50, warm_start=True)
model.fit(X, y)
# Evaluate, decide if more trees needed
model.n_estimators = 100
model.fit(X, y)  # Trains 50 more
```

--------------------------------

### Quick Start: Classification

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Train a multiclass classification model in 5 lines of Python code. Handles various label types and provides probabilities.

```python
from warpgbm import WarpGBM

model = WarpGBM(objective='multiclass', max_depth=5, n_estimators=50)
model.fit(X_train, y_train)  # y can be integers, strings, whatever
probabilities = model.predict_proba(X_test)
labels = model.predict(X_test)
```

--------------------------------

### Install WarpGBM in Colab/Mismatched CUDA

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Install WarpGBM without building extensions locally, suitable for environments like Google Colab or when CUDA versions might not match.

```bash
pip install warpgbm --no-build-isolation
```

--------------------------------

### Warm Start Training and Checkpointing

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Enable incremental training by setting `warm_start=True`. This allows you to train a model in stages, save checkpoints using `save_model`, and resume training later by loading the model and continuing the fit process. Useful for hyperparameter tuning and iterative development.

```python
from warpgbm import WarpGBM
import numpy as np

# Train 50 trees
model = WarpGBM(
    objective='regression',
    n_estimators=50,
    max_depth=5,
    learning_rate=0.1,
    warm_start=True  # Enable incremental training
)
model.fit(X, y)
predictions_50 = model.predict(X_test)

# Save checkpoint
model.save_model('checkpoint_50.pkl')

# Continue training for 50 more trees (total: 100)
model.n_estimators = 100
model.fit(X, y)  # Adds 50 trees on top of existing 50
predictions_100 = model.predict(X_test)

# Or load and continue training later
model_loaded = WarpGBM()
model_loaded.load_model('checkpoint_50.pkl')
model_loaded.warm_start = True
model_loaded.n_estimators = 100
model_loaded.fit(X, y)  # Resumes from 50 → 100 trees
```

--------------------------------

### Example: Multiclass Classification with Early Stopping

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Trains a multiclass classification model using early stopping with a validation set. Configures evaluation metrics and stopping criteria.

```python
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from warpgbm import WarpGBM

# 5-class problem
X, y = make_classification(
    n_samples=10_000, 
    n_features=50,
    n_classes=5, 
    n_informative=30
)

X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2)

model = WarpGBM(
    objective='multiclass',
    max_depth=6,
    n_estimators=200,
    learning_rate=0.1,
    num_bins=32
)

model.fit(
    X_train, y_train,
    X_eval=X_val, y_eval=y_val,
    eval_every_n_trees=10,
    early_stopping_rounds=5,
    eval_metric='logloss'
)
```

--------------------------------

### Gradient Restoration for Warm Start in WarpGBM

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Illustrates the gradient restoration logic within WarpGBM's `_fit_regression` and `_fit_classification` methods when `warm_start` is enabled. This ensures new trees are trained based on accumulated predictions from existing trees.

```python
if self.warm_start and self._is_fitted and self._trees_trained > 0:
    # Restore predictions from existing forest
    self.gradients = torch.zeros_like(self.Y_gpu) + self.base_prediction
    
    # For regression: add predictions from each existing tree
    for tree in self.forest[:self._trees_trained]:
        if tree:
            leaf_updates = self._compute_tree_predictions(tree, self.bin_indices)
            self.gradients += leaf_updates
    
    # For classification: same but per-class
    for round_trees in self.forest[:self._trees_trained]:
        for class_k, tree in enumerate(round_trees):
            if tree:
                leaf_updates = self._compute_tree_predictions(tree, self.bin_indices)
                self.gradients[:, class_k] += leaf_updates
```

--------------------------------

### Get Probabilities and Class Predictions

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Use `predict_proba` to get class probabilities and `predict` to get class labels after training a model. The output shapes are (n_samples, n_classes) for probabilities and (n_samples,) for labels.

```python
probs = model.predict_proba(X_val)  # shape: (n_samples, n_classes)
labels = model.predict(X_val)        # class labels
```

--------------------------------

### Save and Load WarpGBM Model

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Persist the trained model state to a file and load it later for inference or further training. Supports warm start functionality.

```python
# Save and load models
model.save_model('checkpoint.pkl')  # Saves all model state
model_loaded = WarpGBM()
model_loaded.load_model('checkpoint.pkl')  # Restores everything
```

--------------------------------

### Add New Objective: Metric Calculation

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Define a new metric function for the objective, for example, 'poisson_loss_torch', using PyTorch for computation.

```python
# In metrics.py
def poisson_loss_torch(y_true, y_pred):
    return torch.mean(y_pred - y_true * torch.log(y_pred + 1e-8))
```

--------------------------------

### Add New Metric: Get Eval Metric

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Modify the 'get_eval_metric()' method to handle the new metric and return its computed value.

```python
# 3. Add to get_eval_metric()
if self.eval_metric == "custom":
    return my_custom_metric(y_true, y_pred).item()
```

--------------------------------

### Get Feature Importance

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Retrieve feature importance scores. Options include normalized gain-based importance or raw gain values, and per-era importance if applicable.

```python
# Feature importance: gain-based (like LightGBM/XGBoost)
importances = model.get_feature_importance(normalize=True)  # sums to 1.0
raw_gains = model.get_feature_importance(normalize=False)   # raw gain values

# Per-era importance (when era_id was used in training)
per_era_imp = model.get_per_era_feature_importance(normalize=True)  # shape: (n_eras, n_features)
```

--------------------------------

### Get Per-Era Feature Importance

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

When training with `era_id`, you can obtain per-era feature importances to identify features that are stable across different environments. The output shape is (n_eras, n_features). A `threshold` can be used to identify invariant features.

```python
# Train with eras
model.fit(X, y, era_id=era_labels)

# Get per-era importance: shape (n_eras, n_features)
per_era_imp = model.get_per_era_feature_importance()

# Identify invariant features (high importance across ALL eras)
invariant_features = per_era_imp.min(axis=0) > threshold
```

--------------------------------

### Get Feature Importance

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Retrieve normalized feature importances after training a WarpGBM model. This helps in understanding which features contribute most to the model's predictions. Requires `sklearn.datasets` for sample data and `warpgbm` for the model.

```python
from warpgbm import WarpGBM
from sklearn.datasets import load_iris

# Train a model
iris = load_iris()
X, y = iris.data, iris.target

model = WarpGBM(objective='multiclass', max_depth=5, n_estimators=100)
model.fit(X, y)

# Get feature importance (normalized)
importances = model.get_feature_importance()
for name, imp in zip(iris.feature_names, importances):
    print(f"{name}: {imp:.4f}")
```

--------------------------------

### Initialize WarpGBM Model

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Instantiate the WarpGBM model with various hyperparameters. GPU acceleration is recommended.

```python
WarpGBM(
    objective='regression',
    num_bins=10,
    max_depth=3,
    learning_rate=0.1,
    n_estimators=100,
    min_child_weight=20,
    min_split_gain=0.0,
    L2_reg=1e-6,
    colsample_bytree=1.0,
    random_state=None,
    warm_start=False,
    threads_per_block=64,
    rows_per_thread=4,
    device='cuda'
)
```

--------------------------------

### Run All Tests

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Execute all tests in the 'tests/' directory with verbose output. This command may include downloading data, making it a slow process.

```bash
pytest tests/ -v
```

--------------------------------

### Loading and Resuming WarpGBM Training

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Illustrates how to load a previously saved WarpGBM model checkpoint and resume training. The `warm_start` parameter is set to `True`, and `n_estimators` is adjusted to train additional trees.

```python
model = WarpGBM()
model.load_model('ckpt_100.pkl')  # Has 100 trees
model.warm_start = True
model.n_estimators = 200  # Train 100 more
model.fit(X, y)
```

--------------------------------

### WarpGBM Architecture Overview

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Illustrates the high-level data flow during the WarpGBM training process, from user code to data preprocessing, forest growing, and prediction.

```text
User Code
    ↓
WarpGBM.fit()
    ↓
┌─────────────────────────────────────┐
│ 1. Data Preprocessing               │
│    - Label encoding (classification)│
│    - Binning (quantization)         │
│    - GPU transfer                   │
└─────────────────────────────────────┘
    ↓
┌─────────────────────────────────────┐
│ 2. Forest Growing Loop              │
│    For each boosting round:         │
│    ├─ Compute gradients/hessians    │
│    ├─ Build histograms (CUDA)       │
│    ├─ Find best splits (CUDA)       │
│    ├─ Grow tree recursively         │
│    └─ Update predictions            │
└─────────────────────────────────────┘
    ↓
┌─────────────────────────────────────┐
│ 3. Prediction                       │
│    - Traverse trees (CUDA kernel)   │
│    - Aggregate outputs              │
│    - Softmax (classification)       │
└─────────────────────────────────────┘
```

--------------------------------

### Add New Objective: Implement Training Logic

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Implement the specific training logic for the new objective, potentially reusing or adapting existing methods like 'grow_forest()'.

```python
def _fit_poisson(self, X, y, ...):
    # Similar to _fit_regression but with Poisson gradients
    self.bin_indices, ... = self.preprocess_gpu_data(X, y, era_id)
    # ... setup ...
    with torch.no_grad():
        self.grow_forest_poisson()  # Or reuse grow_forest()
    return self
```

--------------------------------

### Progress Logging with Emoji

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Use formatted print statements with emojis for progress logging during training. This pattern provides clear visual feedback on training rounds and loss values.

```python
print(f"🌲 Round {i+1}/{self.n_estimators} | Train Loss: {loss:.6f}")
```

--------------------------------

### Checkpointing WarpGBM Model Training

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Shows how to implement checkpointing during WarpGBM training. The model's state is saved at specified `n_estimators` milestones, allowing for training resumption and evaluation at different stages.

```python
model = WarpGBM(n_estimators=200, warm_start=True)
for checkpoint in [50, 100, 150, 200]:
    model.n_estimators = checkpoint
    model.fit(X_train, y_train)
    model.save_model(f'ckpt_{checkpoint}.pkl')
    eval_score = evaluate(model, X_val, y_val)
    print(f"Checkpoint {checkpoint}: {eval_score:.4f}")
```

--------------------------------

### Load Synthetic Dataset

Source: https://github.com/jefferythewind/warpgbm/blob/main/examples/Spiral Dataset.ipynb

Loads the downloaded synthetic dataset from local .npy files into numpy arrays for training and testing.

```python
'''
Load the real dataset from local .npy files
'''
data_dir = "synthetic_data"
X = np.load(os.path.join(data_dir, "X_train.npy"))
y = np.load(os.path.join(data_dir, "y_train.npy"))
eras = np.load(os.path.join(data_dir, "X_eras.npy"))

X_test = np.load(os.path.join(data_dir, "X_test.npy"))
y_test = np.load(os.path.join(data_dir, "y_test.npy"))
```

--------------------------------

### Profiling Training Time

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

To diagnose slow training, check data binning, `colsample_bytree` settings, and `max_depth`. Use the `time` module to profile the `model.fit()` duration for performance analysis.

```python
import time
start = time.time()
model.fit(X, y)
print(f"Training time: {time.time() - start:.2f}s")
```

--------------------------------

### Maintaining Backward Compatibility with Defaults

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Ensure backward compatibility by always maintaining default values for parameters. New features should be opt-in rather than opt-out to avoid breaking existing user code.

```python
def __init__(self, objective='regression', ...):  # Default = existing behavior
    # New features opt-in, not opt-out
```

--------------------------------

### model.fit

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Trains the WarpGBM model on the provided data.

```APIDOC
## model.fit

### Description
Trains the WarpGBM model on the provided data.

### Parameters
#### Path Parameters
- **X** (np.array) - Required - Features: shape (n_samples, n_features)
- **y** (np.array) - Required - Target: shape (n_samples,)

#### Query Parameters
- **era_id** (np.array or None) - Optional - Era labels for invariant learning
- **X_eval** (np.array or None) - Optional - Validation features
- **y_eval** (np.array or None) - Optional - Validation targets
- **eval_every_n_trees** (int or None) - Optional - Evaluation frequency (in rounds)
- **early_stopping_rounds** (int or None) - Optional - Stop if no improvement for N evaluations
- **eval_metric** (string) - Optional - Evaluation metric: 'mse', 'rmsle', 'corr', 'logloss', 'accuracy' (default: 'mse')
```

--------------------------------

### WarpGBM Constructor

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Initializes a new WarpGBM model with specified hyperparameters.

```APIDOC
## WarpGBM Constructor

### Description
Initializes a new WarpGBM model with specified hyperparameters.

### Parameters
- **objective** (string) - Optional - 'regression', 'binary', or 'multiclass'
- **num_bins** (int) - Optional - Histogram bins for feature quantization (default: 10)
- **max_depth** (int) - Optional - Maximum tree depth (default: 3)
- **learning_rate** (float) - Optional - Shrinkage rate (aka eta) (default: 0.1)
- **n_estimators** (int) - Optional - Number of boosting rounds (default: 100)
- **min_child_weight** (int) - Optional - Min sum of instance weights in child node (default: 20)
- **min_split_gain** (float) - Optional - Min loss reduction to split (default: 0.0)
- **L2_reg** (float) - Optional - L2 leaf regularization (default: 1e-6)
- **colsample_bytree** (float) - Optional - Feature subsample ratio per tree (default: 1.0)
- **random_state** (int or None) - Optional - Random seed for reproducibility (default: None)
- **warm_start** (bool) - Optional - If True, continue training from existing trees (default: False)
- **threads_per_block** (int) - Optional - CUDA block size (tune for your GPU) (default: 64)
- **rows_per_thread** (int) - Optional - Rows processed per thread (default: 4)
- **device** (string) - Optional - 'cuda' or 'cpu' (GPU strongly recommended) (default: 'cuda')
```

--------------------------------

### WarpGBM Git Branching Strategy

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Outlines the recommended Git branching strategy for WarpGBM development. It distinguishes between the stable `main` branch and feature branches, emphasizing the creation of new branches for new features.

```bash
git checkout -b feature_name
# ... make changes ...
git add relevant_files
git commit -m "Descriptive message"
git push -u origin feature_name
```

--------------------------------

### Download Synthetic Dataset Files

Source: https://github.com/jefferythewind/warpgbm/blob/main/examples/Spiral Dataset.ipynb

Downloads necessary .npy files for the synthetic dataset if they are not already present. It converts GitHub blob URLs to raw URLs for direct download.

```python
import numpy as np
from warpgbm import WarpGBM
import lightgbm as lgb
import time
import os
import requests
import matplotlib.pyplot as plt
from sklearn.metrics import mean_squared_error
```

```python
'''
Import Synthetic Memorization Dataset from Parascandolo et. al. https://arxiv.org/abs/2009.00329
'''

def download_file_if_missing(url, local_dir):
    filename = os.path.basename(url)
    local_path = os.path.join(local_dir, filename)

    if os.path.exists(local_path):
        print(f"✅ {filename} already exists, skipping download.")
        return

    # Convert GitHub blob URL to raw URL
    raw_url = url.replace("github.com", "raw.githubusercontent.com").replace("/blob/", "/")

    print(f"⬇️  Downloading {filename}...")
    response = requests.get(raw_url)
    response.raise_for_status()

    os.makedirs(local_dir, exist_ok=True)
    with open(local_path, "wb") as f:
        f.write(response.content)
    print(f"✅ Saved to {local_path}")

urls = [
    "https://github.com/jefferythewind/era-splitting-notebook-examples/blob/main/Synthetic%20Memorization%20Data%20Set/X_train.npy",
    "https://github.com/jefferythewind/era-splitting-notebook-examples/blob/main/Synthetic%20Memorization%20Data%20Set/y_train.npy",
    "https://github.com/jefferythewind/era-splitting-notebook-examples/blob/main/Synthetic%20Memorization%20Data%20Set/X_test.npy",
    "https://github.com/jefferythewind/era-splitting-notebook-examples/blob/main/Synthetic%20Memorization%20Data%20Set/y_test.npy",
    "https://github.com/jefferythewind/era-splitting-notebook-examples/blob/main/Synthetic%20Memorization%20Data%20Set/X_eras.npy",
]


local_folder = "synthetic_data"

for url in urls:
    download_file_if_missing(url, local_folder)
```

--------------------------------

### Add New Objective: Compute Gradients and Hessians

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Implement a function to compute gradients and Hessians specific to the new objective, such as Poisson loss.

```python
# In core.py or metrics.py
def _compute_poisson_gradients_hessians(self, y_true):
    """Compute gradients for Poisson loss"""
    y_pred = torch.exp(self.gradients)
    gradients = y_pred - y_true
    hessians = y_pred
    return gradients, hessians
```

--------------------------------

### Early Validation and Type Checking

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Implement early validation checks for objective consistency and raise informative `ValueError` or `TypeError` exceptions. This pattern helps catch common configuration errors before training begins.

```python
# Validate early
if self.objective == "binary" and self.num_classes != 2:
    raise ValueError(f"binary objective requires exactly 2 classes, got {self.num_classes}")

# Informative errors
if not isinstance(X, np.ndarray):
    raise TypeError(f"X must be numpy array, got {type(X)}")
```

--------------------------------

### model.load_model

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Loads a previously saved model state from a file.

```APIDOC
## model.load_model

### Description
Loads a previously saved model state from a file, restoring all model parameters and state.

### Parameters
#### Path Parameters
- **filepath** (string) - Required - The path to the model file to load (e.g., 'checkpoint.pkl').
```

--------------------------------

### Add New Metric: Implementation

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Implement the custom metric function in 'metrics.py'.

```python
# 1. Add to metrics.py
def my_custom_metric(y_true, y_pred):
    return torch.mean(...)
```

--------------------------------

### Train LightGBM Regressor

Source: https://github.com/jefferythewind/warpgbm/blob/main/examples/Spiral Dataset.ipynb

Trains a LightGBM regressor on the synthetic dataset and evaluates its performance on the test set. This snippet highlights how a naive model might fail to learn the invariant signal.

```python
'''
Naive LightGBM ( or WarpGBM) fails to learn the invariant signal, at test time, it fails

'''

print(f"X shape: {X.shape}, y shape: {y.shape}")

model = lgb.LGBMRegressor(
    max_depth=10,
    max_bin=127,
    n_estimators=50,
    learning_rate=1,
    colsample_bytree=0.9,
    min_child_weight=4
)

start_fit = time.time()
model.fit(
    X,
    y
)
fit_time = time.time() - start_fit
print(f"  Fit time:     {fit_time:.3f} seconds")

start_pred = time.time()
preds = model.predict(X_test)
pred_time = time.time() - start_pred
print(f"  Predict time: {pred_time:.3f} seconds")

corr = np.corrcoef(preds, y_test)[0, 1]
mse = mean_squared_error(preds, y_test)
print(f"  Correlation:  {corr:.4f}")
print(f"  MSE:          {mse:.4f}")

if corr < 0.95: 
    print( f"Out-of-sample correlation too low: {corr:.4f}" )
if mse > 0.02: 
    print( f"Out-of-sample MSE too high: {mse:.4f}" )
```

--------------------------------

### WarpGBM Code Organization Structure

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Provides a detailed breakdown of the WarpGBM project's directory structure, highlighting key files and their functionalities in the core, metrics, and CUDA modules.

```text
warpgbm/
├── __init__.py              # Package exports
├── core.py                  # Main WarpGBM class (1000+ lines)
│   ├── __init__()           # Parameter validation
│   ├── fit()                # Entry point for training
│   ├── _fit_regression()    # Regression-specific training
│   ├── _fit_classification()# Classification-specific training
│   ├── grow_forest()        # Regression boosting loop
│   ├── grow_forest_multiclass() # Classification boosting loop
│   ├── grow_tree()          # Recursive tree building
│   ├── predict()            # Main prediction interface
│   ├── predict_proba()      # Classification probabilities
│   └── [20+ helper methods]
│
├── metrics.py               # Loss functions and metrics
│   ├── rmsle_torch()
│   ├── softmax()
│   ├── log_loss_torch()
│   └── accuracy_torch()
│
└── cuda/
    ├── node_kernel.cpp      # Python/CUDA bridge
    ├── binner.cu            # Quantile binning
    ├── histogram_kernel.cu  # Histogram accumulation
    ├── best_split_kernel.cu # Split finding (with DES)
    └── predict.cu           # Tree traversal

tests/
├── test_fit_predict_corr.py     # Basic regression
├── test_invariant.py            # Era-splitting tests
├── test_multiclass.py           # Classification suite
├── numerai_test.py              # Real-world data
├── numerai_invariant_test.py    # Era splitting + real data
└── full_numerai_test.py         # Full dataset test

examples/
└── Spiral Dataset.ipynb     # OOD generalization demo
```

--------------------------------

### Torch Log Loss Metric

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Computes the log loss (binary cross-entropy) for binary classification tasks using PyTorch. Targets should be probabilities.

```python
def log_loss_torch(y_true, y_pred):
    return -torch.mean(y_true * torch.log(y_pred) + (1 - y_true) * torch.log(1 - y_pred))

```

--------------------------------

### Add New Objective: Unit Test

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Create a unit test for the new objective type, ensuring it fits data and produces meaningful predictions.

```python
# tests/test_poisson.py
def test_poisson_regression():
    X, y = make_poisson_data()
    model = WarpGBM(objective='poisson', ...)
    model.fit(X, y)
    assert model.predict(X).mean() > 0
```

--------------------------------

### Pre-binned Data for Maximum Speed

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

WarpGBM automatically detects and utilizes pre-binned integer data (like Numerai's quantized data) to skip the binning process, resulting in significantly faster training times. Ensure data is cast to `int8` for optimal performance.

```python
import pandas as pd
from numerapi import NumerAPI
from warpgbm import WarpGBM

# Download Numerai data (already quantized to integers)
napi = NumerAPI()
napi.download_dataset('v5.0/train.parquet', 'train.parquet')
train = pd.read_parquet('train.parquet')

features = [f for f in train.columns if 'feature' in f]
X = train[features].astype('int8').values
y = train['target'].values

# WarpGBM detects pre-binned data and skips binning
model = WarpGBM(max_depth=5, n_estimators=100, num_bins=20)
model.fit(X, y)  # Blazing fast!
```

--------------------------------

### Add New Metric: Validation Update

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Update the list of valid metrics in 'core.py' to include the name of the new custom metric.

```python
# 2. Update validation in core.py
valid_metrics = ["mse", "corr", "rmsle", "logloss", "accuracy", "custom"]
```

--------------------------------

### Test Data: Well-Separated Blobs

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Generate well-separated blob data using scikit-learn's 'make_blobs' for sanity checks in classification tests.

```python
# Well-separated (for sanity checks)
X, y = make_blobs(n_samples=1000, centers=5, random_state=42)
```

--------------------------------

### Add New Objective: Fit Path

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Modify the 'fit()' method to include a conditional branch for the new objective, directing the flow to the specific fitting logic.

```python
# In fit()
elif self.objective == "poisson":
    return self._fit_poisson(X, y, era_id, ...)
```

--------------------------------

### Train and Evaluate WarpGBM Model

Source: https://github.com/jefferythewind/warpgbm/blob/main/examples/Spiral Dataset.ipynb

Trains a WarpGBM model and evaluates its performance using fit time, predict time, correlation, and MSE. This snippet is useful for benchmarking and assessing model accuracy.

```python
'''
Naive LightGBM ( or WarpGBM) fails to learn the invariant signal, at test time, it fails

'''

print(f"X shape: {X.shape}, y shape: {y.shape}")

model = WarpGBM(
    max_depth=10,
    num_bins=127,
    n_estimators=50,
    learning_rate=1,
    threads_per_block=128,
    rows_per_thread=4,
    colsample_bytree=0.9,
    min_child_weight=4
)

start_fit = time.time()
model.fit(
    X,
    y,
    era_id=era
)
fit_time = time.time() - start_fit
print(f"  Fit time:     {fit_time:.3f} seconds")

start_pred = time.time()
preds = model.predict(X_test)
pred_time = time.time() - start_pred
print(f"  Predict time: {pred_time:.3f} seconds")

corr = np.corrcoef(preds, y_test)[0, 1]
mse = mean_squared_error(preds, y_test)
print(f"  Correlation:  {corr:.4f}")
print(f"  MSE:          {mse:.4f}")

if corr < 0.95: 
    print( f"Out-of-sample correlation too low: {corr:.4f}" )
if mse > 0.02: 
    print( f"Out-of-sample MSE too high: {mse:.4f}" )
```

--------------------------------

### Accumulating Per-Era Feature Importance

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

This code snippet shows the logic within `grow_tree()` for accumulating feature importance, considering per-era gains when a split is accepted.

```python
# When a split is accepted:
global_feature_idx = self.feat_indices_tree[local_feature].item()
per_era_gains = self.per_era_gain[:, local_feature, best_bin]  # [num_eras]

for era_idx in range(self.num_eras):
    self.per_era_feature_importance_[era_idx, global_feature_idx] += per_era_gains[era_idx].item()
```

--------------------------------

### Add New Regularization: L1 Leaf Value Computation

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Implement L1 regularization by modifying the leaf value computation in 'grow_tree()' to include soft thresholding.

```python
# In grow_tree(), update leaf value computation:
def compute_leaf_value(self, grad_sum, hess_sum):
    # Current: -G / (H + λ_L2)
    # With L1: soft thresholding
    raw_value = -grad_sum / (hess_sum + self.L2_reg)
    if self.L1_reg > 0:
        raw_value = np.sign(raw_value) * max(0, abs(raw_value) - self.L1_reg)
    return raw_value
```

--------------------------------

### Train WarpGBM Model

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Fit the model to the training data. Optional validation sets and early stopping can be provided.

```python
model.fit(
    X,
    y,
    era_id=None,
    X_eval=None,
    y_eval=None,
    eval_every_n_trees=None,
    early_stopping_rounds=None,
    eval_metric='mse'
)
```

--------------------------------

### Test Data: Regression

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Generate synthetic regression data using scikit-learn's 'make_regression' for testing purposes.

```python
# Regression
from sklearn.datasets import make_regression

X, y = make_regression(
    n_samples=5000,
    n_features=50,
    noise=0.1,
    random_state=42
)
```

--------------------------------

### Aggregating Per-Era Feature Importance

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

This code demonstrates how per-era feature importances are summed to calculate the total feature importance after training, typically in `grow_forest()` or `grow_forest_multiclass()`.

```python
# In grow_forest() and grow_forest_multiclass():
self.feature_importance_ = self.per_era_feature_importance_.sum(axis=0)
```

--------------------------------

### Make Predictions with WarpGBM

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Generate predictions on new data. Use `predict` for regression or classification labels, and `predict_proba` for classification probabilities.

```python
# Regression: returns predicted values
predictions = model.predict(X)

# Classification: returns class labels (decoded)
labels = model.predict(X)

# Classification: returns class probabilities
probabilities = model.predict_proba(X)  # shape: (n_samples, n_classes)
```

--------------------------------

### Scatter Plot of Data Dimensions

Source: https://github.com/jefferythewind/warpgbm/blob/main/examples/Spiral Dataset.ipynb

Visualizes the first two dimensions of the training data (X) colored by their target values (y). This helps in understanding the structure of the invariant signal.

```python
'''
Scatter Plot of the first two dimensions colored by target value

The invariant signal is encoded in the first two dimensions
'''

plt.scatter(X[:,0], X[:,1], c=y)
```

--------------------------------

### model.save_model

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Saves the trained model state to a file.

```APIDOC
## model.save_model

### Description
Saves the trained model state to a file.

### Parameters
#### Path Parameters
- **filepath** (string) - Required - The path to save the model file (e.g., 'checkpoint.pkl').
```

--------------------------------

### Handling CUDA Out of Memory Errors

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

When encountering `RuntimeError: CUDA out of memory`, consider reducing `num_bins`, processing data in chunks, using `torch.cuda.empty_cache()`, or processing classification classes sequentially.

```text
RuntimeError: CUDA out of memory

Solutions:
- Reduce `num_bins`
- Process data in chunks
- Use `torch.cuda.empty_cache()` between operations
- For classification: Process classes sequentially, not in parallel
```

--------------------------------

### Invariant Learning with Era IDs

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Train a WarpGBM model to learn signals that generalize across all data eras by passing an `era_id` array during fitting. This helps the model ignore spurious correlations specific to certain time periods or environments.

```python
model = WarpGBM(
    objective='regression',
    max_depth=8,
    n_estimators=100
)

model.fit(
    X, y, 
    era_id=era_labels  # Array marking which era each sample belongs to
)
```

--------------------------------

### Plot Decision Surface with Predictions

Source: https://github.com/jefferythewind/warpgbm/blob/main/examples/Spiral Dataset.ipynb

Visualizes the decision surface using scatter plot, colored by predictions. Useful for understanding model behavior on test data.

```python
'''
Plot decision surface over invariant signal
'''
plt.scatter(X_test[:,0], X_test[:,1], c=preds)
```

--------------------------------

### Running Specific Failing Pytest Tests

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

When tests fail after code changes, use `pytest` with the `-v -s` flags to run a specific failing test function and view verbose output, aiding in debugging.

```bash
pytest tests/test_name.py::test_function -v -s
```

--------------------------------

### Access Model Attributes

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Inspect various attributes of the trained model, including class labels, number of classes, tree structures, loss histories, and feature importances.

```python
model.classes_
model.num_classes
model.forest
model.training_loss
model.eval_loss
model.feature_importance_
model.per_era_feature_importance_
```

--------------------------------

### Test Data: Classification

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Generate synthetic classification data using scikit-learn's 'make_classification' for testing purposes.

```python
# Classification
from sklearn.datasets import make_classification, make_blobs

X, y = make_classification(
    n_samples=1000,
    n_features=20,
    n_classes=3,
    n_informative=15,
    random_state=42
)
```

--------------------------------

### model.predict

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Generates predictions using the trained model.

```APIDOC
## model.predict

### Description
Generates predictions using the trained model. For regression, returns predicted values. For classification, returns class labels (decoded).

### Parameters
#### Path Parameters
- **X** (np.array) - Required - Features: shape (n_samples, n_features)

### Response
- **predictions** (np.array) - Predicted values or class labels.
```

--------------------------------

### Torch Accuracy Metric

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Calculates accuracy for classification tasks using PyTorch. Assumes predictions are class labels.

```python
def accuracy_torch(y_true, y_pred):
    return torch.mean((y_pred == y_true).float())

```

--------------------------------

### Torch RMSLE Metric

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Calculates the Root Mean Squared Logarithmic Error using PyTorch. Ensure data is in the correct format for PyTorch tensors.

```python
def rmsle_torch(y_true, y_pred):
    return torch.sqrt(torch.mean(torch.pow(torch.log1p(y_pred) - torch.log1p(y_true), 2)))

```

--------------------------------

### model.get_feature_importance

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Retrieves feature importance scores.

```APIDOC
## model.get_feature_importance

### Description
Retrieves feature importance scores. Can be normalized to sum to 1.0.

### Parameters
#### Query Parameters
- **normalize** (bool) - Optional - If True, normalize scores to sum to 1.0 (default: True)

### Response
- **importances** (np.array) - Feature importance scores.
```

--------------------------------

### Classification vs. Regression Objective Branching

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Use this pattern to branch logic based on the objective function, handling classification and regression paths separately. Ensure parameters like `early_stopping_rounds` are validated before branching.

```python
def fit(self, X, y, ...):
    # Validate first
    early_stopping_rounds = self.validate_fit_params(...)
    
    # Branch by objective
    if self.objective in ["multiclass", "binary"]:
        # Classification path
        self.label_encoder = LabelEncoder()
        y_encoded = self.label_encoder.fit_transform(y)
        return self._fit_classification(X, y_encoded, ...)
    else:
        # Regression path (default)
        return self._fit_regression(X, y, ...)
```

--------------------------------

### Reusing CUDA Kernels for Multiclass Objectives

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

When implementing multiclass objectives, reuse the existing histogram kernel by iterating through each class and setting the residual gradients accordingly. This avoids modifying core CUDA kernels.

```python
# For multiclass: call histogram kernel K times
for class_k in range(self.num_classes):
    self.residual = -grads[:, class_k]  # Set per-class gradients
    hist_g, hist_h = self.compute_histograms(...)  # Reuse existing kernel
    # ... process ...
```

--------------------------------

### Tree Node Structure in WarpGBM

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Trees are represented as nested Python dictionaries. Leaf nodes contain 'leaf_value' and 'samples', while internal nodes specify 'feature', 'bin', and 'left'/'right' subtrees.

```python
{
    'feature': 5,
    'bin': 3,
    'left': {...},
    'right': {...}
}

# Leaf node:
{
    'leaf_value': 0.234,
    'samples': 150
}
```

--------------------------------

### Fit Model with Directional Era-Splitting

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Use `era_id` during model fitting to enable Directional Era-Splitting. If `era_id` is omitted, the model behaves like standard GBDT.

```python
model.fit(X, y, era_id=era_labels)  # era_labels: integer array
```

--------------------------------

### Debugging Mismatched Shapes Errors

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

For `RuntimeError: shape '[...]' is invalid for input of size [...]`, print the shapes of relevant tensors like `X`, `y`, and `gradients` to identify discrepancies. A common cause is not handling 2D gradients for multiclass objectives.

```python
print(f"X shape: {X.shape}")
print(f"y shape: {y.shape}")
print(f"gradients shape: {self.gradients.shape}")
```

--------------------------------

### model.get_per_era_feature_importance

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Retrieves feature importance scores per era.

```APIDOC
## model.get_per_era_feature_importance

### Description
Retrieves feature importance scores per era, useful when `era_id` was used during training. Scores can be normalized.

### Parameters
#### Query Parameters
- **normalize** (bool) - Optional - If True, normalize scores to sum to 1.0 (default: True)

### Response
- **per_era_imp** (np.array) - Per-era feature importance, shape (n_eras, n_features).
```

--------------------------------

### Add New Objective: Validation Update

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

When adding a new objective type, update the validation logic to include the new objective in the allowed list.

```python
# In _validate_hyperparams()
if kwargs["objective"] not in ["regression", "multiclass", "binary", "poisson"]:
    raise ValueError(...)
```

--------------------------------

### Torch Softmax Function

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

Applies the softmax function to input tensor. Useful for converting raw scores into probabilities in classification tasks.

```python
def softmax(x):
    return torch.exp(x) / torch.sum(torch.exp(x), dim=1, keepdim=True)

```

--------------------------------

### model.predict_proba

Source: https://github.com/jefferythewind/warpgbm/blob/main/README.md

Generates class probabilities for classification tasks.

```APIDOC
## model.predict_proba

### Description
Generates class probabilities for classification tasks.

### Parameters
#### Path Parameters
- **X** (np.array) - Required - Features: shape (n_samples, n_features)

### Response
- **probabilities** (np.array) - Class probabilities, shape (n_samples, n_classes).
```

--------------------------------

### Resolving Label Encoding Issues

Source: https://github.com/jefferythewind/warpgbm/blob/main/AGENT_GUIDE.md

If `ValueError: y contains previously unseen labels` occurs, ensure the `label_encoder` is fitted on the training data and consistently used for evaluation or test data.

```text
ValueError: y contains previously unseen labels

Solution: Ensure `label_encoder` is fitted on train data and used on eval/test.
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.