### Initialize Environment and Imports

Source: https://github.com/pilancilab/scnn/blob/main/notebooks/convex-vs-nonconvex.ipynb

Setup path and import necessary libraries for data generation and optimization.

```python
"""
Compare convex and non-convex optimization for a realizable classification problem.
"""

import sys
sys.path.append("..")
```

```python
import numpy as np
import matplotlib.pyplot as plt
import torch


from scnn.private.utils.data import gen_classification_data

from scnn.optimize import optimize
from scnn.regularizers import NeuronGL1
```

--------------------------------

### Install scnn from source

Source: https://github.com/pilancilab/scnn/blob/main/README.md

Use these commands to clone the repository and perform a manual installation.

```bash
git clone https://github.com/pilancilab/scnn.git
python -m pip install ./scnn
```

--------------------------------

### Install scnn from GitHub

Source: https://github.com/pilancilab/scnn/blob/main/docs/quick_start.md

Install the latest development version of scnn from its GitHub repository. This involves cloning the repository and then installing it using pip.

```bash
git clone https://github.com/pilancilab/scnn
python -m pip install .
```

--------------------------------

### Initialize Environment and Imports

Source: https://github.com/pilancilab/scnn/blob/main/notebooks/solver-demo.ipynb

Setup the Python path and import necessary modules for data generation, model definition, and optimization.

```python
"""
Train shallow neural networks on a synthetic classification dataset using convex optimization.
"""

import sys
sys.path.append("..")
```

```python
import numpy as np
import matplotlib.pyplot as plt

from scnn.private.utils.data import gen_classification_data


from scnn.models import ConvexGatedReLU, ConvexReLU
from scnn.solvers import RFISTA, AL, LeastSquaresSolver, CVXPYSolver, ApproximateConeDecomposition
from scnn.regularizers import NeuronGL1, L2, L1
from scnn.metrics import Metrics
from scnn.activations import sample_gate_vectors
from scnn.optimize import optimize_model, optimize
```

--------------------------------

### ConvexReLU Model Setup

Source: https://context7.com/pilancilab/scnn/llms.txt

Demonstrates the setup for a `ConvexReLU` model, which represents the convex reformulation of a two-layer ReLU network using gate vectors for activation patterns.

```python
import numpy as np
from scnn.models import ConvexReLU
from scnn.solvers import AL
from scnn.regularizers import NeuronGL1
from scnn.metrics import Metrics
from scnn.optimize import optimize_model
from scnn.activations import sample_gate_vectors

# Setup
np.random.seed(42)
n, d, c = 500, 20, 1  # samples, features, output dim
X_train = np.random.randn(n, d)
y_train = np.sign(np.random.randn(n))
```

--------------------------------

### Install scnn via pip

Source: https://github.com/pilancilab/scnn/blob/main/docs/index.md

Use this command to install the library in your Python environment.

```bash
pip install pyscnn
```

--------------------------------

### Configure and Train with Augmented Lagrangian Solver

Source: https://context7.com/pilancilab/scnn/llms.txt

Sets up and trains a ConvexReLU model using the Augmented Lagrangian (AL) solver. This example configures the solver with specific iteration limits, tolerances, and a penalty strength, along with metrics for constraint gaps and Lagrangian gradients.

```python
import numpy as np
from scnn.models import ConvexReLU
from scnn.solvers import AL
from scnn.regularizers import NeuronGL1
from scnn.metrics import Metrics
from scnn.optimize import optimize_model
from scnn.activations import sample_gate_vectors

np.random.seed(42)
n, d = 500, 20
X_train = np.random.randn(n, d)
y_train = np.sign(np.random.randn(n))

G = sample_gate_vectors(seed=123, d=d, n_samples=500)
model = ConvexReLU(G)

solver = AL(
    model=model,
    max_primal_iters=10000,  # Max iterations for primal problem
    max_dual_iters=100,      # Max dual parameter updates
    tol=1e-6,                # KKT tolerance
    constraint_tol=1e-6,     # Constraint violation tolerance
    delta=1000               # Initial penalty strength
)

metrics = Metrics(
    constraint_gaps=True,
    lagrangian_grad=True,
    train_accuracy=True
)

model, metrics = optimize_model(
    model, solver, metrics,
    X_train, y_train,
    regularizer=NeuronGL1(0.01),
    verbose=True
)
```

--------------------------------

### Import SCNN Utilities

Source: https://github.com/pilancilab/scnn/blob/main/notebooks/regularization-path.ipynb

Imports necessary modules from the SCNN library for optimization, data generation, models, solvers, regularizers, and metrics. Ensure the SCNN library is correctly installed and accessible.

```python
import sys
sys.path.append(".."

```

```python
import numpy as np
import matplotlib.pyplot as plt

from scnn.optimize import optimize_path
from scnn.private.utils.data import gen_classification_data
from scnn.models import ConvexGatedReLU, ConvexReLU
from scnn.solvers import RFISTA, AL
from scnn.regularizers import NeuronGL1
from scnn.metrics import Metrics
from scnn.activations import sample_gate_vectors

```

--------------------------------

### Get and Set Model Parameters

Source: https://context7.com/pilancilab/scnn/llms.txt

Retrieves the learned parameters of the model and prints their shapes. This is useful for inspecting the model's weights and biases.

```python
params = model.get_parameters()
print(f"Parameter shapes: {[p.shape for p in params]}")
```

--------------------------------

### Optimize SCNN Model with ECOS Solver

Source: https://github.com/pilancilab/scnn/blob/main/notebooks/solver-demo.ipynb

Use this snippet to optimize an SCNN model with the ECOS solver. Ensure commercial solvers like MOSEK/Gurobi are installed if needed. The `clean_sol=True` argument helps in cleaning the solver output.

```python
solver = CVXPYSolver(model, "ecos", clean_sol=True)
regularizer = NeuronGL1(0.01)
cvxpy_model, cvxpy_metrics = optimize_model(
    model,
    solver,
    metrics,
    X_train, 
    y_train, 
    X_test, 
    y_test,
    regularizer=regularizer,
    verbose=True,
)
```

--------------------------------

### Configure Optimization Parameters

Source: https://github.com/pilancilab/scnn/blob/main/notebooks/solver-demo.ipynb

Set regularization parameters, sample gate vectors, and initialize metrics for tracking training progress.

```python
lam = 0.001
max_neurons = 500
G = sample_gate_vectors(123, d, max_neurons)
metrics = Metrics(metric_freq=25, model_loss=True, train_accuracy=True, train_mse=True, test_mse=True, test_accuracy=True, neuron_sparsity=True)
```

--------------------------------

### Configure and Train with RFISTA Solver

Source: https://context7.com/pilancilab/scnn/llms.txt

Demonstrates configuring the RFISTA solver with specific parameters like max_iters and tol, and training a ConvexGatedReLU model. It also sets up metrics for objective, gradient norm, and time.

```python
import numpy as np
from scnn.models import ConvexGatedReLU
from scnn.solvers import RFISTA
from scnn.regularizers import NeuronGL1, L1
from scnn.metrics import Metrics
from scnn.optimize import optimize_model
from scnn.activations import sample_gate_vectors

np.random.seed(42)
n, d = 500, 20
X_train = np.random.randn(n, d)
y_train = np.sign(np.random.randn(n))

G = sample_gate_vectors(seed=123, d=d, n_samples=500)
model = ConvexGatedReLU(G)

solver = RFISTA(
    model=model,
    max_iters=10000,  # Maximum iterations
    tol=1e-6          # Convergence tolerance (squared gradient norm)
)

metrics = Metrics(
    metric_freq=50,
    objective=True,
    grad_norm=True,
    time=True
)

model, metrics = optimize_model(
    model, solver, metrics,
    X_train, y_train,
    regularizer=NeuronGL1(0.01),
    verbose=True
)
```

--------------------------------

### Initialize CVXPY Model

Source: https://github.com/pilancilab/scnn/blob/main/notebooks/solver-demo.ipynb

Prepare a model for high-accuracy optimization using CVXPY interior point methods.

```python
model = ConvexGatedReLU(G)
```

--------------------------------

### Monitor Training with Metrics

Source: https://context7.com/pilancilab/scnn/llms.txt

Configures the Metrics class to track various statistics during model optimization. Requires an initialized model and solver.

```python
import numpy as np
from scnn.metrics import Metrics
from scnn.models import ConvexGatedReLU
from scnn.solvers import RFISTA
from scnn.regularizers import NeuronGL1
from scnn.optimize import optimize_model
from scnn.activations import sample_gate_vectors

# Setup
np.random.seed(42)
n, d = 500, 20
X_train = np.random.randn(n, d)
y_train = np.sign(np.random.randn(n))
X_test = np.random.randn(100, d)
y_test = np.sign(np.random.randn(100))

G = sample_gate_vectors(seed=123, d=d, n_samples=500)
model = ConvexGatedReLU(G)
solver = RFISTA(model, tol=1e-6)

# Configure comprehensive metrics
metrics = Metrics(
    metric_freq=25,          # Record every 25 iterations
    objective=True,          # Optimization objective
    grad_norm=True,          # Gradient norm (convergence)
    time=True,               # Wall-clock time
    model_loss=True,         # Loss without penalties
    train_accuracy=True,     # Training accuracy
    test_accuracy=True,      # Test accuracy
    train_mse=True,          # Training MSE
    test_mse=True,           # Test MSE
    neuron_sparsity=True,    # Fraction of inactive neurons
    active_neurons=True,     # Number of active neurons
    feature_sparsity=True,   # Fraction of unused features
    weight_sparsity=True     # Fraction of zero weights
)

model, metrics = optimize_model(
    model, solver, metrics,
    X_train, y_train,
    X_test, y_test,
    regularizer=NeuronGL1(0.01),
    verbose=True
)

# Access collected metrics (numpy arrays)
print(f"Iterations recorded: {len(metrics.objective)}")
print(f"Final objective: {metrics.objective[-1]:.6f}")
print(f"Final train accuracy: {metrics.train_accuracy[-1]:.4f}")
print(f"Final test accuracy: {metrics.test_accuracy[-1]:.4f}")
print(f"Active neurons: {metrics.active_neurons[-1]}")
print(f"Total training time: {metrics.time[-1]:.2f}s")
```

--------------------------------

### Create and Train ConvexGatedReLU Model with RFISTA Solver

Source: https://context7.com/pilancilab/scnn/llms.txt

Sets up and trains a ConvexGatedReLU model using the RFISTA solver, suitable for unconstrained Gated ReLU problems. Includes data generation, model initialization, solver configuration, and training.

```python
import numpy as np
from scnn.models import ConvexGatedReLU
from scnn.solvers import RFISTA
from scnn.regularizers import NeuronGL1
from scnn.metrics import Metrics
from scnn.optimize import optimize_model
from scnn.activations import sample_gate_vectors

np.random.seed(42)
n, d = 500, 20
X_train = np.random.randn(n, d)
y_train = np.sign(np.random.randn(n))

G = sample_gate_vectors(seed=123, d=d, n_samples=500)
model = ConvexGatedReLU(G, c=1, bias=False)

solver = RFISTA(model, max_iters=10000, tol=1e-6)
regularizer = NeuronGL1(lam=0.01)
metrics = Metrics(model_loss=True, neuron_sparsity=True)

model, metrics = optimize_model(
    model, solver, metrics,
    X_train, y_train,
    regularizer=regularizer,
    verbose=True
)
```

--------------------------------

### Instantiate Model and Regularization Path

Source: https://github.com/pilancilab/scnn/blob/main/notebooks/regularization-path.ipynb

Sets up the parameters for model training, including the maximum number of neurons, the regularization strength path, gate vectors, and metrics collection. The lambda_path defines the different regularization strengths to explore.

```python
# Instantiate convex model and other options.
max_neurons = 500
lambda_path = [0.1, 0.01, 0.001, 0.0001, 0.00001]

G = sample_gate_vectors(np.random.default_rng(123), d, max_neurons)
path = [NeuronGL1(lam) for lam in lambda_path]
metrics = Metrics(metric_freq=25, model_loss=True, train_accuracy=True, train_mse=True, test_mse=True, test_accuracy=True, neuron_sparsity=True)

```

--------------------------------

### Initialize CVXPYSolver

Source: https://context7.com/pilancilab/scnn/llms.txt

Initializes the CVXPYSolver for use with CVXPY-compatible solvers like ECOS, MOSEK, or Gurobi. This solver is suitable for high-accuracy solutions to convex optimization problems.

```python
import numpy as np
from scnn.models import ConvexGatedReLU
from scnn.solvers import CVXPYSolver
from scnn.regularizers import NeuronGL1
from scnn.metrics import Metrics
from scnn.optimize import optimize_model
from scnn.activations import sample_gate_vectors
```

--------------------------------

### Optimize ConvexReLU Model

Source: https://github.com/pilancilab/scnn/blob/main/notebooks/regularization-path.ipynb

Initializes and optimizes a ConvexReLU model using the AL solver. Similar to the GatedReLU optimization, this explores the regularization path and collects metrics.

```python
model = ConvexReLU(G)
solver = AL(model)

relu_model_path, relu_metric_path = optimize_path(
    model,
    solver,
    path,
    metrics,
    X_train, 
    y_train, 
    X_test, 
    y_test,
    verbose=True,
)

```

--------------------------------

### Train with Different Regularizers

Source: https://context7.com/pilancilab/scnn/llms.txt

Demonstrates training a Gated ReLU model using various regularizers to control sparsity and weight decay. Each regularizer induces a different sparsity pattern: NeuronGL1 for neuron sparsity, FeatureGL1 for feature sparsity, L1 for element-wise weight sparsity, and L2 for weight decay.

```python
import numpy as np
from scnn.regularizers import NeuronGL1, FeatureGL1, L1, L2
from scnn.models import ConvexGatedReLU
from scnn.solvers import RFISTA
from scnn.metrics import Metrics
from scnn.optimize import optimize_model
from scnn.activations import sample_gate_vectors

# Setup
np.random.seed(42)
n, d = 500, 20
X_train = np.random.randn(n, d)
y_train = np.sign(np.random.randn(n))

G = sample_gate_vectors(seed=123, d=d, n_samples=500)

def train_with_regularizer(regularizer, name):
    model = ConvexGatedReLU(G)
    solver = RFISTA(model, tol=1e-6)
    metrics = Metrics(neuron_sparsity=True, feature_sparsity=True)
    model, metrics = optimize_model(
        model, solver, metrics,
        X_train, y_train,
        regularizer=regularizer,
        verbose=False
    )
    print(f"{name}: neuron_sparsity={metrics.neuron_sparsity[-1]:.2%}")
    return model

# NeuronGL1: Produces neuron sparsity (entire neurons become inactive)
model1 = train_with_regularizer(NeuronGL1(lam=0.01), "NeuronGL1")

# FeatureGL1: Produces feature sparsity (entire features become unused)
model2 = train_with_regularizer(FeatureGL1(lam=0.01), "FeatureGL1")

# L1: Element-wise sparsity in weights
model3 = train_with_regularizer(L1(lam=0.01), "L1")

# L2: Weight decay (ridge regularization)
model4 = train_with_regularizer(L2(lam=0.01), "L2")
```

--------------------------------

### Train ConvexReLU Model with Augmented Lagrangian Solver

Source: https://context7.com/pilancilab/scnn/llms.txt

Trains a ConvexReLU model using the Augmented Lagrangian (AL) solver for constrained optimization. Requires setting up the model, solver, metrics, and regularizer before calling optimize_model.

```python
solver = AL(model, max_primal_iters=10000, max_dual_iters=100, tol=1e-6)
regularizer = NeuronGL1(lam=0.01)
metrics = Metrics(model_loss=True, train_accuracy=True)

model, metrics = optimize_model(
    model, solver, metrics,
    X_train, y_train,
    regularizer=regularizer,
    verbose=True
)
```

--------------------------------

### Train with Object-Oriented API

Source: https://github.com/pilancilab/scnn/blob/main/notebooks/solver-demo.ipynb

Use optimize_model for greater control over model instantiation and solver selection.

```python
# Instantiate convex model and other options.
model = ConvexGatedReLU(G)
solver = RFISTA(model, tol=1e-6)
regularizer = NeuronGL1(0.01)
```

```python
grelu_model, grelu_metrics = optimize_model(
    model,
    solver,
    metrics,
    X_train, 
    y_train, 
    X_test, 
    y_test,
    regularizer=regularizer,
    verbose=True,
)
```

```python
# Training Accuracy
np.sum(np.sign(grelu_model(X_train)) == y_train) / len(y_train)
```

--------------------------------

### Generate Gate Vectors and Create ConvexReLU Model

Source: https://context7.com/pilancilab/scnn/llms.txt

Generates gate vectors and initializes a ConvexReLU model. This is typically the first step before training a constrained ReLU network.

```python
max_neurons = 500
G = sample_gate_vectors(seed=123, d=d, n_samples=max_neurons)
model = ConvexReLU(G, c=c, bias=False)
```

--------------------------------

### Train a ConvexReLU model with object-oriented interface

Source: https://github.com/pilancilab/scnn/blob/main/docs/quick_start.md

Train a neural network using the object-oriented interface for more control. This involves instantiating model, regularizer, solver, and metrics objects before calling `optimize_model`. Use this for fine-tuning parameters and custom configurations.

```python
from scnn.optimize import optimize_model, sample_gate_vectors
from scnn.models import ConvexReLU
from scnn.solvers import RFISTA
from scnn.metrics import Metrics

# create convex reformulation
max_neurons = 500
G = sample_gate_vectors(np.random.default_rng(123), d, max_neurons)
model = ConvexReLU(G)
# specify regularizer and solver
regularizer = NeuronGL1(lam=0.001)
solver = AL(model, tol=1e-6)
# choose metrics to collect during training
metrics = Metrics(model_loss=True,
                  train_accuracy=True,
                  test_accuracy=True,
                  neuron_sparsity=True)
# train model!
model, metrics = optimize_model(model,
                                solver,
                                metrics,
                                X_train,
                                y_train,
                                X_test,
                                y_test,
                                regularizer,
                                device="cpu")

# training accuracy
train_acc = np.sum(np.sign(model(X_train)) == y_train) / len(y_train)
```

--------------------------------

### Configure CVXPY Solver for Gated ReLU

Source: https://context7.com/pilancilab/scnn/llms.txt

Sets up the CVXPY solver for training Gated ReLU models. Supports various underlying solvers like 'ecos', 'scs', 'mosek', and 'gurobi'. The 'clean_sol' option post-processes the solution for sparsity.

```python
import numpy as np
from scnn.models import ConvexGatedReLU
from scnn.solvers import CVXPYSolver
from scnn.regularizers import NeuronGL1
from scnn.metrics import Metrics
from scnn.optimize import optimize_model
from scnn.activations import sample_gate_vectors

# Setup
np.random.seed(42)
n, d = 200, 10
X_train = np.random.randn(n, d)
y_train = np.sign(np.random.randn(n))

G = sample_gate_vectors(seed=123, d=d, n_samples=100)
model = ConvexGatedReLU(G)

# Configure CVXPY solver
# Available solvers: "ecos", "scs", "mosek", "gurobi" (if installed)
solver = CVXPYSolver(
    model=model,
    solver="ecos",       # Underlying solver
    solver_kwargs={},
    clean_sol=True       # Post-process for sparsity
)

metrics = Metrics(model_loss=True, neuron_sparsity=True)

model, metrics = optimize_model(
    model, solver, metrics,
    X_train, y_train,
    regularizer=NeuronGL1(0.01),
    verbose=True
)

print(f"Solution sparsity: {metrics.neuron_sparsity[-1]:.2%}")
```

--------------------------------

### Configure Non-Convex Training Parameters

Source: https://github.com/pilancilab/scnn/blob/main/notebooks/convex-vs-nonconvex.ipynb

Set hyperparameters for the non-convex optimization process.

```python
# model parameters
lam = 0.001

# optimization parameters
tol = 1e-6 
max_epochs = 1000

# try playing with the step size...

# lr = 0.01
# lr = 0.001
# lr = 0.0001
lr = 0.00001
```

--------------------------------

### Sample Dense Gates

Source: https://github.com/pilancilab/scnn/blob/main/docs/activations.md

Generate dense gate vectors by random sampling.

```APIDOC
## POST /scnn/activations/sample_dense_gates

### Description
Generate dense gate vectors by random sampling.

### Method
POST

### Endpoint
/scnn/activations/sample_dense_gates

### Parameters
#### Request Body
- **rng** (Generator) - Required - a NumPy random number generator.
- **d** (int) - Required - the dimensionality of the gate vectors.
- **n_samples** (int) - Required - the number of samples to use.

### Response
#### Success Response (200)
- **G** (ndarray) - a d x n_samples matrix of gate vectors.

#### Response Example
{
  "G": "[[0.1 0.5] [-0.2 0.8]]"
}
```

--------------------------------

### Print RFISTA Convergence Metrics

Source: https://context7.com/pilancilab/scnn/llms.txt

Prints the number of iterations taken for convergence and the final gradient norm after training with the RFISTA solver. Useful for evaluating solver performance.

```python
print(f"Converged in {len(metrics.objective)} iterations")
print(f"Final gradient norm: {metrics.grad_norm[-1]:.2e}")
```

--------------------------------

### Prepare Data for PyTorch

Source: https://github.com/pilancilab/scnn/blob/main/notebooks/convex-vs-nonconvex.ipynb

Convert numpy arrays to tensors and create a DataLoader for batch processing.

```python
# cast data and create loader
tX_train, ty_train, tX_test, ty_test = [torch.tensor(z, dtype=torch.float) for z in [X_train, y_train, X_test, y_test]]

loader = torch.utils.data.DataLoader(torch.utils.data.TensorDataset(tX_train, ty_train), batch_size=32, shuffle=True)
```

--------------------------------

### Optimize ConvexGatedReLU Model

Source: https://github.com/pilancilab/scnn/blob/main/notebooks/regularization-path.ipynb

Initializes and optimizes a ConvexGatedReLU model using the RFISTA solver. This function call explores the regularization path defined by `path` and collects metrics.

```python
model = ConvexGatedReLU(G)
solver = RFISTA(model, tol=1e-6)

gated_model_path, gated_metric_path = optimize_path(
    model,
    solver,
    path,
    metrics,
    X_train, 
    y_train, 
    X_test, 
    y_test,
    verbose=True,
)

```

--------------------------------

### Sample Gate Vectors

Source: https://github.com/pilancilab/scnn/blob/main/docs/activations.md

Generate gate vectors by random sampling with specified type and parameters.

```APIDOC
## POST /scnn/activations/sample_gate_vectors

### Description
Generate gate vectors by random sampling.

### Method
POST

### Endpoint
/scnn/activations/sample_gate_vectors

### Parameters
#### Request Body
- **seed** (int) - Required - the random seed to use when generating the gates.
- **d** (int) - Required - the dimensionality of the gate vectors.
- **n_samples** (int) - Required - the number of samples to use.
- **gate_type** (typing_extensions.Literal['dense', 'feature_sparse', 'convolutional']) - Optional - the type of gates to sample. Must be one of: 'dense', 'feature_sparse'. Defaults to 'dense'.
- **order** (int | None) - Optional - the maximum order of feature sparsity to consider. Only used for gate_type='feature_sparse'.

### Response
#### Success Response (200)
- **G** (ndarray) - a d x n_samples matrix of gate vectors.

#### Response Example
{
  "G": "[[0.1 0.5] [-0.2 0.8]]"
}
```

--------------------------------

### Execute Convex Reformulation

Source: https://github.com/pilancilab/scnn/blob/main/notebooks/convex-vs-nonconvex.ipynb

Train a gated ReLU network using convex optimization methods.

```python
# number of activation patterns to use.
max_neurons = 1000

cvx_model, metrics = optimize("gated_relu", 
                          max_neurons,
                          X_train, 
                          y_train, 
                          X_test, 
                          y_test, 
                          regularizer=NeuronGL1(lam),
                          verbose=True,  
                          device="cpu")

# Acc After Training
print("\n \n")
print("Post-Training Test Accuracy:", accuracy(cvx_model(X_test), y_test))
print(f"Hidden Layer Size: {cvx_model.parameters[0].shape[0]}")
```

--------------------------------

### Train SCNN Model with Fast Least-Squares Solver

Source: https://github.com/pilancilab/scnn/blob/main/notebooks/solver-demo.ipynb

Utilize this snippet for a super-fast least-squares solver, particularly effective for gated ReLU models by reformulating the problem as quadratic. This approach yields comparable practical performance and can be trained quickly on CPUs.

```python
# Super-fast least-squares solver.
model = ConvexGatedReLU(G)
solver = LeastSquaresSolver(model, tol=1e-8)
regularizer = L2(0.01)
lstsq_model, lstsq_metrics = optimize_model(
    model,
    solver,
    metrics,
    X_train, 
    y_train, 
    X_test, 
    y_test,
    regularizer=regularizer,
    verbose=True,
)
```

--------------------------------

### scnn.solvers.CVXPYSolver

Source: https://github.com/pilancilab/scnn/blob/main/docs/solvers.md

Interface for solving convex reformulations using CVXPY.

```APIDOC
## class scnn.solvers.CVXPYSolver

### Description
Uses CVXPY as an interface to various interior-point solvers. Note: Only supports CPU computation.

### Parameters
- **model** (Model) - Required - The model to be optimized.
- **solver** (object) - Required - The underlying solver to use.
- **solver_kwargs** (dict) - Optional - Keyword arguments for the underlying solver.
- **clean_sol** (bool) - Optional - Whether to clean the solution using a proximal-gradient step (default: False).
```

--------------------------------

### Generate Gate Vectors with sample_gate_vectors

Source: https://context7.com/pilancilab/scnn/llms.txt

Generates random gate vectors for defining activation patterns. Supports dense and feature-sparse configurations.

```python
import numpy as np
from scnn.activations import sample_gate_vectors, compute_activation_patterns

# Generate dense gate vectors (standard approach)
d = 20           # Input dimension
n_samples = 500  # Number of gate vectors
G_dense = sample_gate_vectors(
    seed=123,
    d=d,
    n_samples=n_samples,
    gate_type="dense"  # "dense", "feature_sparse", or "convolutional"
)
print(f"Dense gates shape: {G_dense.shape}")  # (d, n_samples)

# Generate feature-sparse gates (for interpretable models)
G_sparse = sample_gate_vectors(
    seed=123,
    d=d,
    n_samples=n_samples,
    gate_type="feature_sparse",
    order=2  # Maximum sparsity order (pairs of features)
)
print(f"Sparse gates shape: {G_sparse.shape}")

# Compute activation patterns from gates
X = np.random.randn(100, d)  # Sample data
D, G_filtered = compute_activation_patterns(
    X=X,
    G=G_dense,
    filter_duplicates=True,  # Remove duplicate patterns
    filter_zero=True         # Remove all-zero patterns
)
print(f"Unique activation patterns: {D.shape[1]}")
```

--------------------------------

### Configure LeastSquaresSolver for Gated ReLU

Source: https://context7.com/pilancilab/scnn/llms.txt

Utilizes the LeastSquaresSolver for fast training of Gated ReLU models with L2 regularization. It solves the equivalent ridge regression problem using either LSMR or LSQR algorithms. Requires L2 regularization.

```python
import numpy as np
from scnn.models import ConvexGatedReLU
from scnn.solvers import LeastSquaresSolver
from scnn.regularizers import L2
from scnn.metrics import Metrics
from scnn.optimize import optimize_model
from scnn.activations import sample_gate_vectors

# Setup
np.random.seed(42)
n, d = 1000, 50
X_train = np.random.randn(n, d)
y_train = np.sign(np.random.randn(n))

G = sample_gate_vectors(seed=123, d=d, n_samples=500)
model = ConvexGatedReLU(G)

# Configure least squares solver (LSMR or LSQR)
solver = LeastSquaresSolver(
    model=model,
    solver="lsmr",    # "lsmr" or "lsqr"
    max_iters=1000,
    tol=1e-8
)

# Note: LeastSquaresSolver requires L2 regularization
regularizer = L2(lam=0.01)
metrics = Metrics(model_loss=True, train_accuracy=True, time=True)

model, metrics = optimize_model(
    model, solver, metrics,
    X_train, y_train,
    regularizer=regularizer,
    verbose=True
)

print(f"Training time: {metrics.time[-1]:.4f} seconds")
```

--------------------------------

### Train Regularization Paths with optimize_path

Source: https://context7.com/pilancilab/scnn/llms.txt

Trains models across a sequence of regularization strengths. Uses warm-starting to improve efficiency during hyperparameter tuning.

```python
import numpy as np
from scnn.optimize import optimize_path
from scnn.models import ConvexGatedReLU
from scnn.solvers import RFISTA
from scnn.regularizers import NeuronGL1
from scnn.metrics import Metrics
from scnn.activations import sample_gate_vectors

# Setup
np.random.seed(42)
n, d = 500, 20
X_train = np.random.randn(n, d)
y_train = np.sign(np.random.randn(n))
X_test = np.random.randn(100, d)
y_test = np.sign(np.random.randn(100))

G = sample_gate_vectors(seed=123, d=d, n_samples=500)
model = ConvexGatedReLU(G)
solver = RFISTA(model, tol=1e-6)

# Define regularization path (decreasing lambda)
lambdas = [0.1, 0.05, 0.01, 0.005, 0.001]
path = [NeuronGL1(lam) for lam in lambdas]

metrics = Metrics(
    model_loss=True,
    train_accuracy=True,
    test_accuracy=True,
    neuron_sparsity=True
)

# Train across regularization path
models, metrics_list = optimize_path(
    model=model,
    solver=solver,
    path=path,
    metrics=metrics,
    X_train=X_train,
    y_train=y_train,
    X_test=X_test,
    y_test=y_test,
    warm_start=True,    # Use previous solution as starting point
    save_path=None,     # Set path to save models to disk
    verbose=True
)

# Analyze results
print("\nRegularization Path Results:")
print("-" * 50)
for i, (lam, m) in enumerate(zip(lambdas, metrics_list)):
    print(f"λ={lam:.4f}: train_acc={m.train_accuracy[-1]:.4f}, "
          f"test_acc={m.test_accuracy[-1]:.4f}, "
          f"sparsity={m.neuron_sparsity[-1]:.2%}")
```

--------------------------------

### Train a ReLU model with optimize function

Source: https://github.com/pilancilab/scnn/blob/main/docs/quick_start.md

Train a neural network using the `optimize` function. Specify the formulation (e.g., 'relu'), maximum neurons, training data, regularizer, and device. This method is suitable for quick training with default settings.

```python
from scnn.optimize import optimize
from scnn.regularizers import NeuronGL1

model, metrics = optimize(formulation="relu",
                          max_neurons=500,
                          X_train=X,
                          y_train=y,
                          regularizer=NeuronGL1(0.001),
                          device="cpu")

# training accuracy
train_acc = np.sum(np.sign(model(X_train)) == y_train) / len(y_train)
```

--------------------------------

### Train with Functional API

Source: https://github.com/pilancilab/scnn/blob/main/notebooks/solver-demo.ipynb

Use the optimize function for a simplified training workflow with gated ReLU activations.

```python
model, _ = optimize("gated_relu", 
                          max_neurons, 
                          X_train, 
                          y_train, 
                          X_test, 
                          y_test,
                          regularizer=NeuronGL1(0.01),
                          verbose=True,  
                          device="cpu")
```

--------------------------------

### Configure ApproximateConeDecomposition Solver

Source: https://context7.com/pilancilab/scnn/llms.txt

Implements a two-step training process for ReLU networks. It first solves the Gated ReLU problem and then decomposes the solution to obtain a valid ReLU network. Parameters like 'rho' control the decomposition accuracy.

```python
import numpy as np
from scnn.models import ConvexGatedReLU
from scnn.solvers import ApproximateConeDecomposition
from scnn.regularizers import NeuronGL1
from scnn.metrics import Metrics
from scnn.optimize import optimize_model
from scnn.activations import sample_gate_vectors

# Setup
np.random.seed(42)
n, d = 500, 20
X_train = np.random.randn(n, d)
y_train = np.sign(np.random.randn(n))

G = sample_gate_vectors(seed=123, d=d, n_samples=500)
# Start with Gated ReLU model; ReLU model will be output
model = ConvexGatedReLU(G)

# Configure cone decomposition solver
solver = ApproximateConeDecomposition(
    model=model,
    max_iters=10000,     # RFISTA iterations for Gated ReLU
    tol=1e-6,
    rho=1e-10,           # Decomposition penalty (smaller = better approx)
    d_max_iters=10000,   # Decomposition iterations
    d_tol=1e-10
)

metrics = Metrics(model_loss=True, train_accuracy=True)

# Returns a ReLU model, not Gated ReLU
relu_model, metrics = optimize_model(
    model, solver, metrics,
    X_train, y_train,
    regularizer=NeuronGL1(0.01),
    verbose=True
)

print(f"Output model type: {type(relu_model).__name__}")
```

--------------------------------

### Sample Convolutional Gates

Source: https://github.com/pilancilab/scnn/blob/main/docs/activations.md

Generate convolutional gate vectors by random sampling.

```APIDOC
## POST /scnn/activations/sample_convolutional_gates

### Description
Generate convolutional gate vectors by random sampling.

### Method
POST

### Endpoint
/scnn/activations/sample_convolutional_gates

### Parameters
#### Request Body
- **rng** (Generator) - Required - a NumPy random number generator.
- **d** (int) - Required - the dimensionality of the gate vectors.
- **n_samples** (int) - Required - the number of samples to use.

### Response
#### Success Response (200)
- **G** (ndarray) - a d x n_samples matrix of gate vectors.

#### Response Example
{
  "G": "[[0.1 0.5] [-0.2 0.8]]"
}
```

--------------------------------

### Access Model Properties

Source: https://context7.com/pilancilab/scnn/llms.txt

Prints key properties of the trained model, such as input dimension, output dimension, and number of neurons. Useful for verifying model configuration after training.

```python
print(f"Input dimension: {model.d}")
print(f"Output dimension: {model.c}")
print(f"Number of neurons: {model.p}")
```

--------------------------------

### scnn.optimize.optimize_path

Source: https://github.com/pilancilab/scnn/blob/main/docs/optimize.md

Trains a neural network by convex reformulation, exploring a regularization path.

```APIDOC
## scnn.optimize.optimize_path

### Description
Train a neural network by convex reformulation.

### Method
POST

### Endpoint
/pilancilab/scnn/optimize/optimize_path

### Parameters
#### Path Parameters
None

#### Query Parameters
- **warm_start** (bool) - Optional - whether or not to warm-start each optimization problem in the path using the previous solution.
- **save_path** (str) - Optional - string specifying a directory where models in the regularization path should be saved. All models will be retained in memory and returned if save_path = None.
- **return_convex** (bool) - Optional - whether or not to return the convex reformulation instead of the final non-convex model.
- **unitize_data** (bool) - Optional - whether or not to unitize the column norms of the training set. This can improve conditioning during optimization.
- **verbose** (bool) - Optional - whether or not the solver should print verbosely during optimization.
- **log_file** (str) - Optional - a path to an optional log file.
- **device** (typing_extensions.Literal['cpu', 'cuda']) - Optional - the device on which to run. Must be one of cpu (run on CPU) or cuda (run on cuda-enabled GPUs if available).
- **dtype** (typing_extensions.Literal['float32', 'float64']) - Optional - the floating-point type to use. “float32” is faster than “float64” but can lead to excessive numerical errors on badly conditioned datasets.
- **seed** (int) - Optional - an integer seed for reproducibility.

#### Request Body
- **model** (Model) - Required - a convex reformulation of a neural network model.
- **solver** (Optimizer) - Required - the optimizer to use when solving the reformulation.
- **path** (List[Regularizer]) - Required - a list of regularizer objects specifying the regularization path to explore.
- **metrics** (Metrics) - Required - a object specifying which metrics to collect during optimization.
- **X_train** (ndarray) - Required - an $n \times d$ matrix of training examples.
- **y_train** (ndarray) - Required - an $n \times c$ or vector matrix of training targets.
- **X_test** (ndarray | None) - Optional - an $m \times d$ matrix of test examples.
- **y_test** (ndarray | None) - Optional - an $n \times c$ or vector matrix of test targets.

### Request Example
{
  "model": "<Model Object>",
  "solver": "<Optimizer Object>",
  "path": ["<Regularizer Object 1>", "<Regularizer Object 2>"],
  "metrics": "<Metrics Object>",
  "X_train": "<Training Data Matrix>",
  "y_train": "<Training Targets Matrix>",
  "X_test": "<Test Data Matrix>",
  "y_test": "<Test Targets Matrix>",
  "warm_start": true,
  "save_path": "/path/to/save/models",
  "return_convex": false,
  "unitize_data": true,
  "verbose": true,
  "log_file": "/path/to/log.txt",
  "device": "cuda",
  "dtype": "float64",
  "seed": 1234
}

### Response
#### Success Response (200)
- **models** (List[Model | str]) - A list of trained models or their save paths.
- **metrics** (List[Metrics]) - A list of collected metrics for each model in the path.

#### Response Example
{
  "models": ["<Model Object 1>", "/path/to/save/models/model_2.pth"],
  "metrics": ["<Metrics Object 1>", "<Metrics Object 2>"]
}
```

--------------------------------

### SCNN Public API Overview

Source: https://github.com/pilancilab/scnn/blob/main/docs/documentation.md

This section outlines the primary modules and functions available in the SCNN library.

```APIDOC
## SCNN Library Modules

### Description
The SCNN library is organized into several functional modules for neural network optimization and analysis.

### Modules
- **scnn.optimize**: Functions for model optimization (`optimize`, `optimize_model`, `optimize_path`).
- **scnn.models**: Neural network architectures including `ConvexGatedReLU`, `ConvexReLU`, `GatedModel`, `LinearModel`, `Model`, `NonConvexGatedReLU`, and `NonConvexReLU`.
- **scnn.regularizers**: Regularization methods such as `FeatureGL1`, `L1`, `L2`, `NeuronGL1`, and `Regularizer`.
- **scnn.solvers**: Optimization solvers including `AL`, `ApproximateConeDecomposition`, `CVXPYSolver`, `ExactConeDecomposition`, `LeastSquaresSolver`, `Optimizer`, and `RFISTA`.
- **scnn.metrics**: Performance evaluation via `Metrics`.
- **scnn.loss_functions**: Loss definitions like `SquaredLoss`.
- **scnn.activations**: Utilities for activation patterns (`compute_activation_patterns`, `generate_index_lists`, `sample_convolutional_gates`, `sample_dense_gates`, `sample_gate_vectors`, `sample_sparse_gates`).
```

--------------------------------

### Generate Synthetic Data

Source: https://github.com/pilancilab/scnn/blob/main/notebooks/solver-demo.ipynb

Create a realizable synthetic classification dataset for training and testing.

```python
# Generate realizable synthetic classification problem (ie. Figure 1)
n_train = 1000
n_test = 1000
d = 50
hidden_units = 100
kappa = 10  # condition number

(X_train, y_train), (X_test, y_test) = gen_classification_data(123, n_train, n_test, d, hidden_units, kappa)
```

--------------------------------

### compute_activations

Source: https://github.com/pilancilab/scnn/blob/main/docs/models.md

Computes activations for models with fixed gate vectors.

```APIDOC
## compute_activations(X: ndarray)

### Description
Compute activations for models with fixed gate vectors.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **X** (ndarray) - Required - (n x d) matrix of input examples.

### Request Example
```json
{
  "X": "(n x d) matrix of input examples"
}
```

### Response
#### Success Response (200)
- **activations** (ndarray) - (n x p) matrix of activation patterns.

#### Response Example
```json
{
  "activations": "(n x p) matrix of activation patterns"
}
```
```

--------------------------------

### LeastSquaresSolver

Source: https://github.com/pilancilab/scnn/blob/main/docs/solvers.md

Direct solver for the unregularized or \(\\ell_2\)-regularized Gated ReLU training problem. Solves a convex quadratic problem equivalent to ridge regression using conjugate-gradient (CG) type methods like LSMR or LSQR.

```APIDOC
## LeastSquaresSolver

### Description
Direct solver for the unregularized or $\ell_2$-regularized Gated ReLU training problem. This is a convex quadratic problem equivalent to ridge regression which this optimizer solves using conjugate-gradient (CG) type methods. Either LSMR or LSQR can be used.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Notes
- This solver only supports computation on CPU.
- This solver only supports [`L2`](regularizers.md#scnn.regularizers.L2) regularization or no regularization.

### References
[1] D. C.-L. Fong and M. A. Saunders, LSMR: An iterative algorithm for sparse least-squares problems, SIAM J. Sci. Comput. 33:5, 2950-2971, published electronically Oct 27, 2011.
[2] C. C. Paige and M. A. Saunders, LSQR: An algorithm for sparse linear equations and sparse least squares, TOMS 8(1), 43-71 (1982)

### Constructor Parameters
- **model** ([Model](models.md#scnn.models.Model)) - The model that should be optimized.
- **solver** (str, optional) - The underlying CG-type solver to use. Defaults to 'lsmr'.
- **max_iters** (int, optional) - The maximum number of iterations to run the optimization method. Defaults to 1000.
- **tol** (float, optional) - The tolerance for terminating the optimization procedure early. Defaults to 1e-06.
```

--------------------------------

### scnn.activations.sample_sparse_gates

Source: https://github.com/pilancilab/scnn/blob/main/docs/activations.md

Generates feature-sparse gate vectors by random sampling.

```APIDOC
## scnn.activations.sample_sparse_gates

### Description
Generate feature-sparse gate vectors by random sampling.

### Method
N/A (This is a Python function, not an HTTP endpoint)

### Endpoint
N/A

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
* **rng** (Generator) - a NumPy random number generator.
* **d** (int) - the dimensionality of the gate vectors.
* **n_samples** (int) - the number of samples to use.
* **sparsity_indices** (List[List[int]]) - lists of indices (i.e. features) for which sparse gates should be generated. Each index list will get n_samples / len(sparsity_indices) gates which are sparse in every feature except the given indices.

### Notes
It is possible to obtain more than n_samples gate vectors if len(sparsity_indices) does not divide evenly into n_samples. In this case, we use the ceiling to avoid sampling zero gates for some sparsity patterns.

### Returns
* **G** (ndarray) - a $d \times \text{n_samples}$ matrix of gate vectors.
```

--------------------------------

### Train with Augmented Lagrangian

Source: https://github.com/pilancilab/scnn/blob/main/notebooks/solver-demo.ipynb

Directly solve the convex formulation of the ReLU training problem using the augmented Lagrangian method.

```python
model = ConvexReLU(G)
solver = AL(model)
relu_model, relu_metrics = optimize_model(
    model,
    solver,
    metrics,
    X_train, 
    y_train, 
    X_test, 
    y_test,
    regularizer,
    verbose=True,
)
```

```python
# Training Accuracy
np.sum(np.sign(relu_model(X_train)) == y_train) / len(y_train)
```

--------------------------------

### Make Predictions with Trained Model

Source: https://context7.com/pilancilab/scnn/llms.txt

Calculates predictions using the trained ConvexGatedReLU model and computes the training accuracy. This demonstrates how to use the model for inference after optimization.

```python
predictions = model(X_train)
accuracy = np.sum(np.sign(predictions) == y_train) / len(y_train)
print(f"Training accuracy: {accuracy:.4f}")
```

--------------------------------

### scnn.solvers.AL

Source: https://github.com/pilancilab/scnn/blob/main/docs/solvers.md

Augmented Lagrangian (AL) method for ReLU model optimization.

```APIDOC
## class scnn.solvers.AL

### Description
Augmented Lagrangian (AL) method for ReLU model optimization. This solver alternates between primal minimization and dual parameter updates.

### Parameters
- **model** (Model) - Required - The model to be optimized.
- **max_primal_iters** (int) - Optional - Maximum iterations for primal optimization (default: 10000).
- **max_dual_iters** (int) - Optional - Maximum dual updates (default: 10000).
- **tol** (float) - Optional - Tolerance for KKT conditions (default: 1e-06).
- **constraint_tol** (float) - Optional - Maximum constraint violation (default: 1e-06).
- **delta** (float) - Optional - Initial penalty strength (default: 1000).
```

--------------------------------

### Print AL Solver Constraint Gap

Source: https://context7.com/pilancilab/scnn/llms.txt

Prints the final constraint gap achieved by the Augmented Lagrangian solver after training. This metric indicates how well the model satisfies the optimization constraints.

```python
print(f"Final constraint gap: {metrics.constraint_gaps[-1]:.2e}")
```