SparseSC

https://sparsesc.readthedocs.io/en/latest/
Admin
SparseSC is a Python package that implements machine learning-enhanced Synthetic Control...
Tokens:23,577
Snippets:138
Trust Score:9.6
Update:1 week ago
Show doc for...
Tokens:
Raw
### Install super_batch library with pip

Source: https://sparsesc.readthedocs.io/en/latest/azure_batch.html

Install the required super_batch library from GitHub. This is needed to run SparseSC with Azure Batch. The module has only been tested with Python 3.7, and you will also need pyyaml and psutil.

```bash
pip install git+https://github.com/jdthorpe/batch-config.git

```

--------------------------------

### Fit a SparseSC model and make predictions

Source: https://sparsesc.readthedocs.io/en/latest/fit.html

Use the fit() function to fit a synthetic control model, then use the model's predict() method to get in-sample predictions for the targets and predictions for additional held-out features.

```python
from SparseSC import fit

# Fit the model:
fitted_model = fit(features,targets,...)

# Get the fitted synthetic controls for `targets`:
in_sample_predictions = fitted_model.predict()

# Make predictions for a held out set of fetures (targets_hat) 
# using the fitted synthetic controls model:
additional_predictions = fitted_model.predict(targets_additional)
```

--------------------------------

### Reconnect to Azure Batch job and load results

Source: https://sparsesc.readthedocs.io/en/latest/azure_batch.html

Use this code to reconnect to a previously created Azure Batch job and download the results after a disconnection. It assumes you have recreated the batch_client as shown in the example above.

```python
batch_client.load_results()
fitted_model = aggregate_batch_results(batch_dir)

```

--------------------------------

### Fit SparseSC.estimate_effects and access results

Source: https://sparsesc.readthedocs.io/en/latest/estimate-effects.html

Example usage of the `estimate_effects()` function from the SparseSC package to conduct a differences-in-differences style analysis with sparse synthetic controls. The snippet shows how to fit the model, print a summary, extract the p-value and confidence interval of the average joint effect, and access the fitted synthetic controls model. The returned object is of class `SparseSCEstResults`.

```python
import SparseSC

# Fit the model:
fitted_estimates = SparseSC.estimate_effects(outcomes,unit_treatment_periods,covariates=X,fast=True,...)

# Print summary of the model including effect size estimates, 
# p-values, and confidendence intervals:
print(fitted_estimates)

# Extract model attributes:
fitted_estimates.pl_res_post.avg_joint_effect.p_value
fitted_estimates.pl_res_post.avg_joint_effect.CI

# access the fitted Synthetic Controls model:
fitted_model = fitted_estimates.fit

```

--------------------------------

### Custom donor pool for interpolation bias example

Source: https://sparsesc.readthedocs.io/en/latest/overview.html

Uses calipers to restrict matching to units with a maximum difference of thresh on a key metric B (an N-vector), limiting interpolation bias. Combines the caliper restriction with the default donor pool.

```python
custom_donor_pool = (np.abs(B[:,np.newaxis] - B) < thresh)[:,control_units].astype('bool')
custom_donor_pool = np.logical_and(custom_donor_pool, base_custom_donor_pool)

```

--------------------------------

### Prepare batch parameters with fit()

Source: https://sparsesc.readthedocs.io/en/latest/azure_batch.html

Create the parameters required for a batch job by calling fit() with a batchDir argument. The parameters files are stored in the specified directory.

```python
from SparseSC import fit
batch_dir = "/path/to/my/batch/data/"

# initialize the batch parameters in the directory `batch_dir`
fit(x, y, ... , batchDir = batch_dir)

```

--------------------------------

### Run unittest commands for SparseSC tests

Source: https://sparsesc.readthedocs.io/en/latest/dev_notes.html

Commands to run the built-in unittest suite from the repo root. The first runs a test file (requires Python >=3.5), the second runs a module, the third runs a class, and the fourth runs a single test function.

```bash
python -m unittest test/test_fit.py #file (only Python >=3.5)
python -m unittest test.test_fit #module
python -m unittest test.test_fit.TestFit #class
python -m unittest test.test_fit.TestFit.test_retrospective #function

```

--------------------------------

### SparseSC.get_W

Source: https://sparsesc.readthedocs.io/en/latest/api_ref.html

Get W (np.ndarray 2D or pd.DataFrame depends on what was passed in) for one of the treatment periods.

```APIDOC
## SparseSC.get_W

### Description
Get W (np.ndarray 2D or pd.DataFrame depends on what was passed in) for one of the treatment periods.

### Parameters
- **treatment_period** (optional) - The treatment period to get W for.
```

--------------------------------

### SparseSC.__init__

Source: https://sparsesc.readthedocs.io/en/latest/api_ref.html

Initializes the SparseSC estimator with outcomes, fits, treatment period information, and placebo results. This constructor sets up the model for estimating treatment effects.

```APIDOC
## SparseSC.__init__

### Description
Initializes the SparseSC estimator with outcomes, fits, treatment period information, and placebo results. This constructor sets up the model for estimating treatment effects.

### Method
Constructor

### Parameters
- **outcomes** - Outcome for the whole sample
- **fits** (dictionary of period->SparseSCFit) - The fit() return objects
- **unit_treatment_periods** - Vector of treatment periods for each unit (use np.NaN if never treated, np.datetime64('NaT') if using DateTime)
- **unit_treatment_periods_idx** - The conversion of unit_treatment_periods to indexes
- **unit_treatment_periods_idx_fit** - The treatment period indexes passed to fit
- **T0** - Pre-history to match over
- **T1** - post-history to evaluate over
- **pl_res_pre** (PlaceboResults) - Statistics for the average fit of the treated units in the pre-period
- **pl_res_post** (PlaceboResults) - Statistics for the average treatment effect in the post-period
- **pl_res_post_scaled** (PlaceboResults) - Statistics for the average scaled treatment effect
- **max_n_pl** - Maximum number of placebo effects used for inference
- **covariates** (optional) - Nxk matrix of full baseline covariates
- **ind_CI** (optional, dictionary of period->CI_int) - Confidence intervals for SC predictions at the unit level
- **model_type** (optional) - Model type string
- **T2** (optional) - T2 (if prospective-type design)
- **pl_res_post_fit** (optional) - PlaceboResults for target period used for fit (if prospective-type design)
```

--------------------------------

### Custom donor pool for interference example

Source: https://sparsesc.readthedocs.io/en/latest/overview.html

Constructs a custom donor pool from an adjacency matrix A to prevent treatment spillover to adjacent units (interference/SUTVA violations). It combines the adjacency-based restriction with the default donor pool.

```python
custom_donor_pool = (1 - A)[:,control_units].astype('bool')
custom_donor_pool = np.logical_and(custom_donor_pool, base_custom_donor_pool)

```

--------------------------------

### PlaceboResults.__init__

Source: https://sparsesc.readthedocs.io/en/latest/api_ref.html

Holds statistics for a vector of effects, including the full vector and two choices of aggregates (average and RMS).

```APIDOC
## PlaceboResults.__init__

### Description
Holds statistics for a vector of effects, including the full vector and two choices of aggregates (average and RMS).

### Parameters
- **effect_vec** (EstResultCI) - Statistics for a vector of time-specific effects.
- **avg_joint_effect** (EstResultCI) - Statistics for the average effect.
- **rms_joint_effect** (EstResultCI) - Statistics for the RMS effect.
- **N_placebo** (EstResultCI) - Number of placebos used for the statistics.
```

--------------------------------

### Execute Azure Batch job with super_batch Client

Source: https://sparsesc.readthedocs.io/en/latest/azure_batch.html

Creates a Batch configuration and runs the job. Credentials are taken from environment variables. The job name must be unique, and a timestamp is used. The pool is kept after the run to save time for multiple jobs. After running, results are aggregated into a fitted model.

```python
import os
from datetime import datetime
from super_batch import Client
from SparseSC.utils.AzureBatch import (
    DOCKER_IMAGE_NAME,
    create_job,
)
# Batch job names must be unique, and a timestamp is one way to keep it uniquie across runs
timestamp = datetime.utcnow().strftime("%H%M%S")
batch_dir = "/path/to/my/batch/data/"

batch_client = Client(
    # Name of the VM pool
    POOL_ID= name,
    # number of standard nodes
    POOL_NODE_COUNT=5,
    # number of low priority nodes
    POOL_LOW_PRIORITY_NODE_COUNT=5,
    # VM type
    POOL_VM_SIZE= "STANDARD_A1_v2",
    # Job ID.  Note that this must be unique.
    JOB_ID= name + timestamp,
    # Name of the storage container for storing parameters and results
    CONTAINER_NAME= name,
    # local directory with the parameters, and where the results will go
    BATCH_DIRECTORY= batch_dir,
    # Keep the pool around after the run, which saves time when doing
    # multiple batch jobs, as it typically takes a few minutes to spin up a
    # pool of VMs. (Optional. Default = False)
    DELETE_POOL_WHEN_DONE=False,
    # Keeping the job details can be useful for debugging:
    # (Optional. Default = False)
    DELETE_JOB_WHEN_DONE=False
)

create_job(batch_client, batch_dir)
# run the batch job
batch_client.run()

# aggregate the results into a fitted model instance
fitted_model = aggregate_batch_results(batch_dir)

```

--------------------------------

### Create Anaconda environment from YAML file

Source: https://sparsesc.readthedocs.io/en/latest/dev_notes.html

Creates a conda environment from the test/SparseSC_36.yml file. Use 'update' instead of 'create' to update an existing environment, but ensure the environment is not currently active to avoid potential bugs.

```bash
conda env create -f test/SparseSC_36.yml

```

--------------------------------

### Grid search over v_pen and w_pen using itertools.product

Source: https://sparsesc.readthedocs.io/en/latest/fit.html

Use this snippet to evaluate a grid of penalty parameters when both v_pen and w_pen are vectors. Passing vectors to both parameters directly raises an error, so this approach is required for an N x N grid.

```python
from intertools import product
fitted_models = [ fit(..., v_pen=v, w_pen=w) for v,w in product(v_pen,w_pen)]

```

--------------------------------

### SparseSCFit.summary

Source: https://sparsesc.readthedocs.io/en/latest/api_ref.html

A summary of the model fit / penalty selection. This illustrates that (a) the grid function could / should be better, and (b) currently more than two iterations is typically useless.

```APIDOC
## SparseSCFit.summary

### Description
A summary of the model fit / penalty selection. This illustrates that (a) the grid function could / should be better, and (b) currently more than two iterations is typically useless.

### Method
Function call
```

--------------------------------

### fit()

Source: https://sparsesc.readthedocs.io/en/latest/fit.html

Fits a synthetic control model using provided penalty parameters. Supports passing v_pen and w_pen as floats or vectors, with optional grid search and cross-validation.

```APIDOC
## fit()

### Description
Fits a synthetic control model using penalty parameters for the V and W matrices. The function can accept single penalty values or perform a grid search over one penalty parameter while holding the other fixed, using cross-validation to select the optimal model.

### Method
Function call

### Endpoint
fit(v_pen, w_pen, ...)

### Parameters
#### Penalty Parameters
- **v_pen** (float or array-like) - Required - Penalty for the V matrix. If a single float, used directly. If an array, a grid search is performed over these values.
- **w_pen** (float or array-like) - Required - Penalty for the W matrix. If a single float, used directly. If an array, a grid search is performed over these values.

#### Selection Parameters
- **choice** (string or callable) - Optional - Model selection rule when searching over a vector of penalties. Options: "min" (default) selects the model with smallest out-of-sample error, "1se" implements the one-standard-error rule, or a custom function.

#### Grid Search Parameters
- **grid_length** (int) - Optional - Number of points in the search grid (default 20).
- **grid_min** (float) - Optional - Lower bound for the search grid.
- **grid_max** (float) - Optional - Upper bound for the search grid.
- **grid** (array-like) - Optional - Array of values between 0 and 1, multiplied by the relevant grid_max to determine the search grid.
- **stopping_rule** (parameter) - Optional - Determines how long the coordinate descent alternates between searching over V and W penalties.

#### Constraint Parameters
- **constrain** (string) - Optional - Constraint for the V matrix: "simplex" (default) or "orthant".

#### Fold Parameters
- **cv_fold** (int or list-of-lists) - Optional - Number of folds or list of unit indices for cross-validation folds.
- **gradient_fold** (int or list-of-lists) - Optional - Number of folds or list of unit indices for gradient descent folds.
- **cv_seed** (int) - Optional - Seed for cross-validation fold generation.
- **gradient_seed** (int) - Optional - Seed for gradient descent fold generation.

### Request Example
```python
from intertools import product
fitted_models = [ fit(..., v_pen=v, w_pen=w) for v,w in product(v_pen,w_pen)]
```

### Response
#### Success Response
Returns a fitted synthetic control model.

#### Error Handling
- Passing vectors to both `v_pen` and `w_pen` raises an error.

```

--------------------------------

### AA_results.__init__(diffs_pre, diffs_post, level=0.95, sym_CI=True)

Source: https://sparsesc.readthedocs.io/en/latest/api_ref.html

Constructs a simple results object from an AA test that has already been fit (model_type="full") and diffs constructed. Provides typical stats for simulations of estimation.

```APIDOC
## AA_results.__init__(diffs_pre, diffs_post, level=0.95, sym_CI=True)

### Description
Constructs a simple results object from an AA test that has already been fit (model_type="full") and diffs constructed. Provides typical stats for simulations of estimation.

### Method
Constructor

### Endpoint
AA_results.__init__

### Parameters
#### Path Parameters
- **diffs_pre** - Required - Pre-treatment diffs
- **diffs_post** - Required - vector of CI lower bounds
- **level** (float) - Optional - level (default: 0.95)
- **sym_CI** (bool) - Optional - Return symmetric CIs. Will also set "treatment" effect to 0 rather than mean of placebo diffs. (default: True)

### Returns
AA_results
```

--------------------------------

### CI_int.__init__

Source: https://sparsesc.readthedocs.io/en/latest/api_ref.html

Class to hold information for a confidence interval (for single point or for a vector).

```APIDOC
## CI_int.__init__

### Description
Class to hold information for a confidence interval (for single point or for a vector).

### Parameters
- **ci_low** (scalar, vector, or pd.Series) - Low-bound
- **ci_high** (scalar, vector, or pd.Series) - High-bound
- **level** (float) - Level (1-alpha) for the CI interval
```

--------------------------------

### fit() and fit_fast()

Source: https://sparsesc.readthedocs.io/en/latest/fit.html

These functions fit synthetic controls using the provided features and targets matrices. The model_type parameter selects among four model types, and the treated_units parameter indicates which units are treated.

```APIDOC
## fit() and fit_fast()

### Description
Fit synthetic controls using the provided features and targets matrices. The model_type parameter selects among four model types, and the treated_units parameter indicates which units are treated.

### Method
Function call

### Endpoint
fit(features, targets, model_type, treated_units)
fit_fast(features, targets, model_type, treated_units)

### Parameters
#### features
- **features** (matrix) - Required - Numeric matrix with one row per unit and one column per feature variable.

#### targets
- **targets** (matrix) - Required - Numeric matrix with one row per unit and one column per target variable.

#### model_type
- **model_type** (string) - Required - One of: "retrospective", "prospective", "prospective-restricted", or "full".

#### treated_units
- **treated_units** (vector) - Required for retrospective and prospective models; unused for full model - Indicates which units are treated (affected by intervention or designated to receive treatment).

### Request Example
```
fit(features = feature_matrix, targets = target_matrix, model_type = "retrospective", treated_units = c(1, 2, 3))
```

### Response
#### Success Response
Returns a fitted synthetic control model object.

#### Response Example
```
Synthetic control model object
```
```

--------------------------------

### SparseSC.utils.match_space.Fixed_V_factory

Source: https://sparsesc.readthedocs.io/en/latest/api_ref.html

Returns a MatchSpace function with user-supplied V over raw X. This is used to define a fixed match space.

```APIDOC
## SparseSC.utils.match_space.Fixed_V_factory

### Description
Returns a MatchSpace function with user-supplied V over raw X. This is used to define a fixed match space.

### Method
Function call

### Endpoint
`SparseSC.utils.match_space.Fixed_V_factory(V)`

### Parameters
- **V** - Required - V Matrix on the raw features

### Returns
A function with the signature MatchSpace fn, V vector, best_v_pen, V = function(X,Y)
```

--------------------------------

### SparseSCFit.show

Source: https://sparsesc.readthedocs.io/en/latest/api_ref.html

Display goodness of figures illustrating goodness of fit.

```APIDOC
## SparseSCFit.show

### Description
Display goodness of figures illustrating goodness of fit.

### Method
Function call
```

--------------------------------

### Fit and predict with SparseSC for anomaly detection

Source: https://sparsesc.readthedocs.io/en/latest/examples/prospective_anomaly_detection.html

Fits a synthetic control model for prospective anomaly detection using the `fit()` function from `SparseSC`. Divide the outcome matrix into pre-intervention features (`X`) and post-intervention targets (`Y`) at column `t`. Pass `model_type="full"` to allow every unit to serve as a donor unless `custom_donor_pool` is specified; the fit is computationally intensive. After fitting, use the `predict()` method on new observations, which is fast and suitable for real-time anomaly detection.

```python
from numpy import hstack
from SparseSC import fit

# Let X be the features plus some of the targets
X = hstack([features, targets[:,:t])

# And let Y be the remaining targets
Y = targets[:,t:]

# fit the model:
fitted_model = fit(X=X,
                   Y=Y,
                   model_type="full")

```

```python
synthetic_controls = fitted_model.predict(additional_observations)

```

--------------------------------

### SparseSC.fit

Source: https://sparsesc.readthedocs.io/en/latest/api_ref.html

Handy accessor if there is only one treatment-period.

```APIDOC
## SparseSC.fit

### Description
Handy accessor if there is only one treatment-period.
```

--------------------------------

### SparseSC.utils.match_space.MTLassoMixed_MatchSpace_factory

Source: https://sparsesc.readthedocs.io/en/latest/api_ref.html

Returns a MatchSpace function that will fit a MultiTaskLassoCV for Y ~ X with the penalization optimized to reduce errors on goal units.

```APIDOC
## SparseSC.utils.match_space.MTLassoMixed_MatchSpace_factory

### Description
Returns a MatchSpace function that will fit a MultiTaskLassoCV for Y ~ X with the penalization optimized to reduce errors on goal units.

### Method
Function

### Endpoint
SparseSC.utils.match_space.MTLassoMixed_MatchSpace_factory

### Parameters
#### Parameters
- **v_pens** (optional) - Penalties to evaluate (default is to automatically determine)
- **n_v_cv** (optional) - Number of Cross-Validation folds

### Returns
- MatchSpace fn
- V vector
- best_v_pen
- V
```

--------------------------------

### EstResultCI.__init__

Source: https://sparsesc.readthedocs.io/en/latest/api_ref.html

Holds an estimation result (effect + statistical significance).

```APIDOC
## EstResultCI.__init__

### Description
Holds an estimation result (effect + statistical significance).

### Parameters
- **effect** (scalar, vector, or pd.Series) - Effect
- **p** (scalar or vector) - p-value
- **ci** (CI_int) - Confidence interval
- **placebos** (matrix) - Full matrix of placebos
```

--------------------------------

### SparseSC.get_tr_time_info

Source: https://sparsesc.readthedocs.io/en/latest/api_ref.html

Returns treatment info: a) indexes for the time index (helpful if user used an np datetime index) b) treatment period used in the call to fit (helpful if model is a prospective type).

```APIDOC
## SparseSC.get_tr_time_info

### Description
Returns treatment info: a) indexes for the time index (helpful if user used an np datetime index) b) treatment period used in the call to fit (helpful if model is a prospective type).

### Parameters
- **treatment_period** - Treatment period in the user's view (could be TimeIndex)

### Returns
- (treatment_period_idx, treatment_period_idx_fit, treatment_period_fit)
```

--------------------------------

### SparseSC.utils.match_space.MTLassoCV_MatchSpace_factory

Source: https://sparsesc.readthedocs.io/en/latest/api_ref.html

Returns a MatchSpace function that will fit a MultiTaskLassoCV for Y ~ X. This is used to automatically determine the match space.

```APIDOC
## SparseSC.utils.match_space.MTLassoCV_MatchSpace_factory

### Description
Returns a MatchSpace function that will fit a MultiTaskLassoCV for Y ~ X. This is used to automatically determine the match space.

### Method
Function call

### Endpoint
`SparseSC.utils.match_space.MTLassoCV_MatchSpace_factory(v_pens=None, n_v_cv=5, sample_frac=1, Y_col_block_size=None, se_factor=None, normalize=True)`

### Parameters
- **v_pens** - Optional - Penalties to evaluate (default is to automatically determince)
- **n_v_cv** - Optional - Number of Cross-Validation folds
- **sample_frac** - Optional - Fraction of the data to sample
- **se_factor** - Optional - Allows taking a different penalty than the min mse. Similar to the lambda.1se rule, if not None, it will take the max lambda that has mse < min_mse + se_factor*(MSE standard error).

### Returns
MatchSpace fn, V vector, best_v_pen, V
```
Tokens:
Raw