jumpmodels

https://github.com/yizhan-oliver-shu/jump-models
Admin
jumpmodels is a Python library implementing statistical jump models for unsupervised regime...
Tokens:24,112
Snippets:155
Trust Score:6.3
License:Apache-2.0
Update:1 day ago
Show doc for...
Tokens:
Raw
### Install jumpmodels with example dependencies

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/README.md

Use this command to install jumpmodels along with additional dependencies (yfinance and jupyterlab) needed to run the example notebook.

```bash
pip install jumpmodels[example]
```

--------------------------------

### Install jumpmodels with pip

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/README.md

Use this command to install the jumpmodels package from PyPI. All required dependencies (numpy, pandas, scipy, scikit-learn, matplotlib) are installed automatically.

```bash
pip install jumpmodels
```

--------------------------------

### Setup and Imports for Jump Models

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb

Initializes the notebook environment with autoreload, imports utility functions, and imports the core classes JumpModel and SparseJumpModel along with helper functions for date filtering.

```python
%load_ext autoreload
%autoreload 2
from utils_dir import get_curr_dir, include_home_dir
include_home_dir()

import pandas as pd

from jumpmodels.utils import filter_date_range        # useful helpers
from jumpmodels.jump import JumpModel                 # class of JM & CJM
from jumpmodels.sparse_jump import SparseJumpModel    # class of Sparse JM
```

--------------------------------

### Time-based train/test split with filter_date_range

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb

Splits the data into training (2007-2021) and test (2022-2024) sets using the filter_date_range helper. Prints the actual start and end dates of each split.

```python
train_start, test_start = "2007-1-1", "2022-1-1"
# filter dates
X_train = filter_date_range(data.X, start_date=train_start, end_date=test_start)
X_test = filter_date_range(data.X, start_date=test_start)
# print time split
train_start, train_end = X_train.index[[0, -1]]
test_start, test_end = X_test.index[[0, -1]]
print("Training starts at:", train_start, "and ends at:", train_end)
print("Testing starts at:", test_start, "and ends at:", test_end)
```

```text
Output:
Training starts at: 2007-01-03 and ends at: 2021-12-31
Testing starts at: 2022-01-03 and ends at: 2024-09-27
```

--------------------------------

### Refit and plot with zero jump penalty

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb

Refits the JumpModel with the reset jump penalty and plots the resulting regimes. This comparison illustrates the value of the jump penalty for temporal consistency.

```python
# refit
jm.fit(X_train_processed, data.ret_ser, sort_by="cumret")

# plot
ax, ax2 = plot_regimes_and_cumret(jm.labels_, data.ret_ser, n_c=2, start_date=train_start, end_date=train_end, )
ax.set(title=f"In-Sample Fitted Regimes by the JM ($\\lambda$={jump_penalty})")
savefig_plt(f"{get_curr_dir()}/plots/JM_lambd-{jump_penalty}_train.pdf")
```

```text
Result:
<Figure size 2400x1200 with 2 Axes>
```

--------------------------------

### Fit SparseJumpModel with max_feats and jump_penalty

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb

Initializes and fits a SparseJumpModel with n_components=2, max_feats=3.0, and jump_penalty=50.0. The fit method takes processed training data and returns a fitted model. The output shows the model's parameters.

```python
max_feats=3.
jump_penalty=50.
# init sjm instance
sjm = SparseJumpModel(n_components=2, max_feats=max_feats, jump_penalty=jump_penalty, )
# fit
sjm.fit(X_train_processed, ret_ser=data.ret_ser, sort_by="cumret")
```

```text
Result:
SparseJumpModel(jump_penalty=50.0, max_feats=3.0)
```

--------------------------------

### Import JumpModel and SparseJumpModel classes

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/README.md

Import the two core classes from the jumpmodels package. The comments indicate that JumpModel is for JM and CJM, while SparseJumpModel is for sparse JM.

```python
from jumpmodels.jump import JumpModel                 # JM & CJM class
from jumpmodels.sparse_jump import SparseJumpModel    # Sparse JM class
```

--------------------------------

### Fit JumpModel with sort_by='cumret'

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb

Fits the JumpModel to the training data, using the return series to sort states by cumulative returns. The state with higher cumulative returns is labeled 0 (bull market) and the lower is labeled 1 (bear market).

```python
# call .fit()
jm.fit(X_train_processed, data.ret_ser, sort_by="cumret")
```

```text
Result:
JumpModel(jump_penalty=50.0)
```

--------------------------------

### Instantiate CJM with cont=True

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb

Creates a Continuous Jump Model using the same JumpModel class with cont=True. The jump penalty is set to 600.0, typically 10 times larger than the discrete model's lambda, to achieve similar fittings. The specialized parameters mode_loss and grid_size are left at their defaults.

```python
jump_penalty=600.
cjm = JumpModel(n_components=2, jump_penalty=jump_penalty, cont=True)
```

--------------------------------

### Online probabilistic inference with predict_proba_online

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb

Performs online probabilistic regime inference on test data using predict_proba_online(). The resulting probabilities show smoother transitions, e.g., bear market confidence in late 2024 doesn't fully reach 100%, potentially reducing mislabeling. The plot is saved to a PDF.

```python
# online inference
proba_test_online = cjm.predict_proba_online(X_test_processed)

# plot
ax, ax2 = plot_regimes_and_cumret(proba_test_online, data.ret_ser, start_date=test_start, end_date=test_end, )
ax2.set(ylabel="Regime Probability")
ax.set(title=f"Out-of-Sample Online Inferred Regimes by the CJM ($\\lambda$={jump_penalty})")
savefig_plt(f"{get_curr_dir()}/plots/CJM_lambd-{jump_penalty}_test_online.pdf")
```

```text
Result:
<Figure size 2400x1200 with 2 Axes>
```

--------------------------------

### Predict Regimes Using All Test Data

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb

Uses the .predict() method to decode states using all test data at once, then plots the resulting regimes and cumulative returns. This approach is less realistic for trading but may be useful in other fields.

```python
# make inference using all test data
labels_test = jm.predict(X_test_processed)
# plot
ax, ax2 = plot_regimes_and_cumret(labels_test, data.ret_ser, n_c=2, start_date=test_start, end_date=test_end, )
_ = ax.set(title=f"Out-of-Sample Predicted Regimes by the JM Using All Test Data ($\\lambda$={jump_penalty})")
```

--------------------------------

### Fit CJM and plot in-sample regimes

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb

Fits the CJM on processed training data and plots the resulting regime probabilities. The proba_ attribute stores smooth probability transitions (0-100%) for each period, unlike the discrete model's abrupt assignments. The plot is saved to a PDF.

```python
cjm.fit(X_train_processed, data.ret_ser, sort_by="cumret")

# plot
ax, ax2 = plot_regimes_and_cumret(cjm.proba_, data.ret_ser, n_c=2, start_date=train_start, end_date=train_end, )
ax2.set(ylabel="Regime Probability")
ax.set(title=f"In-Sample Fitted Regimes by the CJM ($\\lambda$={jump_penalty})")
savefig_plt(f"{get_curr_dir()}/plots/CJM_lambd-{jump_penalty}_train.pdf")
```

```text
Result:
<Figure size 2400x1200 with 2 Axes>
```

--------------------------------

### Preprocessing with DataClipperStd and StandardScalerPD

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb

Clips features to three standard deviations and standardizes them. Both transformers are fitted on training data and then applied to test data. Supports pandas DataFrames and numpy arrays.

```python
# Preprocessing
from jumpmodels.preprocess import StandardScalerPD, DataClipperStd
clipper = DataClipperStd(mul=3.)
scalar = StandardScalerPD()
# fit on training data
X_train_processed = scalar.fit_transform(clipper.fit_transform(X_train))
# transform the test data
X_test_processed = scalar.transform(clipper.transform(X_test))
```

--------------------------------

### Plot and Save Online Inferred Regimes

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb

Plots the online-inferred regimes and cumulative returns, then saves the figure to a PDF file. The plot title includes the jump penalty value.

```python
# plot and save
ax, ax2 = plot_regimes_and_cumret(labels_test_online, data.ret_ser, n_c=2, start_date=test_start, end_date=test_end, )
ax.set(title=f"Out-of-Sample Online Inferred Regimes by the JM ($\\lambda$={jump_penalty})")
savefig_plt(f"{get_curr_dir()}/plots/JM_lambd-{jump_penalty}_test_online.pdf")
```

--------------------------------

### Initialize JumpModel with jump penalty

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb

Sets the jump penalty to 50.0 and initializes a JumpModel with 2 components, cont=False for hard clustering. The jump penalty is a crucial hyperparameter that requires tuning.

```python
# set the jump penalty
jump_penalty=50.
# initlalize the JM instance
jm = JumpModel(n_components=2, jump_penalty=jump_penalty, cont=False, )
```

--------------------------------

### Refit Jump Model and Perform Online Inference

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb

Refits the Jump Model with a jump penalty of 50.0 and performs online inference on the test data using predict_online(). The online inference uses only data available up to each period, avoiding future data.

```python
# refit
jump_penalty=50.
jm.set_params(jump_penalty=jump_penalty).fit(X_train_processed, data.ret_ser, sort_by="cumret")
# make online inference 
labels_test_online = jm.predict_online(X_test_processed)
```

--------------------------------

### Plot in-sample SJM regimes

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb

Plots the in-sample fitted regimes using plot_regimes_and_cumret and saves the figure to a PDF. The title includes the jump penalty and max_feats values. The output shows a figure with 2 axes.

```python
# plot
ax, ax2 = plot_regimes_and_cumret(sjm.labels_, data.ret_ser, n_c=2, start_date=train_start, end_date=train_end, )
ax.set(title=f"In-Sample Fitted Regimes by the SJM ($\\lambda$={jump_penalty}, $\\kappa^2$={max_feats})")
savefig_plt(f"{get_curr_dir()}/plots/SJM_lambd-{jump_penalty}_max-feats-{max_feats}_train.pdf")
```

```text
Result:
<Figure size 2400x1200 with 2 Axes>
```

--------------------------------

### Plot in-sample fitted regimes

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb

Uses the jumpmodels.plot module to plot the in-sample fitted regimes and cumulative returns. The labels_ attribute contains the regime assignment for each period. The plot is saved as a PDF.

```python
from jumpmodels.plot import plot_regimes_and_cumret, savefig_plt

ax, ax2 = plot_regimes_and_cumret(jm.labels_, data.ret_ser, n_c=2, start_date=train_start, end_date=train_end, )
ax.set(title=f"In-Sample Fitted Regimes by the JM ($\\lambda$={jump_penalty})")
savefig_plt(f"{get_curr_dir()}/plots/JM_lambd-{jump_penalty}_train.pdf")
```

```text
Result:
<Figure size 2400x1200 with 2 Axes>
```

--------------------------------

### Reset jump penalty to zero with set_params()

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb

Resets the jump penalty to zero using set_params(), effectively reducing the model to a baseline k-means clustering algorithm. This demonstrates the utility of set_params() for refitting with different penalties.

```python
# reset jump_penalty to zero
jump_penalty=0.
jm.set_params(jump_penalty=jump_penalty)
print("The jump penalty of the JM instance has been reset to: jm.jump_penalty =", jm.jump_penalty)
```

```text
Output:
The jump penalty of the JM instance has been reset to: jm.jump_penalty = 0.0
```

--------------------------------

### BibTeX citation for Aydinhan et al. (2024) CJM paper

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/README.md

BibTeX entry for the Continuous Statistical Jump Models (CJM) paper by Aydinhan et al. (2024), published in Annals of Operations Research.

```bibtex
@article{Aydinhan2024CJM,
    title   = {Identifying patterns in financial markets: extending the statistical jump model for regime identification},
    author  = {Afşar Onat Aydınhan and Petter N. Kolm and John M. Mulvey and Yizhan Shu},
    journal = {Annals of Operations Research},
    year    = {2024},
    note    = {To appear},
    doi     = {https://doi.org/10.1007/s10479-024-06035-z},
}
```

--------------------------------

### Online inference with SJM

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb

Performs online inference on test data using sjm.predict_online() and plots the out-of-sample online inferred regimes. The plot is saved to a PDF. The output shows a figure with 2 axes.

```python
# online inference
labels_test_online_sjm = sjm.predict_online(X_test_processed)

# plot
ax, ax2 = plot_regimes_and_cumret(labels_test_online_sjm, data.ret_ser, start_date=test_start, end_date=test_end, )
ax.set(title=f"Out-of-Sample Online Inferred Regimes by the SJM ($\\lambda$={jump_penalty}, $\\kappa^2$={max_feats})")
savefig_plt(f"{get_curr_dir()}/plots/SJM_lambd-{jump_penalty}_max-feats-{max_feats}_test_online.pdf")
```

```text
Result:
<Figure size 2400x1200 with 2 Axes>
```

--------------------------------

### Load data and features with DataLoader

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb

Loads daily returns and engineered features for the NDX index from 2007 to September 2024 using the DataLoader class from feature.py. The loaded data is stored in data.ret_ser (daily returns) and data.X (features). The output shows the first and last rows of each.

```python
from feature import DataLoader

data = DataLoader(ticker="NDX", ver="v0").load(start_date="2007-1-1", end_date="2024-09-30")

print("Daily returns stored in `data.ret_ser`:", "-"*50, sep="\n")
print(data.ret_ser, "-"*50, sep="\n")
print("Features stored in `data.X`:", "-"*50, sep="\n")
print(data.X)
```

```text
Output:
Daily returns stored in `data.ret_ser`:
--------------------------------------------------
date
2007-01-03    0.001406
2007-01-04    0.019064
2007-01-05   -0.004244
2007-01-08    0.001031
2007-01-09    0.004751
                ...   
2024-09-23    0.003067
2024-09-24    0.004667
2024-09-25    0.001392
2024-09-26    0.007156
2024-09-27   -0.005315
Name: NDX, Length: 4465, dtype: float64
--------------------------------------------------
Features stored in `data.X`:
--------------------------------------------------
               ret_5  DD-log_5  sortino_5    ret_20  DD-log_20  sortino_20    ret_60  DD-log_60  sortino_60
date                                                                                                       
2007-01-03 -0.000860 -5.398836  -0.190234  0.000296  -5.220386    0.054761  0.000600  -5.099043    0.098343
2007-01-04  0.001719 -5.468150   0.407422  0.000935  -5.237714    0.176058  0.000812  -5.104819    0.133865
2007-01-05  0.000947 -5.467368   0.224275  0.000759  -5.243912    0.143733  0.000754  -5.107760    0.124659
2007-01-08  0.000958 -5.536682   0.243121  0.000768  -5.261241    0.148030  0.000757  -5.113536    0.125909
2007-01-09  0.001449 -5.605997   0.394142  0.000904  -5.278569    0.177218  0.000803  -5.119312    0.134308
...              ...       ...        ...       ...        ...         ...       ...        ...         ...
2024-09-23  0.003214 -4.944754   0.451332  0.001207  -4.621159    0.122670  0.000993  -4.717871    0.111166
2024-09-24  0.003402 -5.014068   0.512031  0.001325  -4.638488    0.136997  0.001035  -4.723648    0.116560
2024-09-25  0.003142 -5.083383   0.506817  0.001327  -4.655816    0.139633  0.001039  -4.729424    0.117700
2024-09-26  0.003661 -5.152698   0.633045  0.001526  -4.673145    0.163326  0.001110  -4.735200    0.126383
2024-09-27  0.002499 -5.162861   0.436548  0.001293  -4.684799    0.140005  0.001036  -4.738852    0.118410

[4465 rows x 9 columns]
```

--------------------------------

### Print scaled cluster centroids

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb

Prints the scaled cluster centroids stored in the centers_ attribute, indexed by Bull and Bear states. The values are scaled, making direct interpretation hard, but show clear separation between regimes.

```python
print("Scaled Cluster Centroids:", pd.DataFrame(jm.centers_, index=["Bull", "Bear"], columns=X_train.columns), sep="\n" + "-"*50 + "\n")
```

```text
Output:
Scaled Cluster Centroids:
--------------------------------------------------
         ret_5  DD-log_5  sortino_5    ret_20  DD-log_20  sortino_20    ret_60  DD-log_60  sortino_60
Bull  0.154342 -0.296901   0.157169  0.296358  -0.318770    0.274053  0.336652  -0.245783    0.336974
Bear -0.643127  1.237157  -0.654906 -1.234891   1.328281   -1.141952 -1.402793   1.024153   -1.404137
```

--------------------------------

### BibTeX citation for Nystrup et al. (2021) Sparse JM paper

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/README.md

BibTeX entry for the Sparse Jump Models paper by Nystrup, Kolm, and Lindström (2021), published in Expert Systems with Applications.

```bibtex
@article{nystrup2021SJM,
    title     = {Feature selection in jump models},
    author    = {Peter Nystrup and Petter N. Kolm and Erik Lindstr{"o}m},
    journal   = {Expert Systems with Applications},
    volume    = {184},
    pages     = {115558},
    year      = {2021},
    doi       = {https://doi.org/10.1016/j.eswa.2021.115558},
}
```

--------------------------------

### BibTeX citation for Nystrup et al. (2020a) original JM paper

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/README.md

BibTeX entry for the original Statistical Jump Models paper by Nystrup, Lindström, and Madsen (2020a), published in Expert Systems with Applications.

```bibtex
@article{Nystrup2020JM,
    title   = {Learning hidden {Markov} models with persistent states by penalizing jumps},
    author  = {Peter Nystrup and Erik Lindstr{"o}m and Henrik Madsen},
    journal = {Expert Systems with Applications},
    year    = {2020},
    pages   = {113307},
    volume  = {150},
    doi     = {https://doi.org/10.1016/j.eswa.2020.113307},
}
```

--------------------------------

### Print SJM feature weights

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb

Prints the feature weights stored in the sjm.feat_weights attribute. The output shows that features with longer halflives receive higher weights, while noisier features are assigned zero weight.

```python
print("SJM Feature Weights:", "-"*50, sjm.feat_weights, sep="\n")
```

```text
Output:
SJM Feature Weights:
--------------------------------------------------
ret_5         0.000000
DD-log_5      0.251484
sortino_5     0.000000
ret_20        0.000000
DD-log_20     0.604436
sortino_20    0.025404
ret_60        0.753773
DD-log_60     0.000000
sortino_60    0.857114
dtype: float64
```

--------------------------------

### BibTeX citation for Nystrup et al. (2020b) online inference paper

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/README.md

BibTeX entry for the Online Inference for JMs paper by Nystrup, Kolm, and Lindström (2020b), published in The Journal of Financial Data Science.

```bibtex
@article{Nystrup2020onlineJM,
    title   = {Greedy Online Classification of Persistent Market States Using Realized Intraday Volatility Features},
    author  = {Peter Nystrup and Petter N. Kolm and Erik Lindstr{"o}m},
    journal = {The Journal of Financial Data Science}
    year    = {2020},
    volume  = {2},
    number  = {3},
    pages   = {25--39},
    doi     = {https://doi.org/10.3905/jfds.2020.2.3.025},
}
```

--------------------------------

### Result of Online Inference Plot

Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb

Output of the plotting code, indicating a figure with 2 axes was created.

```text
Result:
<Figure size 2400x1200 with 2 Axes>
```
Tokens:
Raw