### Install jumpmodels with example dependencies Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/README.md Use this command to install jumpmodels along with additional dependencies (yfinance and jupyterlab) needed to run the example notebook. ```bash pip install jumpmodels[example] ``` -------------------------------- ### Install jumpmodels with pip Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/README.md Use this command to install the jumpmodels package from PyPI. All required dependencies (numpy, pandas, scipy, scikit-learn, matplotlib) are installed automatically. ```bash pip install jumpmodels ``` -------------------------------- ### Setup and Imports for Jump Models Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb Initializes the notebook environment with autoreload, imports utility functions, and imports the core classes JumpModel and SparseJumpModel along with helper functions for date filtering. ```python %load_ext autoreload %autoreload 2 from utils_dir import get_curr_dir, include_home_dir include_home_dir() import pandas as pd from jumpmodels.utils import filter_date_range # useful helpers from jumpmodels.jump import JumpModel # class of JM & CJM from jumpmodels.sparse_jump import SparseJumpModel # class of Sparse JM ``` -------------------------------- ### Time-based train/test split with filter_date_range Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb Splits the data into training (2007-2021) and test (2022-2024) sets using the filter_date_range helper. Prints the actual start and end dates of each split. ```python train_start, test_start = "2007-1-1", "2022-1-1" # filter dates X_train = filter_date_range(data.X, start_date=train_start, end_date=test_start) X_test = filter_date_range(data.X, start_date=test_start) # print time split train_start, train_end = X_train.index[[0, -1]] test_start, test_end = X_test.index[[0, -1]] print("Training starts at:", train_start, "and ends at:", train_end) print("Testing starts at:", test_start, "and ends at:", test_end) ``` ```text Output: Training starts at: 2007-01-03 and ends at: 2021-12-31 Testing starts at: 2022-01-03 and ends at: 2024-09-27 ``` -------------------------------- ### Refit and plot with zero jump penalty Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb Refits the JumpModel with the reset jump penalty and plots the resulting regimes. This comparison illustrates the value of the jump penalty for temporal consistency. ```python # refit jm.fit(X_train_processed, data.ret_ser, sort_by="cumret") # plot ax, ax2 = plot_regimes_and_cumret(jm.labels_, data.ret_ser, n_c=2, start_date=train_start, end_date=train_end, ) ax.set(title=f"In-Sample Fitted Regimes by the JM ($\\lambda$={jump_penalty})") savefig_plt(f"{get_curr_dir()}/plots/JM_lambd-{jump_penalty}_train.pdf") ``` ```text Result:
``` -------------------------------- ### Fit SparseJumpModel with max_feats and jump_penalty Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb Initializes and fits a SparseJumpModel with n_components=2, max_feats=3.0, and jump_penalty=50.0. The fit method takes processed training data and returns a fitted model. The output shows the model's parameters. ```python max_feats=3. jump_penalty=50. # init sjm instance sjm = SparseJumpModel(n_components=2, max_feats=max_feats, jump_penalty=jump_penalty, ) # fit sjm.fit(X_train_processed, ret_ser=data.ret_ser, sort_by="cumret") ``` ```text Result: SparseJumpModel(jump_penalty=50.0, max_feats=3.0) ``` -------------------------------- ### Import JumpModel and SparseJumpModel classes Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/README.md Import the two core classes from the jumpmodels package. The comments indicate that JumpModel is for JM and CJM, while SparseJumpModel is for sparse JM. ```python from jumpmodels.jump import JumpModel # JM & CJM class from jumpmodels.sparse_jump import SparseJumpModel # Sparse JM class ``` -------------------------------- ### Fit JumpModel with sort_by='cumret' Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb Fits the JumpModel to the training data, using the return series to sort states by cumulative returns. The state with higher cumulative returns is labeled 0 (bull market) and the lower is labeled 1 (bear market). ```python # call .fit() jm.fit(X_train_processed, data.ret_ser, sort_by="cumret") ``` ```text Result: JumpModel(jump_penalty=50.0) ``` -------------------------------- ### Instantiate CJM with cont=True Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb Creates a Continuous Jump Model using the same JumpModel class with cont=True. The jump penalty is set to 600.0, typically 10 times larger than the discrete model's lambda, to achieve similar fittings. The specialized parameters mode_loss and grid_size are left at their defaults. ```python jump_penalty=600. cjm = JumpModel(n_components=2, jump_penalty=jump_penalty, cont=True) ``` -------------------------------- ### Online probabilistic inference with predict_proba_online Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb Performs online probabilistic regime inference on test data using predict_proba_online(). The resulting probabilities show smoother transitions, e.g., bear market confidence in late 2024 doesn't fully reach 100%, potentially reducing mislabeling. The plot is saved to a PDF. ```python # online inference proba_test_online = cjm.predict_proba_online(X_test_processed) # plot ax, ax2 = plot_regimes_and_cumret(proba_test_online, data.ret_ser, start_date=test_start, end_date=test_end, ) ax2.set(ylabel="Regime Probability") ax.set(title=f"Out-of-Sample Online Inferred Regimes by the CJM ($\\lambda$={jump_penalty})") savefig_plt(f"{get_curr_dir()}/plots/CJM_lambd-{jump_penalty}_test_online.pdf") ``` ```text Result:
``` -------------------------------- ### Predict Regimes Using All Test Data Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb Uses the .predict() method to decode states using all test data at once, then plots the resulting regimes and cumulative returns. This approach is less realistic for trading but may be useful in other fields. ```python # make inference using all test data labels_test = jm.predict(X_test_processed) # plot ax, ax2 = plot_regimes_and_cumret(labels_test, data.ret_ser, n_c=2, start_date=test_start, end_date=test_end, ) _ = ax.set(title=f"Out-of-Sample Predicted Regimes by the JM Using All Test Data ($\\lambda$={jump_penalty})") ``` -------------------------------- ### Fit CJM and plot in-sample regimes Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb Fits the CJM on processed training data and plots the resulting regime probabilities. The proba_ attribute stores smooth probability transitions (0-100%) for each period, unlike the discrete model's abrupt assignments. The plot is saved to a PDF. ```python cjm.fit(X_train_processed, data.ret_ser, sort_by="cumret") # plot ax, ax2 = plot_regimes_and_cumret(cjm.proba_, data.ret_ser, n_c=2, start_date=train_start, end_date=train_end, ) ax2.set(ylabel="Regime Probability") ax.set(title=f"In-Sample Fitted Regimes by the CJM ($\\lambda$={jump_penalty})") savefig_plt(f"{get_curr_dir()}/plots/CJM_lambd-{jump_penalty}_train.pdf") ``` ```text Result:
``` -------------------------------- ### Preprocessing with DataClipperStd and StandardScalerPD Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb Clips features to three standard deviations and standardizes them. Both transformers are fitted on training data and then applied to test data. Supports pandas DataFrames and numpy arrays. ```python # Preprocessing from jumpmodels.preprocess import StandardScalerPD, DataClipperStd clipper = DataClipperStd(mul=3.) scalar = StandardScalerPD() # fit on training data X_train_processed = scalar.fit_transform(clipper.fit_transform(X_train)) # transform the test data X_test_processed = scalar.transform(clipper.transform(X_test)) ``` -------------------------------- ### Plot and Save Online Inferred Regimes Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb Plots the online-inferred regimes and cumulative returns, then saves the figure to a PDF file. The plot title includes the jump penalty value. ```python # plot and save ax, ax2 = plot_regimes_and_cumret(labels_test_online, data.ret_ser, n_c=2, start_date=test_start, end_date=test_end, ) ax.set(title=f"Out-of-Sample Online Inferred Regimes by the JM ($\\lambda$={jump_penalty})") savefig_plt(f"{get_curr_dir()}/plots/JM_lambd-{jump_penalty}_test_online.pdf") ``` -------------------------------- ### Initialize JumpModel with jump penalty Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb Sets the jump penalty to 50.0 and initializes a JumpModel with 2 components, cont=False for hard clustering. The jump penalty is a crucial hyperparameter that requires tuning. ```python # set the jump penalty jump_penalty=50. # initlalize the JM instance jm = JumpModel(n_components=2, jump_penalty=jump_penalty, cont=False, ) ``` -------------------------------- ### Refit Jump Model and Perform Online Inference Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb Refits the Jump Model with a jump penalty of 50.0 and performs online inference on the test data using predict_online(). The online inference uses only data available up to each period, avoiding future data. ```python # refit jump_penalty=50. jm.set_params(jump_penalty=jump_penalty).fit(X_train_processed, data.ret_ser, sort_by="cumret") # make online inference labels_test_online = jm.predict_online(X_test_processed) ``` -------------------------------- ### Plot in-sample SJM regimes Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb Plots the in-sample fitted regimes using plot_regimes_and_cumret and saves the figure to a PDF. The title includes the jump penalty and max_feats values. The output shows a figure with 2 axes. ```python # plot ax, ax2 = plot_regimes_and_cumret(sjm.labels_, data.ret_ser, n_c=2, start_date=train_start, end_date=train_end, ) ax.set(title=f"In-Sample Fitted Regimes by the SJM ($\\lambda$={jump_penalty}, $\\kappa^2$={max_feats})") savefig_plt(f"{get_curr_dir()}/plots/SJM_lambd-{jump_penalty}_max-feats-{max_feats}_train.pdf") ``` ```text Result:
``` -------------------------------- ### Plot in-sample fitted regimes Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb Uses the jumpmodels.plot module to plot the in-sample fitted regimes and cumulative returns. The labels_ attribute contains the regime assignment for each period. The plot is saved as a PDF. ```python from jumpmodels.plot import plot_regimes_and_cumret, savefig_plt ax, ax2 = plot_regimes_and_cumret(jm.labels_, data.ret_ser, n_c=2, start_date=train_start, end_date=train_end, ) ax.set(title=f"In-Sample Fitted Regimes by the JM ($\\lambda$={jump_penalty})") savefig_plt(f"{get_curr_dir()}/plots/JM_lambd-{jump_penalty}_train.pdf") ``` ```text Result:
``` -------------------------------- ### Reset jump penalty to zero with set_params() Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb Resets the jump penalty to zero using set_params(), effectively reducing the model to a baseline k-means clustering algorithm. This demonstrates the utility of set_params() for refitting with different penalties. ```python # reset jump_penalty to zero jump_penalty=0. jm.set_params(jump_penalty=jump_penalty) print("The jump penalty of the JM instance has been reset to: jm.jump_penalty =", jm.jump_penalty) ``` ```text Output: The jump penalty of the JM instance has been reset to: jm.jump_penalty = 0.0 ``` -------------------------------- ### BibTeX citation for Aydinhan et al. (2024) CJM paper Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/README.md BibTeX entry for the Continuous Statistical Jump Models (CJM) paper by Aydinhan et al. (2024), published in Annals of Operations Research. ```bibtex @article{Aydinhan2024CJM, title = {Identifying patterns in financial markets: extending the statistical jump model for regime identification}, author = {Afşar Onat Aydınhan and Petter N. Kolm and John M. Mulvey and Yizhan Shu}, journal = {Annals of Operations Research}, year = {2024}, note = {To appear}, doi = {https://doi.org/10.1007/s10479-024-06035-z}, } ``` -------------------------------- ### Online inference with SJM Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb Performs online inference on test data using sjm.predict_online() and plots the out-of-sample online inferred regimes. The plot is saved to a PDF. The output shows a figure with 2 axes. ```python # online inference labels_test_online_sjm = sjm.predict_online(X_test_processed) # plot ax, ax2 = plot_regimes_and_cumret(labels_test_online_sjm, data.ret_ser, start_date=test_start, end_date=test_end, ) ax.set(title=f"Out-of-Sample Online Inferred Regimes by the SJM ($\\lambda$={jump_penalty}, $\\kappa^2$={max_feats})") savefig_plt(f"{get_curr_dir()}/plots/SJM_lambd-{jump_penalty}_max-feats-{max_feats}_test_online.pdf") ``` ```text Result:
``` -------------------------------- ### Load data and features with DataLoader Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb Loads daily returns and engineered features for the NDX index from 2007 to September 2024 using the DataLoader class from feature.py. The loaded data is stored in data.ret_ser (daily returns) and data.X (features). The output shows the first and last rows of each. ```python from feature import DataLoader data = DataLoader(ticker="NDX", ver="v0").load(start_date="2007-1-1", end_date="2024-09-30") print("Daily returns stored in `data.ret_ser`:", "-"*50, sep="\n") print(data.ret_ser, "-"*50, sep="\n") print("Features stored in `data.X`:", "-"*50, sep="\n") print(data.X) ``` ```text Output: Daily returns stored in `data.ret_ser`: -------------------------------------------------- date 2007-01-03 0.001406 2007-01-04 0.019064 2007-01-05 -0.004244 2007-01-08 0.001031 2007-01-09 0.004751 ... 2024-09-23 0.003067 2024-09-24 0.004667 2024-09-25 0.001392 2024-09-26 0.007156 2024-09-27 -0.005315 Name: NDX, Length: 4465, dtype: float64 -------------------------------------------------- Features stored in `data.X`: -------------------------------------------------- ret_5 DD-log_5 sortino_5 ret_20 DD-log_20 sortino_20 ret_60 DD-log_60 sortino_60 date 2007-01-03 -0.000860 -5.398836 -0.190234 0.000296 -5.220386 0.054761 0.000600 -5.099043 0.098343 2007-01-04 0.001719 -5.468150 0.407422 0.000935 -5.237714 0.176058 0.000812 -5.104819 0.133865 2007-01-05 0.000947 -5.467368 0.224275 0.000759 -5.243912 0.143733 0.000754 -5.107760 0.124659 2007-01-08 0.000958 -5.536682 0.243121 0.000768 -5.261241 0.148030 0.000757 -5.113536 0.125909 2007-01-09 0.001449 -5.605997 0.394142 0.000904 -5.278569 0.177218 0.000803 -5.119312 0.134308 ... ... ... ... ... ... ... ... ... ... 2024-09-23 0.003214 -4.944754 0.451332 0.001207 -4.621159 0.122670 0.000993 -4.717871 0.111166 2024-09-24 0.003402 -5.014068 0.512031 0.001325 -4.638488 0.136997 0.001035 -4.723648 0.116560 2024-09-25 0.003142 -5.083383 0.506817 0.001327 -4.655816 0.139633 0.001039 -4.729424 0.117700 2024-09-26 0.003661 -5.152698 0.633045 0.001526 -4.673145 0.163326 0.001110 -4.735200 0.126383 2024-09-27 0.002499 -5.162861 0.436548 0.001293 -4.684799 0.140005 0.001036 -4.738852 0.118410 [4465 rows x 9 columns] ``` -------------------------------- ### Print scaled cluster centroids Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb Prints the scaled cluster centroids stored in the centers_ attribute, indexed by Bull and Bear states. The values are scaled, making direct interpretation hard, but show clear separation between regimes. ```python print("Scaled Cluster Centroids:", pd.DataFrame(jm.centers_, index=["Bull", "Bear"], columns=X_train.columns), sep="\n" + "-"*50 + "\n") ``` ```text Output: Scaled Cluster Centroids: -------------------------------------------------- ret_5 DD-log_5 sortino_5 ret_20 DD-log_20 sortino_20 ret_60 DD-log_60 sortino_60 Bull 0.154342 -0.296901 0.157169 0.296358 -0.318770 0.274053 0.336652 -0.245783 0.336974 Bear -0.643127 1.237157 -0.654906 -1.234891 1.328281 -1.141952 -1.402793 1.024153 -1.404137 ``` -------------------------------- ### BibTeX citation for Nystrup et al. (2021) Sparse JM paper Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/README.md BibTeX entry for the Sparse Jump Models paper by Nystrup, Kolm, and Lindström (2021), published in Expert Systems with Applications. ```bibtex @article{nystrup2021SJM, title = {Feature selection in jump models}, author = {Peter Nystrup and Petter N. Kolm and Erik Lindstr{"o}m}, journal = {Expert Systems with Applications}, volume = {184}, pages = {115558}, year = {2021}, doi = {https://doi.org/10.1016/j.eswa.2021.115558}, } ``` -------------------------------- ### BibTeX citation for Nystrup et al. (2020a) original JM paper Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/README.md BibTeX entry for the original Statistical Jump Models paper by Nystrup, Lindström, and Madsen (2020a), published in Expert Systems with Applications. ```bibtex @article{Nystrup2020JM, title = {Learning hidden {Markov} models with persistent states by penalizing jumps}, author = {Peter Nystrup and Erik Lindstr{"o}m and Henrik Madsen}, journal = {Expert Systems with Applications}, year = {2020}, pages = {113307}, volume = {150}, doi = {https://doi.org/10.1016/j.eswa.2020.113307}, } ``` -------------------------------- ### Print SJM feature weights Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb Prints the feature weights stored in the sjm.feat_weights attribute. The output shows that features with longer halflives receive higher weights, while noisier features are assigned zero weight. ```python print("SJM Feature Weights:", "-"*50, sjm.feat_weights, sep="\n") ``` ```text Output: SJM Feature Weights: -------------------------------------------------- ret_5 0.000000 DD-log_5 0.251484 sortino_5 0.000000 ret_20 0.000000 DD-log_20 0.604436 sortino_20 0.025404 ret_60 0.753773 DD-log_60 0.000000 sortino_60 0.857114 dtype: float64 ``` -------------------------------- ### BibTeX citation for Nystrup et al. (2020b) online inference paper Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/README.md BibTeX entry for the Online Inference for JMs paper by Nystrup, Kolm, and Lindström (2020b), published in The Journal of Financial Data Science. ```bibtex @article{Nystrup2020onlineJM, title = {Greedy Online Classification of Persistent Market States Using Realized Intraday Volatility Features}, author = {Peter Nystrup and Petter N. Kolm and Erik Lindstr{"o}m}, journal = {The Journal of Financial Data Science} year = {2020}, volume = {2}, number = {3}, pages = {25--39}, doi = {https://doi.org/10.3905/jfds.2020.2.3.025}, } ``` -------------------------------- ### Result of Online Inference Plot Source: https://github.com/yizhan-oliver-shu/jump-models/blob/master/examples/nasdaq/example.ipynb Output of the plotting code, indicating a figure with 2 axes was created. ```text Result:
```