Machine Learning for Algorithmic Trading

https://github.com/stefan-jansen/machine-learning-for-trading
Admin
This book and accompanying repository demonstrate how to apply machine learning techniques to...
Tokens:6,643,345
Snippets:31,338
Trust Score:9
License:MIT
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### Local setup with uv: install, clone, configure, and run

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/docs/running-notebooks.md

Commands to install uv, clone the repository, copy the environment template, install dependencies (twelve compile from source, requiring a C/C++ compiler and Python headers), run a notebook, and start Jupyter Lab. The ML4T_DATA_PATH prefix gives data loaders an absolute path; without it, notebooks report data as missing. uv run does not read .env, so export the data path in your shell profile if needed.

```bash
# Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh
# The installer puts uv in ~/.local/bin, which this shell does not know about
# yet. Load it now, or `uv sync` below fails with "uv: command not found":
source $HOME/.local/bin/env        # sh, bash, zsh;  env.fish for fish

# Clone the repository
git clone https://github.com/stefan-jansen/machine-learning-for-trading.git
cd machine-learning-for-trading

# Copy the environment template. The defaults work as they are — nothing in it
# needs editing to start. Come back to it when a chapter asks for an API key.
cp .env.example .env

# Install all dependencies. Twelve of them compile from source, so a C/C++
# compiler and the Python headers must already be installed — see
# docs/installation.md, "What you need before either path".
uv sync

# Run a notebook
uv run python 11_ml_pipeline/01_ols_inference.py

# Or start Jupyter Lab, from the repo root, and open the URL it prints
ML4T_DATA_PATH="${ML4T_DATA_PATH:-$PWD/data}" uv run jupyter lab
```

--------------------------------

### Install uv, clone repo, sync dependencies, and start Jupyter Lab

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/docs/installation.md

Commands to set up the project locally with uv. Includes installing uv via the official installer, sourcing the environment, cloning the repository, syncing dependencies, copying the environment template, verifying the installation, and starting Jupyter Lab with the ML4T_DATA_PATH environment variable.

```bash
# Install uv — use this installer, not `pip install uv`. Most current systems either
# ship no `pip` at all or refuse the install with `externally-managed-environment`.
curl -LsSf https://astral.sh/uv/install.sh | sh
# The installer puts uv in ~/.local/bin, which your current shell does not know about
# yet. Load it now rather than opening a new terminal:
source $HOME/.local/bin/env        # sh, bash, zsh;  env.fish for fish

# Clone and enter the repository (about 0.9 GB of history)
git clone https://github.com/stefan-jansen/machine-learning-for-trading.git
cd machine-learning-for-trading

# Install all dependencies (creates .venv/, installs ~300 packages, about 11 GB).
# Twelve of them compile from source, so a C/C++ compiler must already be installed —
# see "What you need before either path" above.
uv sync

# Copy environment template (defaults work as-is; no editing needed to start)
cp .env.example .env
# API keys are optional and only needed for specific datasets later —
# see data/README.md when a chapter asks for one.

# Verify — the one PASS/FAIL gate, same script on both paths
uv run python scripts/verify_installation.py

# Start Jupyter Lab, from the repo root, and open the URL it prints
ML4T_DATA_PATH="${ML4T_DATA_PATH:-$PWD/data}" uv run jupyter lab
```

--------------------------------

### Load configuration and print evaluation settings

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/case_studies/us_equities_panel/05_evaluation.ipynb

Loads setup from config/setup.yaml, resolves the primary label and horizon, sets holdout start, walk-forward splits, and minimum cross-section. Prints a summary of these settings.

```python
CASE_STUDY_ID = "us_equities_panel"
CASE_DIR = get_case_study_dir(CASE_STUDY_ID)
EVAL_DIR = CASE_DIR / "evaluation"
EVAL_DIR.mkdir(exist_ok=True)

DATE_COL = "timestamp"
ENTITY_COL = "symbol"
JOIN_COLS = [DATE_COL, ENTITY_COL]

SETUP = load_setup_config(CASE_STUDY_ID)
PRIMARY_LABEL = SETUP["labels"]["primary"]
LABEL_BUFFER = resolve_label_buffer(CASE_STUDY_ID, PRIMARY_LABEL, SETUP)
assert LABEL_BUFFER, f"no label buffer configured for {PRIMARY_LABEL}"
LABEL_HORIZON = int(resolve_label_horizon(CASE_STUDY_ID, PRIMARY_LABEL, SETUP).rstrip("Dd"))
HOLDOUT_START = date.fromisoformat(str(SETUP["evaluation"]["holdout_start"]))
N_SPLITS = SETUP["evaluation"]["n_splits"]
TOP_K_PER_SIDE = max(SETUP["backtest"]["sweep"]["top_k_grid"][PRIMARY_LABEL])
MIN_CROSS_SECTION = 2 * TOP_K_PER_SIDE

set_global_seeds(SEED)

print(f"Label            {PRIMARY_LABEL}, the return over the next {LABEL_HORIZON} session")
print(f"Holdout          opens {HOLDOUT_START}; everything below stops before it")
print(f"Walk-forward     {N_SPLITS} folds of {SETUP['evaluation']['train_size']} training")
print(f"                 scored over {SETUP['evaluation']['val_size']} each")
print(f"Cross-section    at least {MIN_CROSS_SECTION} names quoted, which is both sides of a")
print(f"                 book holding {TOP_K_PER_SIDE} long and {TOP_K_PER_SIDE} short")
```

```text
Output:
Label            fwd_ret_1d, the return over the next 1 session
Holdout          opens 2016-01-01; everything below stops before it
Walk-forward     16 folds of 10Y training
                 scored over 1Y each
Cross-section    at least 100 names quoted, which is both sides of a
                 book holding 50 long and 50 short
```

--------------------------------

### Clone repository and set up environment

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/docs/installation.md

Clones the repository, copies the environment template, pulls the pre-built Docker image, verifies the installation, and starts Jupyter Lab. The defaults in .env.example work as-is.

```bash
# 1. Clone the repository
git clone https://github.com/stefan-jansen/machine-learning-for-trading.git
cd machine-learning-for-trading

# 2. Copy the environment template. The defaults work as they are — nothing in
#    it needs editing to start.
cp .env.example .env

# 3. Pull the pre-built image from Docker Hub
docker compose pull ml4t

# 4. Confirm the install: one PASS/FAIL line per component, exit 0 if ready
docker compose run --rm ml4t python scripts/verify_installation.py

# 5. Start Jupyter Lab. It keeps running in this terminal; leave it running.
docker compose up ml4t
```

--------------------------------

### Create a writable experiment and run a stage against it

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/docs/running-notebooks.md

Produces the modeling dataset, snapshots the installed artifacts and config into a writable experiment, then runs a stage (e.g., 07_gbm.py) against it. The setup copies generated prerequisites and the config tree, changes the release marker to a baseline marker, and makes only the copy writable. ML4T_OUTPUT_DIR routes config reads and new registry rows into the experiment directory, leaving the downloaded release untouched.

```bash
# 1. Produce the modeling dataset the model notebooks need (writes features/ and
#    labels/ into the case study directory; skip any that already exist).
uv run python case_studies/etfs/02_labels.py
uv run python case_studies/etfs/03_financial_features.py
uv run python case_studies/etfs/04_model_based_features.py

# 2. Snapshot the installed artifacts + config into a writable experiment.
uv run python scripts/create_experiment.py \
  --cs etfs \
  --output /tmp/ml4t-etf-experiment

# 3. Edit config in the experiment (see below), then run the stage against it.
ML4T_OUTPUT_DIR=/tmp/ml4t-etf-experiment \
  uv run python case_studies/etfs/07_gbm.py
```

--------------------------------

### Install uv, clone repo, and build environment

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/docs/installation.md

Installs uv, clones the repository, copies the .env.example file, and runs uv sync. The sync installs about 300 packages (~11 GB) and twelve compile from source.

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env    # the installer's own line; puts uv on PATH here and now
git clone https://github.com/stefan-jansen/machine-learning-for-trading.git
cd machine-learning-for-trading
cp .env.example .env           # defaults work as-is; nothing in it needs editing to start
uv sync                        # ~300 packages, about 11 GB; twelve compile from source
```

--------------------------------

### Load setup.yaml from all case studies

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/06_strategy_definition/03_case_study_overview.ipynb

Iterates over case study directories, reads each config/setup.yaml, normalizes it, and returns a dictionary keyed by case study name. Skips directories starting with '_' or missing setup.yaml.

```python
def load_setup_results() -> dict[str, dict]:
    """Load config/setup.yaml from all case studies."""
    results = {}
    for case_dir in sorted(CASE_STUDIES_DIR.iterdir()):
        if case_dir.name.startswith("_") or not case_dir.is_dir():
            continue
        setup_path = case_dir / "config" / "setup.yaml"
        if not setup_path.exists():
            continue
        cfg = yaml.safe_load(setup_path.read_text())
        results[case_dir.name] = _normalize_setup_yaml(case_dir.name, cfg)
    return results
```

--------------------------------

### Set up case study paths, seeds, and explorer

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/case_studies/us_firm_characteristics/17_strategy_analysis.ipynb

Sets up primary label, periods per year, case study directory, output directory, global seeds, loads setup.yaml, and creates a BacktestExplorer. The output directory is created if it does not exist.

```python
PRIMARY_LABEL = "fwd_ret_1m"  # setup.yaml primary; benchmark series keyed here
PERIODS_PER_YEAR = 12  # monthly cadence
CASE_DIR = get_case_study_dir(CASE_STUDY)
OUTPUT_DIR = get_output_dir(20, CASE_STUDY)
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
set_global_seeds(SEED)

with open(CASE_DIR / "config" / "setup.yaml") as f:
    setup = yaml.safe_load(f)

explorer = BacktestExplorer(CASE_STUDY)
print(explorer)
```

--------------------------------

### Load holdout start date from YAML config

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/26_mlops_governance/05_feast_feature_store.ipynb

Reads the setup YAML file, extracts the holdout start and end dates, and prints the start date. The output shows 'Holdout starts on 2016-01-01'.

```python
setup = yaml.safe_load(SETUP_PATH.read_text())
holdout_start = pd.Timestamp(setup["evaluation"]["holdout_start"])
holdout_end = pd.Timestamp(setup["evaluation"]["holdout_end"])
print(f"Holdout starts on {holdout_start.date()}")
```

--------------------------------

### Install Chrome in running container

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/docs/installation.md

Start the container and install Chrome using plotly_get_chrome -y, since docker compose run --rm discards changes.

```bash
docker compose up -d ml4t
docker compose exec ml4t plotly_get_chrome -y
```

--------------------------------

### Setup: Initialize clients and load pinned trace

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/24_autonomous_agents/06_multi_agent_research.ipynb

In live mode, creates LLM and search clients and loads the chapter clear question. In replay mode, loads a pinned trace and rebuilds the question. Prints the mode, provider, question, market price, and agent count.

```python
if RUN_LIVE:
    llm = create_llm_client(LLM_PROVIDER)
    search = create_search_client()
    question = get_chapter_clear_question()
    provider_name = llm.model_name
    n_agents = N_AGENTS
else:
    # Replay: load the pinned trace and rebuild the question it forecast.
    pinned_run = RunTrace.load(TRACES_DIR / PINNED_TRACE)
    question = pinned_run.question_obj()
    provider_name = pinned_run.provider
    n_agents = len(pinned_run.agents)

print(f"Mode:         {'LIVE' if RUN_LIVE else 'REPLAY (pinned 2026-06-09 trace)'}")
print(f"Provider:     {provider_name}")
print(f"Question:     {question.question}")
print(f"Market p_yes: {question.current_market_price}")
print(f"Agents:       {n_agents}")
```

```text
Output:
Mode:         REPLAY (pinned 2026-06-09 trace)
Provider:     claude-sonnet-4-20250514
Question:     Will the US enter a recession by the end of 2026?
Market p_yes: 0.175
Agents:       3
```

--------------------------------

### PyKX/kdb+ unavailable output

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/02_financial_data_universe/21_storage_benchmark_database.ipynb

Example output when PyKX is not installed, showing the skip message.

```text
Output:
○ PyKX/kdb+: unavailable, skipping (ModuleNotFoundError: No module named 'pykx')
```

--------------------------------

### Load and print configuration from setup.yaml

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/case_studies/nasdaq100_microstructure/01_feasibility_analysis.ipynb

Loads the case study configuration from a YAML file and prints a summary of the development period, universe, costs, and forecast horizons. The configuration includes holdout dates, primary label, symbols, cost assumptions, and a breadth floor calculated from the top-k grid.

```python
CASE_DIR = get_case_study_dir(CASE_STUDY_ID)
SETUP = yaml.safe_load((CASE_DIR / "config" / "setup.yaml").read_text())

HOLDOUT_START = str(SETUP["evaluation"]["holdout_start"])
HOLDOUT_END = str(SETUP["evaluation"]["holdout_end"])
PRIMARY_LABEL = SETUP["labels"]["primary"]
DECLARED_SYMBOLS = set(SETUP["universe"]["symbols"])
BREADTH_FLOOR = 2 * max(max(grid) for grid in SETUP["backtest"]["sweep"]["top_k_grid"].values())
PER_SHARE_USD = float(SETUP["costs"]["per_share"])
FRICTION_FLOOR_BPS = float(SETUP["costs"]["friction_floor_bps"])
COST_FEASIBLE = SETUP["universe"]["cost_feasible"]["validation"]
CADENCE = f"{SETUP['decision']['bar_frequency'].split('_')[0]}m"
LABEL_BUFFER = SETUP["labels"]["buffer"]
# The outcome horizons, not the purge buffers. They differ by one bar here - the label
# reads a quote one bar past the horizon it names - and reading the buffers would put
# 6, 16 and 61 minutes on this figure's axis under labels called 5, 15 and 60.
HORIZONS = sorted({int(h.rstrip("min")) for h in SETUP["labels"]["horizons"].values()})
WINDOW = {
    "start_date": START_DATE,
    "end_date": HOLDOUT_START,
    "max_symbols": MAX_SYMBOLS,
    "symbols": sorted(DECLARED_SYMBOLS),
}

print(f"Sample: {START_DATE} to {HOLDOUT_END}")
print(f"  Development period, used everywhere below:  {START_DATE} to {HOLDOUT_START}")
print(f"  Holdout, not read by this notebook:         {HOLDOUT_START} to {HOLDOUT_END}")
print(f"Universe: {len(DECLARED_SYMBOLS)} index constituents declared")
print(
    f"  {len(COST_FEASIBLE)} of them are frozen as cheap enough to trade in validation, and up to "
    f"{BREADTH_FLOOR // 2} are held per side, so at least {BREADTH_FLOOR} must be quoting at a "
    f"decision bar to fill both legs"
)
print(f"Decision bars: every {CADENCE}, with the trade placed in the following bar")
print(
    f"Assumed cost: ${PER_SHARE_USD} per share in commission, plus half the quoted spread on each "
    f"side, against a declared friction floor of {FRICTION_FLOOR_BPS:.0f} bps"
)
print(
    f"Forecast horizons: {', '.join(f'{h} minutes' for h in HORIZONS)} ahead; "
    f"{PRIMARY_LABEL.rsplit('_', 1)[1]} is the primary label and its {LABEL_BUFFER} buffer sets "
    f"the gap that separates training from validation"
)
```

```text
Output:
Sample: 2020-01-01 to 2021-12-31
  Development period, used everywhere below:  2020-01-01 to 2021-07-01
  Holdout, not read by this notebook:         2021-07-01 to 2021-12-31
Universe: 115 index constituents declared
  50 of them are frozen as cheap enough to trade in validation, and up to 20 are held per side, so at least 40 must be quoting at a decision bar to fill both legs
Decision bars: every 15m, with the trade placed in the following bar
Assumed cost: $0.0035 per share in commission, plus half the quoted spread on each side, against a declared friction floor of 5 bps
Forecast horizons: 5 minutes, 15 minutes, 60 minutes ahead; 15m is the primary label and its 16min buffer sets the gap that separates training from validation
```

--------------------------------

### Install optional LLM provider dependencies

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/24_autonomous_agents/01_react_reasoning.ipynb

Install the optional dependencies needed for live LLM and web-search providers. The chapter runs in offline replay mode without them. Ollama is installed separately from https://ollama.com, then started with `ollama serve` and a model pulled with `ollama pull qwen3:8b`.

```bash
uv pip install anthropic openai httpx        # cloud providers
# Ollama (free, local) - install from https://ollama.com, then:
#   ollama serve && ollama pull qwen3:8b
```

--------------------------------

### Set up output directory and print configuration

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/15_causal_estimation/05_momentum_causal_trading.ipynb

Sets the global seed, creates the output directory, and prints the configuration summary.

```python
set_global_seeds(SEED)

CACHE_TAG = RUN_TAG
OUTPUT_DIR = get_output_dir(15, "momentum_causal_trading")
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
DML_CACHE_PATH = OUTPUT_DIR / f"dml_artifacts_{CASE_STUDY_ID}_{PRIMARY_LABEL}_{CACHE_TAG}.json"

print("Momentum Causal Trading Configuration:")
print(f"  Forward: {FORWARD_DAYS} days")
print(f"  Transaction cost: {TRANSACTION_COST_BPS} bps")
print(f"  Run tag: {RUN_TAG}")
print(f"  Retrain DML artifacts: {RETRAIN}")
```

```text
Output:
Momentum Causal Trading Configuration:
  Forward: 21 days
  Transaction cost: 10 bps
  Run tag: full
  Retrain DML artifacts: True
```

--------------------------------

### Setup and Configuration for Strategy Analysis

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/case_studies/sp500_options/18_strategy_analysis.ipynb

Defines case study constants, loads setup configuration, and initializes the BacktestExplorer. Prints the explorer to show the number of runs.

```python
CASE_STUDY = "sp500_options"
# The immutable grid `12_backtest` publishes; the DSR below deflates for its members.
BASELINE_POPULATION = "sp500-options-baseline-validation-v1"
# The nominees `13_portfolio_management` publishes; the selected configuration has to be one of
# them.
STRATEGY_CANDIDATES = "sp500-options-strategy-candidates-v1"
PRIMARY_LABEL = "ret_to_expiry"  # registered HTM strategy label (Appendix A)
PERIODS_PER_YEAR = 252
CASE_DIR = get_case_study_dir(CASE_STUDY)
OUTPUT_DIR = get_output_dir(20, CASE_STUDY)
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)

with open(CASE_DIR / "config" / "setup.yaml") as f:
    setup = yaml.safe_load(f)

explorer = BacktestExplorer(CASE_STUDY)
_selection_study = Study.open(CASE_STUDY)
print(explorer)
```

```text
BacktestExplorer('sp500_options', 879 runs: allocation=60, cost_sensitivity=32, holdout=1, signal=786)
```

--------------------------------

### Setup DataManager, HiveStorage, and Universe with logging configuration

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/02_financial_data_universe/18_data_management.ipynb

Configures structlog to filter debug logs, imports core ml4t-data classes, and initializes a clean storage directory for the notebook's examples. This setup is required before using DataManager, Universe, or HiveStorage.

```python
"""Data Management — DataManager, Universe, HiveStorage, and incremental updates."""

import logging
import shutil
from datetime import datetime
from pathlib import Path

import plotly.graph_objects as go
import polars as pl
import structlog

# ml4t-data emits structured debug logs on every fetch/store; route them
# through stdlib logging at WARNING so the notebook output stays focused on
# the demonstration.
structlog.configure(
    wrapper_class=structlog.make_filtering_bound_logger(logging.WARNING),
)

# ml4t-data core imports
from ml4t.data import DataManager
from ml4t.data.storage import HiveStorage
from ml4t.data.storage.backend import StorageConfig
from ml4t.data.universe import Universe

from utils.downloading import update_through_last_complete_bar
from utils.paths import REPO_ROOT, get_output_dir
from utils.style import COLORS, show_plotly_with_alt


def _rel(path):
    """Repo-relative display path (keeps absolute machine paths out of outputs)."""
    try:
        return path.relative_to(REPO_ROOT)
    except ValueError:
        return path


# Working directory for this notebook's storage examples. Wipe any artifacts
# from a previous run so the demo is fully reproducible.
STORAGE_DIR = get_output_dir(2, "data_management")
if STORAGE_DIR.exists():
    shutil.rmtree(STORAGE_DIR)
STORAGE_DIR.mkdir(parents=True, exist_ok=True)

print(f"Storage directory: {_rel(STORAGE_DIR)}")
```

--------------------------------

### Setup Techniques Summary Table Output

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/06_strategy_definition/03_case_study_overview.ipynb

Displays the resulting DataFrame with columns for Case Study, Setup Type, and Position Mapping, showing the mapping for each case study.

```text
Result:
shape: (9, 3)
┌───────────────────────────┬──────────────────────────────┬─────────────────────────────────┐
│ Case Study                ┆ Setup Type                   ┆ Position Mapping                │
│ ---                       ┆ ---                          ┆ ---                             │
│ str                       ┆ str                          ┆ str                             │
╞═══════════════════════════╪══════════════════════════════╪═════════════════════════════════╡
│ CME Futures               ┆ long_short_carry_rank        ┆ rank_by_carry_or_momentum       │
│ Crypto Perps Funding      ┆ long_short_funding_aligned   ┆ threshold_or_rank_based         │
│ ETFs                      ┆ long_only_rank_and_rebalance ┆ rank_selection_top_n            │
│ FX Pairs                  ┆ long_short_rank_rebalance    ┆ rank_by_momentum_or_carry       │
│ NASDAQ-100 Microstructure ┆ intraday_rank_and_trade      ┆ rank_or_threshold               │
│ S&P 500 Equity+Options    ┆ long_only_rank_and_rebalance ┆ rank_by_iv_signal               │
│ S&P 500 Options           ┆ systematic_straddle_sell     ┆ sell_atm_straddle_weekly        │
│ US Equities Panel         ┆ long_short_decile_rebalance  ┆ decile_sort_long_top_short_bot… │
│ US Firm Characteristics   ┆ long_short_top_k_rebalance   ┆ rank_top_k_long_bottom_k_short  │
└───────────────────────────┴──────────────────────────────┴─────────────────────────────────┘
```

--------------------------------

### Extract baseline and ensemble metric blocks from comparison result

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/24_autonomous_agents/11_research_operator.ipynb

Calls _matched_comparison on the result to get the allocation header and per-model metric blocks, then selects the baseline (key starting with 'LSTM') and ensemble (key starting with 'ENSEMBLE') blocks and prints the header under which the comparison was scored.

```python
allocation_header, etf_blocks = _matched_comparison(result)
baseline_key = next(k for k in etf_blocks if k.startswith("LSTM"))
ensemble_key = next(k for k in etf_blocks if k.startswith("ENSEMBLE"))
baseline, ensemble = etf_blocks[baseline_key], etf_blocks[ensemble_key]
print(f"Comparison scored under: {allocation_header}")
```

```text
Output:
Comparison scored under: === score_weighted_top_k, top_k=20 (matching LSTM 0.92 config) ===

```

--------------------------------

### Load case study paths, setup config, and feature windows

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/case_studies/fx_pairs/04_model_based_features.ipynb

Sets up directories, loads the setup configuration, and derives the Kalman trend window and USD volatility window from the configuration. Comments explain the rationale for choosing these windows.

```python
CASE_DIR = get_case_study_dir(CASE_STUDY_ID)
LABELS_DIR = CASE_DIR / "labels"
FEATURES_DIR = CASE_DIR / "features"

# A Spearman IC over fewer pairs than this is a rank correlation over a handful of
# points; dates below the floor are dropped from the series rather than averaged in.
MIN_PAIRS_PER_DATE = 8

SETUP = load_setup_config(CASE_STUDY_ID)
SESSION_CALENDAR = SETUP["decision"]["session_calendar"]

# Two windows this notebook needs are already decided in `setup.yaml`, and both are read
# from it rather than typed, so a configuration change reaches the models rather than
# leaving them measuring against a window the feature stage no longer uses.
#
# `kalman_trend` is the fitted level less a moving average of the price, and it is the
# middle of the three moving-average windows the feature configuration declares: the
# shortest sits inside the filter's own responsiveness, so the difference would be mostly
# filter noise, and the longest is slower than a fold's validation year. Taking the same
# window `03_financial_features` gives `price_to_ma_63d` also means the two columns
# measure price against one reference rather than two.
KALMAN_TREND_WINDOW = int(sorted(SETUP["features"]["windows"]["moving_average"])[1])
# The dollar-regime model is given the shortest close-to-close volatility window the
# configuration declares - about a trading month, long enough for a stable estimate and
# short enough to move when the market does.
USD_VOL_WINDOW = int(min(SETUP["features"]["windows"]["close_to_close_volatility"]))
USD_VOL_COL = f"usd_vol_{USD_VOL_WINDOW}d"
```

--------------------------------

### Output of leaver/survivor classification

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/02_financial_data_universe/15_survivorship_bias_detection.ipynb

Example output showing the counts and percentages for a window with 3,095 symbols at start, 752 leavers (24.3%), and 2,343 survivors.

```text
Output:
Universe at start:   3,095
Left during window:  752 (24.3%)
Survived the window: 2,343
```

--------------------------------

### Open study and initialize explorer

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/case_studies/etfs/20_strategy_analysis.ipynb

Sets up the case study, output directory, loads setup.yaml, and creates a BacktestExplorer. The study is opened before any path or registry read, which activates a workspace and rewrites ML4T_OUTPUT_DIR process-wide under the preview tier.

```python
CASE_STUDY = "etfs"
PRIMARY_LABEL = "fwd_ret_21d"
PERIODS_PER_YEAR = 252  # NYSE calendar, daily bars
study = open_study(CASE_STUDY, execution_tier=EXECUTION_TIER, workspace=WORKSPACE or None)
CASE_DIR = get_case_study_dir(CASE_STUDY)
OUTPUT_DIR = get_output_dir(20, CASE_STUDY)
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)

with open(CASE_DIR / "config" / "setup.yaml") as f:
    setup = yaml.safe_load(f)

explorer = BacktestExplorer(CASE_STUDY)
print(explorer)
```

--------------------------------

### Import libraries and set multiprocessing start method

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/case_studies/cme_futures/04_model_based_features.ipynb

Imports and setup for the CME futures model-based feature notebook. Pins the multiprocessing start method to 'fork' before any pool-using import to avoid Python 3.14's forkserver re-executing the script in workers. Applies the notebook warning policy.

```python
"""CME Futures: Temporal Feature Engineering."""

import multiprocessing
import os
import re
import time
import warnings
from concurrent.futures import ProcessPoolExecutor
from datetime import date

# Pin the start method to fork before any pool-using import: Python 3.14 defaults to
# forkserver, which re-executes this script in every worker process the ARIMA walk spawns.
if multiprocessing.get_start_method(allow_none=True) is None:
    multiprocessing.set_start_method("fork")

import numpy as np
import pandas as pd
import plotly.graph_objects as go
import polars as pl
from hmmlearn.hmm import GaussianHMM
from plotly.subplots import make_subplots
from statsmodels.tsa.arima.model import ARIMA
from threadpoolctl import threadpool_limits

from case_studies.utils.artifact_digest import value_digest
from case_studies.utils.artifact_quality import (
    label_universe,
    quality_report,
    render_quality_report,
)
from case_studies.utils.temporal import (
    arima_one_step_forecast,
    filtered_state_probs,
    fit_hmm_kmeans_init,
    refit_boundaries,
    sort_states_by_mean,
    walk_forward_feature,
    write_model_based,
)
from case_studies.utils.warning_policy import apply_notebook_warning_policy
from data import load_cme_futures
from utils.artifact_specs import load_setup_config, resolve_label_buffer
from utils.cv_splits import generate_cv_splits, load_evaluation_config, select_folds
from utils.paths import get_case_study_dir
from utils.reproducibility import set_global_seeds
from utils.style import COLORS, show_plotly_with_alt

apply_notebook_warning_policy()
```

--------------------------------

### Verify WSL2 installation (Windows 11)

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/docs/installation.md

Run this command in PowerShell to check that Ubuntu is listed with STATE Running or Stopped and VERSION 2. If you get 'Windows Subsystem for Linux has no installed distributions', step 3 did not complete. If VERSION reads 1, run wsl --set-version Ubuntu 2.

```powershell
wsl --list --verbose
```

--------------------------------

### Configuration: Setup paths, parameters, and schedule

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/case_studies/cme_futures/04_model_based_features.ipynb

Sets up directories, loads setup.yaml, extracts feature windows and model parameters (ARIMA and HMM), applies overrides and excluded sessions, and prints the estimation schedule. The output shows the actual values used.

```python
CASE_DIR = get_case_study_dir(CASE_STUDY_ID)
FEATURES_DIR = CASE_DIR / "features"
LABELS_DIR = CASE_DIR / "labels"
STRATEGY_ID = CASE_STUDY_ID
set_global_seeds(SEED)

SETUP = load_setup_config(CASE_STUDY_ID)
PRODUCT_GROUPS = SETUP["universe"]["product_groups"]
ALL_PRODUCTS = [p for products in PRODUCT_GROUPS.values() for p in products]
assert len(ALL_PRODUCTS) == SETUP["universe"]["n_products"], (
    f"setup.yaml declares {SETUP['universe']['n_products']} products, "
    f"product_groups lists {len(ALL_PRODUCTS)}"
)

CARRY_SMOOTHING = int(SETUP["features"]["windows"]["carry_smoothing"])
CARRY_ZSCORE_WINDOW = int(SETUP["features"]["windows"]["carry_zscore"][0])

# The estimation schedule, read rather than typed, so the comments in `setup.yaml` that
# say what each count decides stay next to the value the notebook uses.
MODEL_BASED = SETUP["model_based"]
ARIMA_BURNIN = int(MODEL_BASED["arima"]["burnin"])
ARIMA_REFIT_FREQ = int(MODEL_BASED["arima"]["refit_every"])
ARIMA_ORDER = tuple(int(v) for v in MODEL_BASED["arima"]["order"])
HMM_BURNIN = int(MODEL_BASED["hmm"]["burnin"])
HMM_REFIT_EVERY = int(MODEL_BASED["hmm"]["refit_every"])
HMM_N_STATES = int(MODEL_BASED["hmm"]["n_states"])
if REFIT_EVERY_OVERRIDE:
    ARIMA_REFIT_FREQ = HMM_REFIT_EVERY = REFIT_EVERY_OVERRIDE
    print(f"Reduced run: both refit cadences replaced with {REFIT_EVERY_OVERRIDE}")

# Two sessions carry one clearing venue's settlement file and not the other's; `setup.yaml`
# says which and why. They are dropped here so no series is differenced across a date on
# which half the universe has no settlement price.
EXCLUDED_SESSIONS = [
    date.fromisoformat(str(d)) for d in SETUP["universe"].get("excluded_sessions", [])
]

if MAX_PRODUCTS > 0:
    ARIMA_PRODUCTS = ALL_PRODUCTS[:MAX_PRODUCTS]
else:
    ARIMA_PRODUCTS = ALL_PRODUCTS

print(f"Carry is smoothed over {CARRY_SMOOTHING} sessions before anything reads it.")
print(
    f"Its z-score is taken against the previous {CARRY_ZSCORE_WINDOW} sessions of that "
    f"smoothed series."
)
print(f"Modelling {len(ARIMA_PRODUCTS)} of the {len(ALL_PRODUCTS)} products in the universe.")
print("Estimation schedule, in sessions of each model's own series:")
print(f"  ARIMA        burn-in {ARIMA_BURNIN:>4}, refit every {ARIMA_REFIT_FREQ:>3}")
print(f"  carry regime burn-in {HMM_BURNIN:>4}, refit every {HMM_REFIT_EVERY:>3}")
```

```text
Carry is smoothed over 21 sessions before anything reads it.
Its z-score is taken against the previous 63 sessions of that smoothed series.
Modelling 30 of the 30 products in the universe.
Estimation schedule, in sessions of each model's own series:
  ARIMA        burn-in  252, refit every  21
  carry regime burn-in  504, refit every  63
```

--------------------------------

### Load cost grid and per-share commission from setup.yaml

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/case_studies/etfs/17_costs.ipynb

Loads the cost grid and per-share commission from setup.yaml, failing loudly if the per_share key is missing. Prints the case study, label, and cost grids. The output shows the actual values for the etfs case study.

```python
# Configuration only. Anything that reads the registry resolves the directory at call time,
# because `Study.activate()` moves the output root for a preview run and this binding predates it.
CASE_DIR = get_case_study_dir(CASE_STUDY_ID)
bt_config = get_backtest_config(CASE_STUDY_ID)
if TOP_N_COMBOS is None:
    TOP_N_COMBOS = get_top_n_predictions(CASE_STUDY_ID, "cost_sensitivity")
if not LABEL:
    LABEL = bt_config.primary_label

COST_GRID_BPS = get_cost_grid_bps(CASE_STUDY_ID)
COST_GRID_HALF_SPREAD_USD = get_cost_grid_half_spread_usd(CASE_STUDY_ID)
PER_SHARE_COMMISSION = float(
    yaml.safe_load((CASE_DIR / "config" / "setup.yaml").read_text())["costs"]["per_share"]
)

print(f"Case study: {CASE_STUDY_ID}, label: {LABEL}")
print(f"Per-share commission: ${PER_SHARE_COMMISSION}/share")
print(f"Basis-point grid:     {COST_GRID_BPS}")
print(f"Half-spread grid (¢): {[round(v * 100, 2) for v in COST_GRID_HALF_SPREAD_USD]}")
```

```text
Output:
Case study: etfs, label: fwd_ret_21d
Per-share commission: $0.0035/share
Basis-point grid:     [0.0, 1.0, 2.0, 3.0, 5.0, 7.0, 10.0, 15.0, 20.0, 30.0, 50.0]
Half-spread grid (¢): [0.0, 0.5, 1.0, 2.5, 5.0, 10.0]
```

--------------------------------

### Define date range for Kalman filter analysis

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/09_model_based_features/04_kalman_filter.ipynb

Sets the start and end dates for the data used in the Kalman filter examples. The period is from January 1, 2018, to December 31, 2023.

```python
START_DATE = "2018-01-01"
END_DATE = "2023-12-31"
```

--------------------------------

### Setup imports and demo storage directory

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/02_financial_data_universe/19_incremental_updates.ipynb

Sets up the notebook environment: imports ml4t-data modules, configures structlog to filter warnings, and creates a fresh demo storage directory by removing any prior artifacts. Run this before the update lifecycle demos.

```python
"""Incremental Updates — Keeping market data fresh with daily delta fetches."""

import logging
import shutil
from datetime import datetime

import matplotlib.pyplot as plt
import polars as pl
import structlog

# Quiet ml4t-data's structured logger so the notebook focuses on demo output.
structlog.configure(
    wrapper_class=structlog.make_filtering_bound_logger(logging.WARNING),
)

from ml4t.data import DataManager
from ml4t.data.storage import HiveStorage
from ml4t.data.storage.backend import StorageConfig
from ml4t.data.update_manager import GapDetector
from ml4t.data.validation import OHLCVValidator

from utils.downloading import update_through_last_complete_bar
from utils.paths import display_path, get_output_dir
from utils.style import COLORS, show_with_alt

# Storage for this notebook's demos. Wipe any prior-run artifacts so the
# initial-load + update sequence is reproducible.
DEMO_DIR = get_output_dir(2, "incremental_updates")
if DEMO_DIR.exists():
    shutil.rmtree(DEMO_DIR)
DEMO_DIR.mkdir(parents=True, exist_ok=True)

print(f"Demo storage: {display_path(DEMO_DIR)}")
```

--------------------------------

### Load configuration and filter data to pre-holdout period

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/08_financial_features/06_robustness_sensitivity.ipynb

Loads the setup.yaml configuration to get the holdout start date, clamps the end date to the holdout boundary, and filters the ETF data to the analysis window.

```python
setup = yaml.safe_load((get_case_study_dir("etfs") / "config" / "setup.yaml").read_text())
HOLDOUT_START = setup["evaluation"]["holdout_start"]
end_date = min(
    datetime.strptime(END_DATE, "%Y-%m-%d"),
    datetime.strptime(HOLDOUT_START, "%Y-%m-%d"),
)

etfs_filtered = etfs.filter(
    (pl.col("timestamp") >= datetime.strptime(START_DATE, "%Y-%m-%d"))
    & (pl.col("timestamp") < end_date)
).sort(["symbol", "timestamp"])

print(f"ETF data: {len(etfs_filtered):,} rows")
print(f"Symbols: {etfs_filtered['symbol'].n_unique()}")
print(f"Date range: {etfs_filtered['timestamp'].min()} to {etfs_filtered['timestamp'].max()}")
```

--------------------------------

### Setup paths and load configuration

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/case_studies/nasdaq100_microstructure/04_model_based_features.ipynb

Sets up output directories, loads configuration from setup.yaml, and extracts key parameters including labels, universe, calendar, holdout dates, and model-based parameters. The holdout end is made exclusive by adding one day.

```python
CASE_DIR = get_case_study_dir(CASE_STUDY_ID)
FEATURES_DIR = CASE_DIR / "features"
LABELS_DIR = CASE_DIR / "labels"
OUTPUT_DIR = Path(WORKSPACE) / "features" if WORKSPACE else FEATURES_DIR
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)

SETUP = yaml.safe_load((CASE_DIR / "config" / "setup.yaml").read_text())
EVAL_CFG = load_evaluation_config(CASE_STUDY_ID)
PRIMARY_LABEL = SETUP["labels"]["primary"]
CONFIGURED_LABELS = [PRIMARY_LABEL, *SETUP["labels"].get("variants", [])]
UNIVERSE = sorted(SETUP["universe"]["symbols"])
CALENDAR = EVAL_CFG["calendar"]
HOLDOUT_START = pd.Timestamp(EVAL_CFG["holdout_start"])
HOLDOUT_END = pd.Timestamp(EVAL_CFG["holdout_end"])
# The config states the holdout's last date; parsed as a timestamp it is that date at midnight,
# which is before every intraday bar of the session it names. Anything comparing a bar against the
# end of the holdout uses the exclusive bound instead.
HOLDOUT_END_EXCLUSIVE = HOLDOUT_END + pd.Timedelta(days=1)

# The bar is one minute, so a horizon written as a duration converts to a bar count exactly.
BAR = pd.Timedelta(minutes=1)
# The horizon the label measures, not the buffer that purges it. The buffer is one bar
# wider, because the label reads a quote one bar past its own horizon, and sampling IC
# on the buffer would space observations by an interval no label spans.
LABEL_HORIZON = SETUP["labels"]["horizons"][PRIMARY_LABEL]
LABEL_HORIZON_BARS = int(pd.Timedelta(LABEL_HORIZON) // BAR)
IC_SAMPLE_STEP = LABEL_HORIZON_BARS

# The model specification, read from `setup.yaml::model_based`. The three lengths of history the
# HAR regression averages squared returns over, the history each of its refits reads, how often it
# refits, the floor on a refit's usable observations, and the window each spectrum and each
# signature path spans.
MODEL_BASED = SETUP["model_based"]
HAR_COMPONENTS = tuple(MODEL_BASED["har"]["components"])
HAR_FIT_WINDOW = MODEL_BASED["har"]["fit_window"]
HAR_REFIT_EVERY = MODEL_BASED["har"]["refit_every"]
HAR_MIN_TRAIN_OBS = MODEL_BASED["har"]["min_train_obs"]
# The refit window is the burn-in: the walk's first estimate reads `HAR_FIT_WINDOW` bars and the
# bar after them is the first one any parameters speak for.
HAR_BURNIN = HAR_FIT_WINDOW + 1
FFT_WINDOW = MODEL_BASED["spectrum"]["window"]
FFT_LOW_FREQ_PERIOD = MODEL_BASED["spectrum"]["low_frequency_period"]
SIG_WINDOW = MODEL_BASED["signature"]["window"]

# The exchange's regular session. Bars outside it are not part of any decision.
OPEN_HOUR, OPEN_MINUTE, CLOSE_HOUR = 9, 30, 16

# Every table below is short enough to read whole, and a frame shown as ten rows and an
# ellipsis is a table the reader has to take on trust.
pl.Config.set_tbl_rows(40)
```

--------------------------------

### Print feasibility summary with setup.yaml settings

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/case_studies/crypto_perps_funding/01_feasibility_analysis.ipynb

Prints a summary of key settings from setup.yaml (universe.n_assets, decision.cadence, labels.primary, evaluation.n_splits) along with computed breadth statistics and split information. The output shows the breadth floor is never met, and the last validation ends on 2023-12-31.

```python
print(
    f"universe.n_assets {SETUP['universe']['n_assets']}, breadth {breadth['n'].min()} to "
    f"{breadth['n'].max()}, under the floor of {BREADTH_FLOOR} on "
    f"{breadth.filter(pl.col('n') < BREADTH_FLOOR).height:,} of {len(breadth):,} timestamps\n"
    f"decision.cadence {SETUP['decision']['cadence']} | labels.primary {PRIMARY_LABEL}\n"
    f"evaluation.n_splits {SETUP['evaluation']['n_splits']}, generated {len(splits)}, last "
    f"validation ends {last_val.date()}, holdout untouched"
)
```

```text
Output:
universe.n_assets 19, breadth 2 to 19, under the floor of 20 on 4,382 of 4,382 timestamps
decision.cadence 8_hour_funding_aligned | labels.primary fwd_ret_8h
evaluation.n_splits 2, generated 2, last validation ends 2023-12-31, holdout untouched

```

--------------------------------

### Set up directories and print paths

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/03_market_microstructure/04_itch_order_lifecycle_analysis.ipynb

Sets up input and output directories for the analysis. Creates the output directory if it does not exist and prints the paths and symbol being analyzed.

```python
NASDAQ_ITCH_OUTPUT = get_output_dir(3, "nasdaq_itch")
MESSAGE_DIR = load_nasdaq_itch(get_base_path=True)
OUTPUT_DIR = NASDAQ_ITCH_OUTPUT / "order_lifecycle"
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)

print(f"Input directory (messages):  {display_path(MESSAGE_DIR)}")
print(f"Output directory (analysis): {display_path(OUTPUT_DIR)}")
print(f"Symbol under analysis: {SYMBOL}")

```

--------------------------------

### Docker commands for pulling image, starting Jupyter, and running notebooks

Source: https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/docs/running-notebooks.md

Commands to pull the Docker image (one-time, ~12 GB on x86, ~3 GB on ARM64), start Jupyter Lab (keeps running and prints its log), run a notebook directly without Jupyter, and run with GPU for deep learning chapters. Requires Docker installed and the repository cloned.

```bash
# Pull the image (one time, ~12 GB on x86, ~3 GB on ARM64)
docker compose pull ml4t

# Start Jupyter Lab. It keeps running and prints its log here; leave it running.
docker compose up ml4t

# Run a notebook directly, without Jupyter
docker compose run --rm ml4t python 11_ml_pipeline/01_ols_inference.py

# Run with GPU (deep learning chapters)
docker compose --profile gpu run --rm ml4t-gpu python 13_dl_time_series/01_core_architectures.py
```
Tokens:
Raw