ML Roadmap — May 2026

Last updated: 2026-05-19 · 9 experiments + 2 audits · 3 promoted, 2 pursue, 4 dropped (1 due to lookahead audit)

🎯 Final standings: synthesis · ⚠️ B4/B5 audit: lookahead_audit (ETF-Flow Exp #18 had a lookahead bias, Master-LGBM stays validated)

This plan documents a second, methods-oriented wave in the ml/ module. The first wave (see ml/experiments/) did exclusively point-in-time slicing — at time t, forming quintiles on vol, hour, funding, etc. Result: 1 robust edge (vol-persistence, now live in ml/forecast/), 1 fragile-real edge (FOMC drift), the rest dead or marginal.

The second wave expands along two orthogonal axes:

1. Methods that exploit the sequence, not just the point in time — jumps, change-points, HMM state, order-flow imbalance.
2. New data sources we haven't touched so far — stablecoin supply, ETF flows, BTC-VIX (DVOL), cross-asset macro.

The selection follows two research passes (methods + data sources, conducted 2026-05-19) and is ranked by expected lift × effort. Walk-forward embargo and causal feature construction are non-negotiable — the first wave showed that 5 of 6 in-sample findings die under walk-forward.

Validation standards (apply to every experiment)

• Walk-forward splits: 12mo train / 3mo test rolling, embargo ≥ max forward horizon. For data sources with a shorter history: 6mo train / 2mo test (e.g. ETF flows since 2024-01).
• Information Coefficient (IC): Spearman rank correlation, bootstrap 95% CI, p-value.
• Stability: at least 70% of walk-forward windows with IC in the expected sign.
• Promote gate: pooled IC > 0.05 + stability ≥ 70% + bootstrap-CI-low > 0.
• Inconclusive: individual windows fit, but no robust pattern.
• Dropped: IC ≤ 0 or CI-low ≤ 0.

Batch 1 — Methods without new data sources

These experiments run directly on existing 1m OHLCV + funding.

Exp #13 · HAR-RV-J — Jump-augmented vol forecast

Hypothesis: Separating continuous vol from jumps improves predict_vol_4h by ≥3% R² walk-forward over the baseline HAR-RV (which is currently live).

Method: Bipower Variation (Barndorff-Nielsen/Shephard) → continuous part. Lee-Mykland jump test → jump indicator. HAR-RV-J regression with daily, weekly, monthly RV plus jump component.

Data: only 1m BTCUSDT futures.

Expectation: positive improvement in high-vol phases; marginal in low-vol.

Exp #14 · VPIN from 1m bars

Hypothesis: Bulk-volume-classification VPIN spikes > Q95 predict fwd-1h |return| > Q90 with a hit-rate > 55%.

Method: Easley/López de Prado/O'Hara BVC: split the volume of each 1m bar into buy/sell via the normal CDF of the return. Equal-volume buckets, rolling |B−S|/V.

Data: 1m close + volume (optional: Binance derivatives taker-buy-ratio as a better source for the buy/sell split).

Expectation: a "toxicity" signal that spikes before jumps. Published evidence on BTC (Bouri et al. 2026, RIBAF).

Exp #15 · BOCPD change-point feature

Hypothesis: Bayesian Online Change-Point Detection on 15m returns: in bars with P(run_length < 4) > 0.5, the fwd-4h vol is +15-20% higher than baseline.

Method: Adams & MacKay BOCPD with a Gaussian Unknown Parameters model, λ=1/96. Causal by construction — forward only, no smoothing.

Data: only 1m OHLCV (resampled to 15m).

Expectation: complementary to vol-persistence (persistence = stay, change-point = break). Low to medium IC expected, but compositionally valuable.

Exp #16 · HMM regime as a feature (filtered only)

Hypothesis: A 2-state Gaussian HMM on (1h log-return, 4h RV) gives a causal regime tag that calibratably separates state-conditional realized vs. forecast vol.

Method: hmmlearn GaussianHMM(n=2). Rolling 90d fit, forward-filter only (no Viterbi smoothing), weekly refit.

Data: only OHLCV.

Expectation: State 0 = "Quiet", State 1 = "Turbulent". When applied correctly, a sizing input, not a standalone signal.

Pitfall warning: 90% of published HMM papers use smoothing or fit on the full sample → not actionable. We validate strictly forward-only.

Batch 2 — New data sources, single-source experiments

Exp #17 · Stablecoin supply drift

Hypothesis: 7d-Δ(USDT+USDC market cap) > Q80 → fwd-1d BTC return > baseline + 40 bps walk-forward.

Method: Quintile IC with bootstrap CI. Secondary: fwd-4h (presumably a pre-move is visible).

Data: DefiLlama /stablecoincharts/all (free, no key, no rate limit, daily, since 2018).

Expectation: BIS 1270 documents a direct effect in the 1-30d range; open whether a 4h pre-move signal emerges.

Exp #18 · ETF-Flow event study

Hypothesis: Days with net inflow > $500M (IBIT+FBTC+...) show an Asian-session gap-fade pattern; net-outflow days show continuation. Effect is session-dependent.

Method: Event study analogous to calendar_events. Walk-forward on ~28 months of history (since the ETF launch Jan 2024) → 6mo train / 2mo test.

Data: SoSoValue Developer API (free demo) or Farside HTML scrape.

Expectation: small n (~600 days) → cautious bootstrap CIs. 22:00 NY release → Asian open is cleanly separable in time (US-macro noise minimal).

Exp #19 · DVOL/realized-vol spread (variance risk premium)

Hypothesis: VRP = DVOL - rv_4h (annualised). VRP > Q80 → fwd-4h realized vol falls (rich premium reverts); VRP < Q20 → fwd-4h vol rises.

Method: Quintile IC. Pure vol-forecast improvement, no directional signal.

Data: Deribit /public/get_volatility_index_data (free, no auth, since 2021).

Expectation: Textbook edge in equities (Bollerslev/Tauchen/Zhou 2009), under-tested on BTC. Highest expected lift in Batch 2.

Exp #20 · DXY shock → BTC reversal

Hypothesis: A DXY 4h return ≥ 2σ pushes BTC in the opposite direction over fwd-1h with a hit-rate > 55%.

Method: Event study on DXY shocks; conditional BTC fwd-1h distribution vs. baseline.

Data: yfinance DX=F 1h (2 years available) + daily fallback (longer).

Expectation: The BTC–DXY negative beta is robustly documented (arXiv 2501.09911). 4h horizon = US-session-specific.

Batch 3 — Synthesis

Exp #22 · Master-LightGBM on the full feature panel

Hypothesis: A GBM on the lag panel (returns 1/5/15/60/240m + RV same windows + funding + OI Δ + taker ratio + stablecoin Δ + ETF flow + DXY/VIX 4h returns + DVOL + HMM state + VPIN + BOCPD run-length) has fwd-4h IC ≥ 0.06 walk-forward, dominates every single feature.

Method: LGBMRegressor, TimeSeriesSplit with embargo = max horizon (4h). SHAP for feature importance. Robustness: same run with the top-5 features only.

Data: everything so far + all new sources.

Expectation: paper evidence (arXiv 2511.20105 Nov 2025): LGBM beats econometric and RF baselines on BTC, CRPS −23%. Main risk: label leakage — if even a single lag construction is not causal, everything is garbage.

Skip / Defer

⏸ Exp #21 · CVD from Binance aggTrades (deferred)

Status: postponed. Backfill would be ~6-8h compute, storage ~50GB raw. When to come back: if Batch 2 does not deliver sufficient microstructure signals.

❌ Bench (not in this wave)

• Fractional differencing as a preprocessor — add later if Exp #22 stagnates
• Matrix Profile / motif discovery — high overfitting risk, exploratory
• Kimchi premium (Upbit) — more of a 1d/7d horizon
• Coin Metrics Community — free, but lower 4h IC expected
• 25-delta risk reversal (Deribit) — bonus to Exp #19

❌ Skip (with rationale)

• Wavelets as a standalone signal — boundary-lookahead trap in 90% of papers
• DFA/Hurst as a direct signal — BTC Hurst ≈0.5 short-term, diagnostic yes, predictive no
• Granger causality directly — non-linear, LGBM with lag features dominates
• LSTM/Transformer — no GPU, no documented edge before exhausting #13-22
• Glassnode/Tardis/Twitter — budget/ROI not justified

Status tracking — all experiments complete ✓

→ Full evaluation in the synthesis report.

Sources (condensed)

• HAR-RV-J: Andersen/Bollerslev/Diebold "Roughing it up" (2007); Lee & Mykland (2008); Scaillet et al. arXiv 1704.08175 (BTC jumps).
• VPIN: Easley/López de Prado/O'Hara, J. Portfolio Management 2012; Bouri et al., RIBAF 2026 (BTC).
• BOCPD: Adams & MacKay 2007.
• HMM regimes: MDPI Mathematics 13(10):1577 (2025); arXiv 2011.03741.
• VRP: Bollerslev/Tauchen/Zhou, Rev. Financial Studies 2009; Deribit DVOL Insights.
• DXY-BTC: arXiv 2501.09911 (2025).
• LightGBM BTC: arXiv 2511.20105 (Nov 2025).
• Stablecoin → BTC: BIS Working Paper 1270; AInvest 2025/2026.