ML Synthesis — May 2026 wave

2026-05-19 · 9 + 5 validation/audit experiments · 3 promotes + 2 strategy-specific filter deploys, 2 pursue, 4 dropped

⚠️ B4 audit update: Exp #18 ETF-Flow Event Study had a lookahead bias (same-day flow ↔ same-day return). The true forward IC is close to 0. Audit-Doc · Residual check

✅ B5 validation: The Master-LGBM without the ETF feature delivers a marginally higher lift (+10.74 pp vs +10.60 pp). All other features are causally clean. Master-LGBM stays promoted. No-ETF check

🎯 C7-D Definitive Shuffle Test (2026-05-19): BOCPD filter on BB-Extreme: BB_EXTREME_1 + _2 show a real BOCPD signal (z = +7.30, +6.50, real-lift +12.6%/+17.0% vs shuffled ≈ 0%). BB_EXTREME_3 marginal (z = +1.06). Deploy on 2 of 3 wallets. Shuffle test · Walk-Forward+Threshold

This summary covers the second ML wave (Exp #13 — #22, skipping #21 CVD). It complements the roadmap and offers a "what did we learn" view of what got promoted, what is dead, and which steps make sense next.

Final-standings table

Three promotes after the audit correction. ETF-Flow (#18) had a classic same-day-confound lookahead bias that only became visible in the residual check. The remaining three come from: - New methods (BOCPD, VRP) - Synthesis (Master-LGBM, audited)

For comparison: the first wave had 1/12 promoted (vol_clustering). 3/9 is still a very strong rate — and the audits are exactly what you MUST do before anything goes into production.

The 4 real edges, in detail

1. BOCPD change-point (Exp #15) — vol pre-trigger

• What we know: When the Bayesian Online Change-Point Detection diagnoses a "fresh change-point" (P(run < 4) > 0.5), the fwd-4h vol is +27% above baseline.
• Complementary to vol clustering: Persistence says "quiet stays quiet", BOCPD says "break → more vol". Together they are orthogonal.
• Use: Feature for the Master-LGBM (already in); additionally conceivable as an entry filter (no trend trade during fresh-CP phases).
• Caveat: The z-score normalisation uses rolling-std → indirect vol information. But: the CP indicator measures breaks in the standardised return, not the vol level itself.

2. ❌ ETF-Flow → next-day BTC (Exp #18) — lookahead bias found (B4 audit)

• What the presumed edge was: IC +0.37, +463 bps spread between big-inflow and big-outflow days.
• What was actually found: With the correct 22:00-UTC marker (the release time of the ETF data), the IC drops to +0.044, and the same-day-confound IC = +0.405. Exp #18 accidentally labeled the same-day returns as "forward" (via pandas right-closed daily resampling).
• Residual check (Exp #18b): after orthogonalising against ret_pre, the IC remains +0.075, CI [−0.005, +0.152] includes zero.
• Verdict: ❌ DROPPED. ETF-Flow is not a directional signal beyond BTC momentum.
• Anti-lessons: Pandas resampling semantics (label="right", closed="right") are treacherous. ALWAYS check for same-day confound with event data that is timestamped retrospectively.
• ETF as a Master-LGBM feature: has a 1d shift at construction, so it is causal. The audit shows: it even contributes marginally negatively to the lift (-0.14 pp). Disturbs the model minimally; can be dropped.

3. DVOL/VRP (Exp #19) — vol mean-reversion

• What we know: VRP = DVOL (annualised IV) minus trailing 4h RV. High VRP (market prices high) → realized vol comes down. IC = −0.28, 16/16 windows negative.
• Theoretical foundation: Classic Variance Risk Premium (Bollerslev/Tauchen/Zhou 2009). On BTC barely tested systematically by us so far; now validated.
• Use:
• Directly as an input for ml/forecast/ alongside Persistence/HAR-RV.
• VRP-conditioned position sizing: larger positions at high VRP (vol-targeting becomes better calibrated).
• Data source: Deribit DVOL daily, free, no auth, since 2021-03.

4. Master-LightGBM (Exp #22 + #22b validated) — the synthesis

• What we know: LGBM on the full feature panel raises R²(log) for fwd 4h vol by +10.6 pp over HAR-RV (0.554 → 0.660). Pooled IC rises from 0.726 to 0.811.
• The B5 audit showed: All features are causally clean. After removing the "suspicious" ETF feature, the lift even rises slightly to +10.74 pp (Exp #22b). Master-LGBM is robustly validated.
• Top-3 features (without ETF): log_rv_7d_ann (7d trailing RV — vol-persistence dominant), iv_ann (DVOL — the VRP information), stablecoin_d7 (interesting: stablecoin Δ was weak standalone, but LGBM can use it nonlinearly).
• Use: Replaces HAR-RV in ml/forecast/api.py in the medium term. First run it in parallel as a shadow model, then switch over after 4-6 weeks with a calibration-drift check. Leave out the ETF feature at production deploy (no-op).
• Caveats:
• Only 13 walk-forward windows (start 2022-01 due to several feature-availability constraints).
• DXY is NaN in most windows — LGBM only uses it from 2024 on.
• The top feature is NOT one of the new features but the longest vol lag. Confirmed: vol-persistence is and remains the foundation; the new features (above all VRP) deliver edge refinement, not edge replacement.

What we did NOT find, against earlier hopes

• HAR-RV-J (jumps): practically zero added value. 1m BTC returns are dominated by microstructure noise; the "jump" component carries no information of its own. On tick data this could change (deferred → Exp #21).
• VPIN on 1m bars: Bulk-volume classification is too coarse. Real VPIN needs tick data.
• HMM standalone: The high IC (+0.39) is real, but the separation ratio is only 1.42× — the HMM states are not clearly enough separated from each other for standalone routing.
• Stablecoin Δ7d: Daily granularity too slow for a 4h edge; the 1d quintile spread is only +25 bps, below our 40-bps gate.
• DXY shock: Direction fits (USD up → BTC down), but the magnitudes in 1h fwd return are too small (±6-8 bps).

Lessons learned from this wave

1. Causal feature construction is not trivial. ETF-Flow had to be shifted by 1 day, DVOL by 1 hour. Without that, the ICs would be much higher and pure lookahead.

2. Smoothing vs filtering with HMMs is the point where almost all papers go under. We use a strictly forward filter; that explains the lower separation but gives realistic OOS expectations.

3. Native NaN handling in LGBM is a game-changer for mixed data histories. We don't have to reduce all features to the shortest common time window.

4. The "new data sources" research paid off: VRP (DVOL) and ETF-Flow alone justify the effort. DXY/stablecoins did not pay off (too daily, too trend-following), Glassnode/Twitter would have been money thrown away.

5. Methods vs data: The methods research brought BOCPD (clear promote). The data research brought VRP and ETF. Both were necessary.

Next Steps

Immediate (this/next week)

1. Integrate VRP into ml/forecast/api.py — as an additional feature for the existing GBM/HAR-RV pipeline. First an extension of live.py, then a calibration pass.
2. Run Master-LGBM as a shadow model. Daily predict_vol_4h_lgbm() in parallel to the existing API; drift tracking.
3. ETF-Flow daily snapshot as a cron job (analogous to download_derivatives.py) — Farside scrape at 23:00 UTC.

Short-term (2-4 weeks)

1. ETF momentum residual check: How much of the +0.37 IC remains once I residualise the BTC return on day t out of the feature signal?
2. BOCPD as an entry filter for strategies/: no trend entry at p_short > 0.5; mean-reversion entry preferred during this phase.
3. CVD from Binance aggTrades (Exp #21) run it. Backfill ~50GB raw, ~3GB aggregated. Will be the only BTC microstructure signal we can keep locally.

Medium-term (this quarter)

1. Directional counterpart to the Master-LGBM: So far we have only forecast fwd-4h VOL. Test a directional model (fwd-return sign) with the same features. Expectation: much harder; but if IC > 0.03 is walk-forward stable, it is worth it.
2. Integrate the Coinglass liquidation heatmap (once their 500 issue is resolved — see the [[coinglass-switch]] memory). This is the only large free microstructure edge we don't have yet.
3. Bench sweep: Fractional differencing as a preprocessor, Matrix Profile on 15m close, Risk Reversal from Deribit options. ~4h implementation each; only worth it if the Master-LGBM stagnates in production.

Caution

The promote rate of 4/9 is unusually high. Before a production deploy of any LGBM variant, a second walk-forward validation with explicitly purged/embargoed CV — if even a single feature construction is not causal, all 10 pp are garbage. Check specifically: - DVOL shift by 1h: is that enough? - ETF-Flow shift by 1d: correct? - BOCPD posterior at t: does it use only x_1..x_t?

[[har-rv-j]] [[vpin]] [[bocpd]] [[hmm-regime]] [[stablecoin-drift]] [[etf-flow-event]] [[dvol-vrp]] [[dxy-shock]] [[master-lgbm]]