Lookahead Audit — Master-LGBM Feature Panel (May 2026)

2026-05-19 · triggered by the B4 finding (ETF-Flow Exp #18 had a lookahead bias)

Trigger

Exp #18 (ETF-Flow Event Study) reported IC = +0.37 and was "promoted". But the residual check Exp #18b (ETF-Flow Residual) showed:

• With the correct 22:00-UTC marker, the original IC shrinks to +0.044
• Same-day-confound IC = +0.405 (devastatingly high — flow_t and ret_t on the same day correlate strongly, because the ETF-flow data releases reflect the movement ON THE SAME DAY)
• Residualised: IC +0.075, CI includes zero

Root of the bias in Exp #18: Daily resampling with label="right", closed="right" sets the close-bar timestamp to "midnight of the next day". So when flow_t at "2024-01-11 00:00" is aligned with fwd_ret_1d.shift(-1), it shifts onto the BTC return of the same trading day (2024-01-11), not the day that follows. But the flow is only reported at 22:00 UTC of that day → classic lookahead.

Consequence for the other experiments: check all of them for whether the feature construction is causal. The Master-LGBM is the most important piece.

Master-LGBM feature-by-feature audit

Finding: all features are causal. The only minor note: - dxy_z_4h: The yfinance free tier has a ~15min publication delay. If we go live trading, we would have to either add an extra .shift(1) or use a paid real-time source. For a backtest on historical yfinance data the delay is irrelevant, because the values are written retroactively as they were available after the 15 minutes — i.e. the backtest is more conservative than reality.

Why Exp #18 was flawed

Verbose reconstruction of the bias:

flows index:        flow_t @ 2024-01-11 00:00 UTC   (was reported ~22:00 UTC that day)
df_1d index:        right-closed → close_at_2024-01-12_00:00 = end-of-day 2024-01-11
btc_ret_1d:         log(close_2024-01-12_00:00) - log(close_2024-01-11_00:00) = return DURING 2024-01-11
fwd_ret_1d.shift(-1) at index 2024-01-11_00:00 → btc_ret_1d at 2024-01-12_00:00 = return DURING 2024-01-11
                                                                                  ^^^^^^^^^^^^^^^^^^^^^^
                                                                                  same day as flow_t!

In words: for flows_total at "2024-01-11 00:00", the alignment is onto fwd_ret_1d at the same index, which is the return realized during 2024-01-11. But flow_t only became known at 22:00 UTC of that day — the model would effectively have looked into the future by labeling a return as "forward" that was already 90% complete before the feature even existed.

Corrections / consequences

Verdict updates

Synthesis update

4 promotes becomes 3 promotes + 1 lookahead drop:

• ⭐ BOCPD p_short (Exp #15)
• ⭐ DVOL/VRP (Exp #19)
• ⭐ Master-LGBM (Exp #22, validated)
• ❌ ETF-Flow Event Study (Exp #18, lookahead bias found)
• 🟡 HMM regime (Exp #16, pursue)
• 🟡 Stablecoin Δ7d (Exp #17, weak)
• 🟡 DXY shock (Exp #20, weak)
• ❌ HAR-RV-J (Exp #13)
• ❌ VPIN (Exp #14)

Master-LGBM stays promoted — but why?

The ETF feature has a 1-day shift in the Master-LGBM (flow_d.shift(1, freq="D")) and is therefore causal. Since Exp #18 itself gives no lift (residual IC +0.07), the ETF feature will be far down in the LGBM importance ranking (not in the top-3). The R² lift of +10.6 pp comes from:

1. log_rv_7d_ann (top-1 feature) — a vol-persistence extension beyond the HAR-RV cascade (HAR-RV uses 1h/4h/1d; the LGBM additionally gets 7d).
2. iv_ann (top-2) — DVOL brings implied vol as an orthogonal information source. This is the VRP insight from Exp #19.
3. hour_cos (top-3) — vol has a clear intraday pattern; LGBM uses it nonlinearly better than HAR-RV without a time feature.

These three are all causally clean. The ETF feature could even slightly hurt (noise). A robustness variant without ETF would be a sensible validation — see Next Steps.

Recommended follow-up audits

These audits are still open for the other promotes:

BOCPD (Exp #15) — how causal is the z-score?

The BOCPD input is z = r / rolling_std(r, 96 bars). The rolling-std uses past bars — causal. But: r.rolling(96, min_periods=48).std() with the default pandas settings — is that really causal (backward only) and not somehow centered? .rolling(N) is right-aligned by default (label = right edge), so it uses only the last N values. ✅ Causal.

VRP (Exp #19) — DVOL timing exactly

DVOL daily close stamped 00:00 UTC. We ffill onto the 4h panel without an explicit shift. That means: at bar 2024-01-11 00:00, DVOL = today's daily close. But that is only available at 00:00 UTC. If we build a prediction for the NEXT 4h vol (i.e. vol from 00:00-04:00), we use a DVOL that just "closed". Tight but okay — the information is physically live.

In the Master-LGBM we use .shift(1) for a 1h lag — more conservative. ✓

Stablecoin (Exp #17) — sc_daily timestamp semantics

When exactly does DefiLlama publish the totalCirculatingUSD for "2024-01-11"? The API has no explicitly documented release time. Conservative: the 1d shift in the Master-LGBM is sufficient.

Check walk-forward embargo

walk_forward_splits(embargo_minutes=max(1440, 4*60)) is 1440 min (1 day), which is sufficient for a fwd-4h target. But for fwd-1d targets the embargo would have to be ≥ 1d. In the Master-LGBM the target is fwd-4h, so 1440 is enough ≥ 4h.

In Exp #15 BOCPD: embargo_minutes=max(1440, 4*60) = 1440 min, target is fwd-4h. ✓ In Exp #18 ETF-Flow: embargo_minutes=1440 with target fwd-1d. We would need ≥ 1d embargo. It is exactly 1d → borderline but OK.

Lessons Learned

1. Check pandas resampling semantics. label="right", closed="right" on a daily aggregation packs an entire day into the midnight-of-the-FOLLOWING-day label. That is intuitive, but if you then do shift(-1), you land in the past. With non-trivial data sources (e.g. Farside), always reason explicitly about the release time, don't blindly take over the parquet timestamp.

2. Recognise the same-day confound. Every time a feature and a label occur on the same day (even if the label is labeled as "forward" in the DataFrame), check for confound. Standard test: measure the feature_t → ret_pre_t IC. If that is high, cross-check with residualisation.

3. Causal forward filter ≠ smoothing. With HMM/BOCPD features, the choice between a forward-only filter and backward smoothing is the most common lookahead spot. We use strictly forward — good.

4. Feature importance helps with the sanity check. If a presumed edge feature (ETF) does not land in the top importance in the Master-LGBM, that is a hint that the standalone IC may have been confound.

Action Items

1. ✅ Update the synthesis doc — ETF-Flow as ❌ instead of ⭐
2. ⏳ Master-LGBM validation without the ETF feature: expected R² lift identical or slightly higher → confirms that ETF does not contribute materially
3. ⏳ For future experiments: before dropna() and shift(), create a trace plot (plot feature value + label value over time with annotations for release times)
4. ⏳ Set the embargo in walk_forward_splits for 1d targets to 2d (safety buffer)