BOCPD Filter — Definitive Shuffle Test

2026-05-19 · status: inconclusive · 100.0s

Hypothesis: If the BOCPD p_short signal carries real information about trade quality, the +6.9–17.0% PnL lift from bocpd_filter@0.3 should disappear when we shuffle the p_short → timestamp mapping. If shuffled runs show similar lift, the C7 'edge' was just trade-count-reduction with re-routing benefit.

Verdict: MIXED — 2/3 strategies show real BOCPD signal (z>1.5). 1 look like noise, 0 show anti-signal. Consider deploying selectively or pause until more data.

Key metrics

Approach

For each of the 3 BB-Extreme strategies we run the FULL backtest (with re-routing) over 2023-01-01 → 2026-04-01 with three source configurations:

1. baseline_no_filter: bocpd_filter not active (θ=1.0)
2. real: bocpd_filter active at θ=0.3, real BOCPD parquet
3. shuffled (×5): bocpd_filter active at θ=0.3, p_short values randomly permuted across timestamps (same value distribution)

If the lift from real-filter is genuine BOCPD information, shuffled runs should show lift ≈ 0 (= baseline-pnl). If shuffled runs lift similarly, the lift comes from trade-count-reduction + re-routing, not from BOCPD.

Results

Reading the table: real_lift_pct = lift of real-source filter over baseline. shuf_mean_lift_pct = average lift of shuffled-source filter over baseline (across 5 permutations). z_score = (real_pnl − shuf_mean) / shuf_std. |z| > 2 means the real signal is genuinely different from shuffled.

Per-run raw PnL

BB_EXTREME_3

BB_EXTREME_1

BB_EXTREME_2

Verdict

MIXED — 2/3 strategies show real BOCPD signal (z>1.5). 1 look like noise, 0 show anti-signal. Consider deploying selectively or pause until more data.