Backtest Discipline
Engineering lessons from ongoing published-strategy evaluations and selected historical cases at Backtest Discipline.
In these notes I explain the implementation choices behind individual experiments: how I specified the rules, handled source data, checked calculations and preserved failed attempts.
Each note draws on a completed case. When the prose and the code disagree draws on Benchmark 004, where reproducing an external notebook exposed disagreements between its written description and the code that actually produced the result.
I add notes as new evaluations expose something worth explaining. Writing a note does not reopen the experiment or change its result.
This DAR(4) validation got messy before the confirmatory run ever started: failed numerical reconstruction, a bad frozen acquisition boundary, a preserved predecessor, a successor lineage, and finally 1,686 sealed forecasts. Most of the work was research math; the platform’s job was to keep the history honest.
Read the noteA five-ULP match was the visible result. Underneath it were harder engineering questions: what identifies the data, where approximate equality is allowed, how a completed run stays immutable, and how to add a second research shape without creating a second maintenance path.
Worked example: Published Statistic Reproduction 001 — Kitron & Wengrowicz
Read the noteI authenticated Kraken's official BTC/USD hourly archive, but the real source data could not reproduce the continuous 2016–2025 input used by Wu and Pinsky. One predeclared carry-forward hypothesis restored the shape but not the published results, so the candidate was rejected rather than reverse-engineered.
Read the noteA positive return describes one dimension of a backtest. It does not override a multi-criterion acceptance rule that was fixed before the result was known.
Worked example: Research 005 — frozen holdout case study
Read the noteBenchmark 004 reproduced an external channel-breakout notebook exactly enough to expose a harder question: when the written explanation and the executed program differ, which object are you actually testing?
Worked example: Benchmark 004 — independent reproduction
Read the note