Trading research is easy to optimize. Harder to falsify.
A technical publication built from a closed crypto-strategy research program: real case studies, methods, and engineering lessons about deciding whether a backtest deserves trust.
A backtest can be made more attractive in many small ways: change a parameter, move a date, choose another cost assumption, discard an awkward period, or reinterpret the success criterion after seeing the result. The difficult part is building a process that can still say no when the numbers look tempting.
Backtest Discipline documents that process using preserved work from a completed research program. Strategy discovery is closed. The site is not a source of trading signals; it is a record of how hypotheses were reproduced, frozen, evaluated, rejected, or stopped before evaluation when the evidence contract could not be satisfied.
Publication scope
Three ways into the work
The publication is broader than one experiment. Research 005 is the most complete falsification case today, while Benchmark 004 provides the corresponding reproduction case. Methodology explains the controls; engineering notes pull reusable lessons out of those concrete cases.
Case studies
What happened to actual research ideas
Completed falsifications, reproduction benchmarks, and experiments that stopped at source or data-admission gates. Each case should answer what was fixed in advance, what happened, and why the recorded outcome stood.
Prospective freezing, deterministic evaluation, transaction costs, chronological folds, provenance, and falsification — explained as practical controls rather than abstract research ceremony.
Technical notes start from a preserved research case, then pull out a broader lesson about reproducibility, data quality, executable specifications, provenance, or research software boundaries.
The project tested more than one candidate mechanism. A preserved checkpoint already recorded 16 completed empirical studies and 16 falsifications; other candidates stopped earlier at feasibility or data-admission gates. These examples show several ways a controlled research cycle can end.
Real-data evaluation
Attention-shock cycle
A bounded Binance cycle reached deterministic evaluation and terminated as falsified. The result was preserved rather than retuned after the outcome was known.
FALSIFIED
Deterministic evaluation
Reference-price anchor adjustment
A later frozen proposal failed its required return criteria and was registered as falsified rather than being passed forward for qualification.
FALSIFIED
Pre-capture gate
Source-budget feasibility
One proposal required more history per symbol than its frozen request budget could supply. It stopped before any public-data capture or evaluation.
REJECTED BEFORE CAPTURE
Data admission
Coinbase history completeness
A real capture returned 4,995 rows per symbol against a prospectively frozen 5,000-row minimum. The shortfall was not filled or worked around after inspection.
The final closure work ended with two different scientific outcomes: independent reproduction succeeded, while the final prospective strategy evaluation did not satisfy its complete frozen acceptance rule.
Independent reproduction
Reproduction worked
Benchmark 004
Benchmark 004 independently reimplemented an external channel-breakout notebook and then reproduced it using independently captured official Binance data. The point was reproducibility, not an endorsement of the historical return.
Research 005 froze the reproduced executable semantics for 2025-09-01 through 2026-08-31 over a fixed 66-symbol universe. The annualized return looked encouraging. The complete acceptance rule did not pass, so the terminal classification stayed FALSIFIED.
Because a controlled failure can be more informative than another optimized equity curve.
Research 005 is the clearest example, but it is not the whole publication. Other cases show different failure modes: a hypothesis can fail empirically, a source budget can make an experiment impossible before capture, a real dataset can miss a frozen admission requirement, or an independent reproduction can succeed without saying anything about future profitability.
Those distinctions are useful because they stop very different events from collapsing into the same vague label of “the backtest did not work.”
What remains useful
The durable output of the project is a practical research workflow and the evidence left by applying it:
bind data and transformations to explicit provenance;
distinguish an external idea from the executable object actually tested;
freeze the strategy, costs, timing, and acceptance rule before evaluation;
make the evaluator deterministic enough to reproduce exactly;
test independent reproduction before trusting novel results;
preserve negative results instead of retuning them away;
keep research authority separate from any trading runtime.
The publication will grow by explaining those lessons through a small number of evidence-backed cases and technical notes — not by restarting strategy discovery.