Backtest Discipline
Independent evaluations and numerical reproductions of published market research, with selected historical research records kept separate from the main case studies.
I independently evaluate selected published strategies and reproduce selected published research results using my research software. The main record below contains the evaluations and reproduction studies that have full public case-study pages.
The initial open-ended strategy-discovery phase is closed. A small set of older records is retained separately as an archive because they illustrate useful research outcomes, but they are not presented as equivalent to the published case studies.
Complete public write-ups with the research target, frozen evaluation or reproduction contract, result, limitations, and evidence trail.
Published-result reproduction · execution timing
PASSED
The BTC timing ledgers matched the paper's rounded wealth and Sharpe figures within frozen tolerances: 14.41× with corrected timing against 6,094.94× when a close-dependent signal was credited from the preceding open. Numerical parity is not a trading verdict.
Original hypothesis validation · historical holdout frozen before acquisition
DID_NOT_MEET_CRITERIA
The BTCUSDT hourly coefficient was slightly negative, but its one-sided 95% HAC upper bound remained positive. The frozen holdout did not establish the requested reversal relationship.
Published-finding historical out-of-sample validation
DID_NOT_MEET_CRITERIA
The event-weighted 30-minute mean was positive (+2.99 bps), but the one-sided 95% HAC lower bound was −2.77 bps. The fixed criterion was not met; this does not establish absence of the effect.
Original hypothesis validation
DID_NOT_MEET_CRITERIA
ETH's 30-day primary outperformed identically costed buy-and-hold and improved maximum drawdown, but its one-sided 95% HAC lower bound remained below zero. BTC's primary underperformed; neither market could rescue the other.
Original hypothesis validation
DID_NOT_MEET_CRITERIA
All six frozen BTC and ETH 50/20, 55/20 and 60/20 close-channel variants improved maximum drawdown, but none met the full excess-return-plus-drawdown criterion. BTC mean excess returns were negative; ETH point estimates were positive but every HAC lower bound remained below zero.
Published-strategy out-of-sample validation
DID_NOT_MEET_CRITERIA
The paper-highlighted SMA12 returned +284.30% after frozen costs, but buy-and-hold returned +729.21%. The decisive member had negative mean excess return and a negative HAC lower bound; SMA24 failed too.
Published-statistic numerical reproduction
PASSED
Strategy Lab reconstructed the fixed 38,977-bar BTCUSDT 15-minute sample and reproduced the frozen external R_kernel within 5 binary64 ULP, inside the predeclared tolerance.
Published-strategy out-of-sample validation
DID_NOT_MEET_CRITERIA
The strongest 50-day breakout turned $1 into about $19 after frozen costs, but buy-and-hold reached about $23.6. All three frozen 50/150/200-day members had negative mean excess return and none met excess_return_v2.
Published-strategy out-of-sample validation
DID_NOT_MEET_CRITERIA
The source-emphasized 20-day primary substantially lagged buy-and-hold. The 50- and 100-day variants looked attractive descriptively, but all five frozen members had a negative one-sided HAC lower bound and none met the predeclared excess-return criterion.
Prospectively frozen post-publication evaluation
FALSIFIED
A complete one-year BTCUSDT capture produced 739 executed trades. Mean daily net R and its one-sided HAC lower bound were negative, and all four frozen chronological folds lost net R under the primary cost model.
Published-strategy out-of-sample evaluation
FALSIFIED
All four published BTC lookbacks reduced maximum drawdown, but all four had negative mean excess return versus the frozen buy-and-hold benchmark. Zero of four variants survived the predeclared family rule.
Prospective evaluation
FALSIFIED
The frozen rule failed because net Sharpe was about 0.368 versus 0.50 required and the one-sided 95% HAC lower bound was negative. The 3-of-4 positive-fold criterion and the −50% drawdown boundary both passed.
Reproduction benchmark
FULL_REPLICATION_MATCH
Independent research machinery reproduced the externally specified executable behavior and independently recaptured the frozen official-data panel exactly. This established reproducibility, not profitability.
Older records retained because they illustrate useful failure modes or research boundaries. These entries do not have standalone public case-study pages.
Real-data deterministic evaluation
FALSIFIED
Reached evaluation on 1,000 rows and failed. The negative result was preserved rather than used as a prompt to retune the tested rule.
Deterministic evaluation
FALSIFIED
Generated 344 completed round trips, then failed its required aggregate cost-adjusted return and chronological-block criteria.
Prospective experiment freeze
REJECTED BEFORE CAPTURE
Required 100,000 historical rows per symbol series, but the frozen request budget could supply only 4,000 per series. No public-data request or evaluator run followed.
Data admission
CLOSED BEFORE EVALUATION
Required at least 5,000 normalized rows per symbol; the real capture produced 4,995. The five-row shortfall was not filled or worked around after inspection.
DID_NOT_MEET_CRITERIA is the current Strategy Lab terminal wording used when a completed frozen member or family does not satisfy its declared criterion. It does not mean universal unprofitability.
Older research artifacts use FALSIFIED for an empirical proposal that reached its frozen evaluation and failed the predeclared survival criteria. A pre-capture feasibility rejection or a data-admission failure is reported separately because no empirical strategy verdict was produced.
PASSED on Published Statistic Reproduction 001 is a numerical-conformance result: the independently reconstructed fixed sample reproduced the frozen external statistic within its predeclared tolerance. It is not a profitability verdict or a reproduction of every result in the source paper.
FULL_REPLICATION_MATCH belongs to a different category again: it says the earlier independent reproduction benchmark matched its specified behavior and input evidence, not that a trading strategy was profitable or deployable.
I report each outcome under the question that produced it. A source failure supplies no market verdict, and a numerical reproduction supplies no evidence of trading profitability.
A preserved checkpoint from the initial discovery phase recorded 16 completed empirical studies and 16 empirical falsifications. That is a historical checkpoint, not a running count of this publication’s evaluations. Other candidates stopped earlier because a frozen source requirement, data-admission rule, or capability boundary could not be satisfied.
The public record is deliberately selective. It grows through individually scoped evaluations of published strategies, occasional numerical reproductions of published results, and a small number of historical cases tied to preserved evidence. Results are reported against each test’s fixed criteria, whether they pass or fail; evidence limitations and pre-evaluation stops remain separate.
I publish passing, failing and inconclusive results within each test’s scope. A passing backtest alone would not qualify a strategy for trading.