Backtest Discipline

Engineering lesson from Benchmark 004

When the prose and the code disagree

A reproduction problem surfaced by Benchmark 004: what should count as the strategy when comments, documentation, and executed notebook behavior do not line up?
By Backtest Discipline Published 5 Sep 2026 Updated 22 Oct 2026 Based on a completed experiment

When I reproduced the public Statistical Arbitrage in Cryptocurrencies channel-breakout notebook for Benchmark 004, the prose described a 30-period volatility calculation. The executed code used 35 days. A volume-filter comment said 70th percentile; the parameter was 0.68.

I pinned the source to a specific Git commit and recorded those disagreements before running the independent reproduction. I followed the code that had generated the published metrics, keeping the discrepancies as audit findings. The result was FULL_REPLICATION_MATCH, with those findings still attached.

I followed the executed notebook

For Benchmark 004, the reproduction target was the behavior that actually generated the published display metrics. That meant following the executed notebook code where prose, comments and code conflicted.

I could not assume the code was the best interpretation of the author’s intent. But changing it would have changed the numerical result I was trying to reproduce. Several apparent documentation discrepancies affected portfolio behavior.

A 30-day description became a 35-day calculation

The explanatory material described volatility using a 30-period rolling standard deviation. The executed holdout path used a 35-day sample standard deviation.

Five days may not sound dramatic, but volatility enters portfolio sizing. Changing the lookback changes the sequence of weights, turnover, costs and returns. A reproducer who follows the prose has already stopped reproducing the executed result.

Benchmark 004 therefore preserved the 35-day behavior in its frozen executable specification and separately recorded the documentation mismatch.

“Lower exposure to high volatility” was not what the notebook executed

The prose described volatility scaling in the familiar direction: reduce exposure as volatility rises, with additional EWMA smoothing.

The executed holdout path did something materially different. It multiplied positions by the square root of rolling standard deviation, then normalized the portfolio, and did not apply the described EWMA smoothing in that path.

That changes relative asset weights. Implementing the prose might have produced a more conventional strategy, but it would have changed the reproduction target.

A “70th percentile” comment executed as 0.68

The notebook contained an inline description of a 70th-percentile volume threshold. The executable parameter was 0.68.

This is a smaller discrepancy, but it illustrates the same problem. Comments are evidence about intent; they are not necessarily evidence about execution. If the displayed backtest was generated with 0.68, reproducing it with 0.70 changes the target.

The audit kept both facts visible: the code used 0.68, while the nearby wording said 70th percentile.

“20 bps per trade” was actually a turnover formula

The source documentation summarized transaction cost as 20 basis points “per trade.” The executable portfolio calculation charged 20 bps multiplied by daily portfolio turnover.

Those two descriptions are not automatically equivalent. In a multi-asset portfolio with continuous weight changes, turnover is a specific quantitative object. A vague “per trade” interpretation leaves room for several incompatible implementations.

Again, Benchmark 004 reproduced the formula that generated the result and recorded the wording discrepancy instead of resolving it after the fact.

Not every ambiguity is a documentation typo

The source universe also depended on information that was difficult to reconstruct historically. The executable notebook required sufficient non-null close coverage over the full panel through August 2025 before evaluating the portfolio from 2024 onward. Later data availability could therefore influence which symbols appeared in an earlier evaluation period.

There was a second practical problem: the original data-selection path depended on then-current CoinGecko and Binance listings plus an older hourly file. Historical snapshots of that discovery process were not available in a form that could be recreated honestly later.

Benchmark 004 did not invent a substitute universe and call it equivalent. It froze the 66-symbol universe already present in the archived source, then asked a narrower question: could the underlying market fields for that frozen panel be independently recaptured from the official provider?

They could. The independent comparison covered 449,856 high/low/close/volume field cells with zero numeric and zero missingness disagreements. That established reproducibility of the frozen input. It did not retroactively remove the universe-selection limitation.

Why not fix the source before reproducing it?

Because reproduction and repair answer different questions.

  • A reproduction asks whether an independent implementation can recover the result produced by a defined object.
  • A repair asks what the strategy would do after changing a rule considered mistaken, ambiguous or unrealistic.

Both can be legitimate projects. Combining them is dangerous. Once a reproducer starts choosing the “better” interpretation of each discrepancy, the exercise quietly becomes strategy design.

For Benchmark 004, I kept the rule explicit: follow executed behavior for the reproduction target, but preserve disagreements as audit findings rather than laundering them into the reconstructed specification.

A useful reproduction discipline

The worked example suggests a compact set of rules for reproducing computational research:

  1. Pin the source. A moving repository branch is not an immutable specification.
  2. Identify the path that actually produced the reported result. Documentation alone may not be sufficient.
  3. Record prose/code/data disagreements before running the independent reproduction. A successful match should not erase criticism.
  4. Reimplement independently rather than merely rerunning the original notebook. Otherwise you mostly prove that the original environment can execute itself.
  5. Separate input reproduction from strategy reproduction. Matching calculations against an archived dataset is different from independently recovering the dataset.
  6. Do not convert reproduction into repair. Proposed corrections belong in a new experiment with a new specification.
  7. State what the match does not prove. Reproducibility is not evidence of future profitability, realistic execution or methodological soundness.

Why the distinction mattered later

Benchmark 004 ended with FULL_REPLICATION_MATCH. The executable behavior and the frozen data panel were reproducible, including the oddities documented by the audit.

That success still did not answer the question that mattered for future evidence: would the same frozen executable behavior survive a new period that had not been used to produce the historical result?

Research 005 took that next step. It carried the reproduced semantics into a new prospectively frozen 365-day holdout. The result was positive on some headline measures, but it failed two predeclared acceptance criteria and finished FALSIFIED.

I could reproduce the historical result in Benchmark 004. The same executable strategy then failed the later survival rule in Research 005. The reproduction gave me a defined object to carry forward; it supplied no guarantee that the object would pass another test.

Inspect the worked example

The detailed Benchmark 004 case study explains the two-stage reproduction and links the exact upstream notebook at its frozen commit.

The publication also exposes selected preserved evidence used for this note:

No research was replayed for this article. It is an editorial explanation of already-preserved evidence from the closed research program.