Research 005 ended its protected one-year evaluation with a +5.26% compounded net return. The evaluator also reported a +20.50% annualized net return. Yet the terminal verdict was FALSIFIED.
I had required seven conditions to pass before accepting the strategy. Positive return was one of them. The Sharpe and confidence-bound conditions failed, so the gain could not change the verdict.
The 20.5% number needs a label
The frozen evaluator defined +20.50% annualized net return as:
mean daily net return × 365
The actual compounded change in wealth over the 365 evaluation rows was +5.26%.
Those numbers are not supposed to be equal. One is an arithmetic annualization of the average daily return; the other is the compounded path actually recorded by the evaluator. Volatility and the sequence of daily gains and losses separate the two.
I report both quantities with their conventions:
- compounded net return over the holdout: +5.26%;
- arithmetic annualized mean net return: +20.50%.
The difference between gross and net performance matters too. Before the frozen turnover cost, cumulative return was +28.39%. After the fixed cost model, it was +5.26%. The frozen evaluation did not respond to that gap by lowering the cost assumption after seeing the result. The cost rule was part of the experiment.
Positive return was necessary, not sufficient
Before the holdout was evaluated, the experiment defined seven survival criteria. SURVIVED_RESEARCH required all seven to pass.
The strategy passed five:
- exactly 365 evaluation rows;
- annualized net return greater than zero;
- at least three of four chronological folds with positive mean net return;
- maximum drawdown better than −50%;
- all required deterministic metrics finite.
It failed two:
- net Sharpe had to be at least 0.50;
- the one-sided 95% Bartlett-HAC lower bound on mean daily net return had to be above zero.
The recorded Sharpe was about 0.368. The HAC lower bound was about −0.1002% per day.
Once those numbers existed, there was no extra clause saying “unless cumulative return is positive.” Positive return was already represented by its own criterion. It could not be counted twice and used to erase failures elsewhere.
This is what freezing the decision rule before evaluation changes. If the rule is written only after the result appears, almost any attractive number can become the reason to accept the strategy. If the rule is frozen first, the result has to meet the rule rather than negotiate with it.
Freezing before capture and evaluation does not by itself mean the historical market observations occurred after commitment. This should not be read as evidence of a forward-monitored or live-traded strategy.
Why the Sharpe failure mattered
The Sharpe criterion asked a different question from cumulative return.
A positive cumulative return tells us where the compounded path ended relative to where it started. Sharpe relates average return to observed variability. The experiment used its own frozen definition and threshold: annualized net return divided by annualized net volatility, with survival requiring a value of at least 0.50.
At 0.368, it failed. That does not mean 0.50 is a universal boundary between a good and bad strategy. It was simply the boundary this experiment committed to in advance. A different research program could reasonably choose a different threshold, provided it made that choice before looking at the protected result.
Here, the failure says something narrower: the amount of return recorded in the holdout was not high enough relative to its observed variability to satisfy the experiment’s predeclared requirement.
The evaluation also recorded a maximum drawdown of about −29.59%. The drawdown criterion itself passed because the frozen boundary was −50%, but the path still helps interpret the +5.26% net gain: the experiment reached that gain while experiencing a peak-to-trough decline of nearly thirty percent.
Passing the drawdown gate did not make that path disappear, and it did not compensate for the failed Sharpe gate.
What the negative HAC lower bound actually says
The second failure is easier to overstate.
The frozen evaluation used a one-sided 95% Bartlett-HAC lower confidence bound on mean daily net return. The calculation was fixed in advance, including a 33-day lag and the requirement to use all 365 physical daily returns. HAC was used so that the uncertainty calculation could account for serial dependence and changing variance rather than treating the daily observations as independent with constant variance.
The resulting lower bound was approximately −0.1002% per day.
Under the frozen decision rule, survival required that lower bound to be strictly above zero. It was not, so the criterion failed.
The experiment did not establish the required positive lower confidence bound on mean daily net return under its chosen estimator and confidence rule.
It does not mean the calculation proved that the strategy’s true expected return is negative. It does not convert a positive sample mean into proof of negative profitability. It says that, given the variability and dependence present in this sample, the experiment’s required level of statistical support for a positive mean was not reached.
“The lower bound crossed zero” is a statement about uncertainty under a specified procedure. “The true return is negative” would be a much stronger claim, and this result does not establish it.
Three positive folds were encouraging — and still not enough
The 365-day holdout was also split into four frozen chronological blocks.
Their mean daily net returns were approximately:
- fold 1: +8.89 basis points/day;
- fold 2: +14.06 bps/day;
- fold 3: −14.27 bps/day;
- fold 4: +14.00 bps/day.
Three of the four were positive, so the fold criterion passed exactly as specified.
That is useful evidence. Positive performance was not confined to a single short interval. But the folds were never designed to override the other tests. They were one component of a multi-criterion decision.
The negative third fold also prevents an overly smooth story about persistence. The experiment showed positive means in most blocks, one materially negative block, a positive full-period return, weak risk-adjusted performance relative to its threshold, and a confidence bound that did not clear zero.
The experiment was designed to prevent the obvious rescue
After seeing the result, there are several tempting arguments one could make:
- treat 0.368 as close enough to 0.50;
- emphasize the three positive folds or the net gain;
- relax the HAC rule;
- give the passed drawdown condition more weight.
Any of those observations can motivate a later discussion about research design. None of them can change the verdict of this experiment.
The frozen decision specification explicitly prohibited post-result rescue by changing the criteria, costs, parameters, universe, dates, provider, fold selection, direction, or signal. If valid evidence existed and any required survival criterion failed, the terminal outcome was FALSIFIED.
I kept the failed criteria alongside the positive return. Changing their weight afterward would have changed the test.
Describing a result is not the same as accepting it
The strategy produced a positive net return over this holdout and failed the experiment’s frozen survival rule.
There is no paradox unless “made money” is treated as synonymous with “passed.” The test deliberately did not use that definition.
It was not asking whether at least one metric looked favorable. It was asking whether one frozen executable strategy, on one frozen 66-symbol universe, over the protected period from 1 September 2025 through 31 August 2026, satisfied all seven conditions committed to before evaluation.
It failed two conditions. The terminal result was therefore FALSIFIED, and no strategy was authorized for promotion into a trading runtime. The experiment involved neither paper nor live orders.
What a backtest developer can borrow from this
The most reusable part of this case is not its channel-breakout logic. It is the separation between measurement and decision.
Before opening a holdout, write down what would count as survival. Include more than the metric you expect to look best. Define transaction costs, data handling, risk metrics, uncertainty treatment and any stability checks before the result can influence those choices.
Report each outcome before applying the decision rule. In this case I kept the +5.26% cumulative net return and the +20.50% arithmetic annualization alongside the failed Sharpe and confidence conditions. The gains remained part of the record, with no change to the verdict.
The same questions apply when the backtest is someone else’s. Before you trust a backtest collects them into a checklist you can use on any strategy claim.
For the complete experiment record, see Research 005. The broader workflow is described in Methodology. The frozen rules and terminal outputs are available directly in the public evidence files: decision specification, deterministic evaluation, experiment definition, and terminal result.
A concise six-page version is also available as the Research 005 publication report (PDF).
No research was replayed, retuned or extended for this article. It is an editorial explanation of already-preserved evidence from the completed Research 005 experiment.