Someone shows you a backtest with a good headline number. It might be your own, a colleague’s, a paper’s or a video’s. These are the questions I ask before treating that number as evidence. Each group links to a case on this site where the question changed what the result could support.
The checklist helps you find out what is documented and what is missing. It gives no score or certification. An honest set of answers can still describe a strategy that loses money, and an unanswered question does not make a result wrong; it tells you where to look next.
Work through the groups in any order and record each answer as documented, unclear, unavailable, or not applicable with a reason. You can open or save a plain Markdown worksheet with the same questions and fill it in with any text editor; no account or other software is needed.
1. What exactly is being tested?
A strategy description, its code and its published numbers are not always the same object. Before checking anything else, find out which one produced the result.
- Is the source pinned to a version: a commit, a paper DOI, a dated video?
- Where the prose, comments and code disagree, which one generated the reported numbers?
- Are any rules discretionary (“near a key level”, “use judgement on the stop”)? How were they turned into code, and what was left out?
- Is every parameter value stated, or do some have to be inferred?
Example. When I reproduced an external channel-breakout notebook in Benchmark 004, the prose described a 30-period volatility calculation, but the executed code used a 35-day sample standard deviation. It also multiplied positions by the square root of volatility instead of reducing exposure as volatility rose. A comment said “70th percentile” where the parameter was 0.68, and “20 bps per trade” turned out to be 20 bps multiplied by daily portfolio turnover. I followed the executed code because that was what produced the published numbers, and I recorded the disagreements before running the reproduction. When the prose and the code disagree goes through each one.
For unusually large breakouts, the source video behind Research 007 suggested placing the stop at a nearby resistance level or an order block. Those choices cannot be reconstructed from price bars without inventing a rule, so I excluded them, and the result covers only the deterministic core of the strategy.
2. Which data, and is it complete?
A matching checksum shows that a file is identical to the one the provider’s checksum describes. Whether the file contains what the analysis needs is a separate question.
- Which venue, instrument and bar interval? Is it standing in for something broader, such as one exchange pair for “Bitcoin”?
- Can the exact input be identified (a file hash, an archive version) or captured again?
- How many rows were expected and how many arrived? Were gaps filled, dropped, or left missing under a stated rule?
- Was any repair or filling rule chosen after the result was visible?
Example. For one reproduction candidate, the official Kraken archive passed its own checksum chain. It still contained 87,408 hourly rows where the published analysis needed a continuous grid of 87,672, so 264 hours were missing. I tested one carry-forward rule that I had declared in advance. It filled the grid but did not reproduce the published outputs, so I stopped instead of searching for a rule that would. The data were authentic. The reproduction still failed. explains why that is a data stop and not a verdict on the strategy.
By contrast, the independent recapture in Benchmark 004 matched all 449,856 compared high, low, close and volume cells of the archived input, with no numeric or missingness disagreements.
3. Could the backtest know something it shouldn’t?
- When does a signal form (at the bar close?), and when is the trade assumed to execute: the same close, or the next open?
- Was the set of instruments chosen using information from after the test started, such as current listings, survival, or data coverage over the full period?
- When one bar reaches both the stop and the target, which is assumed to happen first?
- Which time zone defines a “day” or a trading session?
The notebook behind Benchmark 004 kept only symbols with at least 90% non-missing closes over the whole panel through August 2025, then evaluated the portfolio from 2024 onward. Later data availability therefore shaped which symbols appeared in the earlier period. I reproduced that behavior because the question was reproducibility, and listed it among the audit findings recorded before the result.
In Research 007 I wrote the execution timing into the frozen rules. Entry happened at the next 5-minute open after the confirming close. When one bar could have reached both the stop and the target, the stop was assumed to come first, and trading dates followed New York time.
4. What do the costs and execution assume?
- What does the cost formula actually charge: per trade, per side, or per unit of turnover? Does it include spread and slippage?
- How large is the gap between the gross and net results?
- Was the cost assumption fixed before the result, and does the conclusion survive a reasonable range of costs?
- If the strategy goes short or uses leverage, are borrowing, funding, margin and liquidation modelled, or is the test run on spot prices alone?
Example. In Research 005, cumulative return over the frozen 365-day holdout was +28.39% before the fixed turnover charge and +5.26% after it. The strategy could also hold negative weights while the data were Binance spot prices, so borrowing, funding and margin were outside the experiment.
In Research 007, the primary cost was 15 bps per side. The lower 10 bps diagnostic, also fixed in advance, was still strongly negative. Costs made that result worse, but they were not why it failed.
5. Which return, and compared with what?
The same equity curve can produce several honest-looking percentages. Check the label on every return figure.
- Is the headline a compounded total, an arithmetic annualized mean, a geometric annualized rate, or a log return?
- Is there a benchmark, and is it charged the same costs?
- What does the strategy hold when it is out of the market: cash earning nothing, or a risk-free rate?
- Is the claim absolute (“it made money”) or relative (“it beat holding the asset”)?
Example. Research 005 reported a +20.50% annualized net return, defined as mean daily net return × 365. The compounded net change over the holdout was +5.26%; Why a +20.5% annualized return still failed the frozen test explains how the two relate.
In my validation of Gerritsen et al.’s breakout rules, the 50-day rule turned $1 into about $19 after frozen costs from 2019 through 2025. Identically costed buy-and-hold turned it into about $23.6, and the rule’s mean daily excess log return over that benchmark was negative. The paper’s investor also earned a risk-free rate while out of Bitcoin. My validation left that capital unchanged, which is one of the substitutions I declared before evaluation.
6. How were the parameters chosen, and when was the test fixed?
- How many variants were tried? Is the reported one the best of many?
- Was a primary variant named before the result, and can another variant replace it afterward?
- What was fixed before the test data were seen? Is “before” measured against the data download or against the market observations themselves?
- Is the test period a live forward test, a later historical period, or the same period used to develop the rules?
In my validation of Grobys et al.’s moving-average rules, I named the 20-day rule primary before evaluation because the paper emphasizes it. The 100-day rule compounded to +3152.03%, against +2260.99% for buy-and-hold, but it had been frozen as a robustness member and could not replace the primary afterward. Its lower confidence bound for excess return was also below zero.
In the Bitcoin hourly reversal test, the recorded timestamps put the method freeze at 17:32:14 UTC on 19 October 2026 and the holdout archive capture at 17:34:58 UTC, which places the freeze before the capture but says nothing about whether the January–August 2026 market observations happened after the study was designed. Methodology explains how I separate the two.
7. Is the result distinguishable from noise?
- How many observations support the estimate (trades, days, events), and do they overlap?
- Is there a confidence interval or bound, and does it allow for dependence between observations?
- Does the result hold across sub-periods, or does one stretch carry it?
- Was the threshold for “established” set before the result was known?
Example. In the Ethereum extreme-move test, the mean 30-minute response after large one-minute gains was +2.99 basis points across 1,485 events. The one-sided 95% lower bound was −2.77 bps, and the frozen rule required it to be above zero. The study closed DID_NOT_MEET_CRITERIA. That outcome does not prove the effect is absent; the article discusses why the design may have had limited power.
Research 007 is a sample-size example. The source video’s Bitcoin walkthrough totals seven example trades, and the video itself says that is not enough. My frozen version produced 739 trades over a year, and all four frozen chronological folds were negative.
8. What can you check yourself?
- Can you get the code, the exact data and the expected outputs? Which of those are missing?
- If someone reproduced the result, did they reproduce a number or validate a strategy? Those are different claims.
- If “matching” allowed a tolerance, was it set before the comparison?
- Are hashes being used to show more than content identity, or self-recorded timestamps to show more than the order of steps?
Example. In Published Statistic Reproduction 001, I reproduced a published Bitcoin statistic within 5 binary64 ULP, inside a tolerance declared in advance. That covers one number on one fixed sample, with no bearing on whether anything is profitable to trade.
Benchmark 004 closed FULL_REPLICATION_MATCH: the notebook’s display metrics and input data were independently recoverable. I then tested the same executable strategy on a frozen 365-day holdout in Research 005, where it failed.
The evidence files on this site are selected artifacts. They allow partial inspection, not a complete independent rerun.
Recording what you find
Copy this block into your notes. Against each group, record its status (documented, unclear, unavailable, or not applicable with a reason), the answer, where you found it, and what is still open. For more room, open or save the Markdown worksheet, which lists every question separately.
Backtest under review:
Source and version:
Reviewed by / date:
1. What is being tested:
2. Data and completeness:
3. Look-ahead and timing:
4. Costs and execution:
5. Returns and benchmark:
6. Parameters and freeze:
7. Uncertainty and sample:
8. Reproducibility:
Open questions:
A first pass usually ends with a few unclear or unavailable rows. Those rows are the useful output: questions to send the author, or the parts of your own backtest to fix before you rely on it.
Going further
- Methodology describes how I separate reproduction from extension, how I freeze tests, and what readers can inspect.
- The research record lists every published evaluation and how each one ended.
- The engineering notes cover several of the examples above in more depth.
Explore the research record for more examples of published evaluations.