Lab Note #05 · All Lab Notes

How to read a backtest without fooling yourself

Net profit is the beginning of the review. We also need to know how the system earned it, what exposure it carried, and which assumptions made it possible.

Check the experiment before the score

Read the dates, instrument, deposit, volume, modelling settings and costs before the headline return. Compare runs on like-for-like terms. Doubling fixed volume can increase monetary profit without improving the strategy. A different deposit can change percentage drawdown and margin outcomes even when the signal rules stay identical.

Use a small set of complementary measures

MeasureQuestion to ask
Net profitWhat remains after the costs actually represented?
Trades and dealsHow many completed opportunities are represented? Deals are execution events, not necessarily independent trades.
Equity drawdownHow far did account value fall from a prior peak, including open-position losses?
Profit factorHow large is gross profit relative to absolute gross loss?
Recovery factorHow much net profit was earned relative to the report’s maximum drawdown denominator?
Sharpe ratioHow large are excess returns relative to their variability, under the stated calculation convention?

MetaTrader defines its fields in the Testing Report reference. Distinguish balance from equity drawdown, and monetary maxima from percentage maxima: they can occur at different times. When reproducing a ratio externally, document the exact denominator, return frequency and annualisation convention.

A worked example, not a strategy result

Suppose an illustrative report has gross profit of $6,000, gross loss of $4,000, and net profit of $2,000, with no additional charges outside those figures. Profit factor is 6,000 / 4,000 = 1.5. If the relevant maximum drawdown is $2,500, recovery factor is 2,000 / 2,500 = 0.8. These invented numbers explain arithmetic; they are not results from our crossover system.

Neither ratio reveals whether almost all the profit came from one trend. Nor does either reveal how long the account remained below its previous high. Inspect the equity curve, the underwater periods, and the individual trades alongside the summary.

Ask where the profit came from

Break the development sample into calendar periods and long versus short positions. Check how much total profit depends on the largest winners. Removing a few winning trades is a useful concentration diagnostic, not a justification for excluding inconvenient observations from the reported record.

Win rate alone is particularly weak evidence. A strategy can win frequently and lose heavily on rare trades. For this baseline, the absence of a stop means a smooth closed-trade balance curve can conceal substantial floating losses. Equity and margin usage deserve close attention.

Count information, not just rows

A hundred trades clustered in one market regime do not necessarily provide a hundred independent observations. There is no universal trade count that makes a strategy reliable. Consider the number of market conditions observed, dependence between trades and the uncertainty around average outcomes. Similarly, a high Sharpe estimate over a short or highly selected interval should trigger questions about the sample and selection process.

Write a review decision

End the review with a concrete finding: implementation issue, insufficient evidence, rejection, or candidate for further validation. Record the reason and the next test. A candidate is not a deployment approval. In the next note, we deliberately search many parameter combinations to see why the best historical score is such a tempting trap.

Building or reviewing your own trading system?

I can help turn the specification, implementation and testing into a coherent engineering process. Discuss your project.