Lab Note #07 · All Lab Notes

Out-of-sample and forward testing

A test becomes more informative when the strategy did not get to choose its rules using the same outcomes it is being judged on.

Separate three different activities

In-sample development uses history to build and select a system. Historical out-of-sample evaluation applies frozen choices to a later interval. Live demo forward testing observes new quotes and operational behaviour as time passes. They answer related but different questions; a historical forward result is not a live trading record.

Draw the boundary before running the search

An illustrative research schedule could use 2020–2022 for development, 2023 for validation and 2024–2025 as a final holdout. These dates describe a possible protocol, not completed research. If you already inspected those later years to improve the strategy, they are no longer untouched for that decision process; use genuinely unseen evidence instead.

WindowPermitted use
DevelopmentDebug, explore and choose parameters under a recorded search plan.
ValidationEvaluate the frozen selection. If used to revise it, treat this period as development evidence thereafter.
Final holdoutEvaluate the completed procedure once, including all earlier selection decisions.

Use MetaTrader’s forward split

In Strategy Tester optimisation settings, choose a forward fraction or custom start date. The configured history is divided into an earlier optimisation portion and a later forward portion; selected optimisation passes are evaluated on the later portion. Keep the same cost and account assumptions, and inspect the saved inputs alongside both results. The exact controls are described in Strategy Optimisation.

This built-in split does not, by itself, run a whole rolling walk-forward programme. Also, choosing whichever candidate looks best in the forward results turns that interval into another selection sample. Reserve separate evidence for the final procedure.

Define what happens at the boundary

Moving averages need preceding history to initialise. Allowing older prices to warm an indicator is not the same as using future prices. Specify the warm-up period and exclude it from reported performance. Decide whether each test starts flat or carries an existing position, and record how trades spanning a boundary are treated.

For this teaching protocol, start each independent evaluation flat, initialise the indicators from preceding data, and wait for a fresh crossover. This may differ from continuous operation, so say so when comparing windows. Do not quietly credit a test with an entry that could only have occurred before its start.

Build a rolling study only with a fixed rule

A walk-forward study might train on three years, freeze the selected inputs, test the next six months, then advance six months and repeat. Set the schedule, objective and transition rules in advance. Report non-overlapping test segments together and retain each segment’s result so a good aggregate cannot hide repeated failures.

If you keep modifying that schedule after inspecting its test segments, you are fitting the research procedure too. Additional windows increase evidence, but they do not erase researcher discretion.

Finish with an honest decision

Compare trade frequency, exposure, drawdown and cost sensitivity as well as profit. Weak results can mean overfitting, a regime change or an implementation mismatch; investigate without rewriting the test into a success. A demo run can then assess restarts, missed signals, broker responses and logs. Demo fills still do not establish live execution quality. The final note turns those uncertainties into a deliberate failure-testing plan.

Building or reviewing your own trading system?

I can help turn the specification, implementation and testing into a coherent engineering process. Discuss your project.