From break-tests to an audit checklist
Lab Note #08 tried to break a system we wrote ourselves. Reviewing someone else's system needs the same experiments, but asked in a fixed order, with the evidence for each answer written down.
When we built the moving-average system in this series, we knew every assumption because we made it. An audit starts from the opposite position: source code of unknown quality, a specification that may exist only in the author's head, and a backtest report that was chosen because it looked good. A checklist does not make the reviewer cleverer. Its job is to make omissions visible, so that "we did not check that" becomes a recorded finding rather than a silent gap.
Order is part of the method
The checks run in the same sequence every time, because an early failure changes what later results can mean. If the code does not do what the specification says, the backtest measured something other than the system being described. If the test used intrabar data the live system will never see, no amount of walk-forward analysis rescues the result. Reading the equity curve first, and then looking for reasons to believe it, is the habit the order is designed to prevent.
Each item records four things: the question, the evidence requested, how the evidence was checked, and the finding. A finding is one of pass, concern, fail or not assessable. The last one matters. If the owner cannot supply source code or a fold log, the honest output is that the question could not be answered, stated plainly, not a pass by default.
The checklist
| Area | Question | Evidence requested |
|---|---|---|
| 1. Specification and ownership | Is every rule written down, and does the system know which positions are its own? | Written specification, source code, magic-number and restart logic |
| 2. Look-ahead and data | Could any decision use information that was not available at that moment? | Signal timing, indicator buffer indices, data source and timezone |
| 3. Curve-fit exposure | Is the reported result typical of its neighbourhood, or the best cell of a large search? | The full optimisation grid, not only the chosen settings |
| 4. Execution realism | Do costs and fills in the test resemble what the account will experience? | Spread, commission, slippage and swap assumptions; tester model |
| 5. Out-of-sample evidence | Was the system evaluated on data that played no part in choosing it? | Walk-forward fold log, split dates, the rules used to pick parameters |
| 6. Live versus tested behaviour | Where the system has run on a demo or live account, does that match the test? | Terminal logs, trade history, error records |
1. Specification and ownership
This is the work of Lab Note #02 in reverse. Instead of writing a specification, the reviewer reconstructs one from whatever exists and lists every place where the code makes a decision the specification does not mention: what happens on a gap, at session close, when volume rounds below the broker minimum. Then the ownership questions from #08: after a restart with a position open, does the system rebuild its state from the account before acting; after a rejected close, does it refuse to open the reverse position?
2. Look-ahead and data
Most look-ahead in MetaTrader systems is ordinary, not exotic: reading index 0 of an indicator buffer when the rule assumes a closed bar, or evaluating on every tick in a test and on bar open in the live terminal. The check is mechanical. Trace each signal to the bar it reads, and confirm the tester model and the live execution path see the same thing. Then confirm the test data came from a source whose prices, sessions and timezone match the account the system is meant for.
3. Curve-fit exposure
Lab Note #06 showed how a search over parameters can manufacture an attractive result from noise. The audit question is how much searching produced the result being reported. That needs the whole grid, not the winning row. Two things are read from it. First, whether the reported settings sit on a plateau, where neighbouring values behave similarly, or on a spike surrounded by much weaker results. Second, how far the chosen combination sits above the average of everything that was tried. A large gap between the selection and the grid it came from is evidence of selection, not of behaviour.
4. Execution realism
The cost experiments from #08 become a standing check: rerun at the documented baseline and at stressed cost assumptions, count both legs of every reversal, and convert pips using the contract specification of the instrument actually traded. The finding is not a single number. It is whether the conclusion survives a reasonable range of cost assumptions, and at what point it stops surviving.
5. Out-of-sample evidence
Lab Note #07 described walk-forward testing. Reviewing someone else's walk-forward study comes down to five questions, each answerable from the fold log:
- Sample sufficiency: how many folds, and how many trades in each? A fold with a dozen trades says very little on its own.
- Consistency: what share of folds were profitable, and how far apart are the mean and the median fold result? A mean well above the median usually means one or two folds carry the total.
- Selection inflation: in each fold, how far did the chosen parameters sit above the average of the whole grid on the training window?
- Parameter dispersion: do the chosen parameters stay in one region from fold to fold, or jump across the grid?
- Degradation: how much weaker is out-of-sample behaviour than in-sample, and is the gap stable?
These questions are mechanical enough to automate. We use a small standard-library Python script that reads a fold log, prints each judgement as a plain sentence, and collapses them into one of three verdicts: the evidence supports the reported result, it is too weak to rely on, or it does not support it. The thresholds are named and argued, not hidden, because a reviewer's thresholds should themselves be open to review.
6. Live versus tested behaviour
If the system has run anywhere other than the tester, compare the two records directly: fill prices against the bar the test used, errors and retries in the terminal log, and trades that appear in one record but not the other. Differences are expected. The finding is whether each difference has an explanation, or whether the system has simply been allowed to drift away from what was tested.
Findings are ranked, not listed
Every concern or failure is given a severity. Blocking means the results cannot be relied on until it is fixed, which covers most specification, ownership and look-ahead failures. Material means it changes how much confidence the evidence deserves. Minor means it should be fixed but does not change the conclusion. The fix list is ordered by severity first and then by cost, so the owner knows which single change matters most.
The output is a decision about evidence
The review ends where #08 ended: reject, revise and retest, or proceed to controlled demo observation, with the reasons written down. That decision is about whether the evidence supports the claims made for the system. It is not a recommendation to trade or not trade it, and it says nothing about future returns. A system can pass every item on this list and still lose money, because the list tests engineering and evidence, not markets.
Where this sits in the series
Lab Notes #01 to #08 built and tested a deliberately boring system so that every step could be inspected. This note turns those steps into the list applied to any system. It is the same list behind the System Audit: Tier A works through the specification and ownership questions from written material, Tier B covers all six areas against the code and test records, and Tier C adds a re-specification and rebuild plan where the findings call for one. No tests or performance outcomes are claimed in this article.
Have a system you want checked against this list?
The System Audit page sets out what each tier covers and what to send. Or describe your project.
