Backtesting Without Fooling Yourself

The four ways a test lies, and how to catch each one.

advanced · 3 min read · 20 XP

A backtest is the cheapest way to reject a bad idea and the easiest way to convince yourself of a worthless one. This lesson is about the second problem.

What a backtest can and cannot tell you

It can tell you an idea is bad. If a strategy loses money on history, it is very unlikely to make money live.

It cannot tell you an idea is good. A profitable backtest is a necessary condition, not close to a sufficient one — because the number of ways to produce one accidentally is enormous.

Hold this asymmetry firmly

Backtesting is a filter, not a proof. Its job is to eliminate, not to validate. Treating a good result as evidence of an edge is the mistake that costs the most money in systematic trading.

Lie 1: curve-fitting

You test parameters, find the best-performing set, and report its results. That number is now meaningless — you selected for historical noise, and noise does not repeat.

The tell: performance that collapses when a parameter moves slightly. A genuine edge is a plateau, not a spike. If 18 and 22 both work and 20 works spectacularly, 20 is fitted.

The fix: out-of-sample testing. Hold back a chunk of data that you never look at while developing. Choose parameters on the in-sample period, then test once on the held-out data. Once — a second look makes it in-sample too.

Lie 2: lookahead bias

Using information the strategy could not have had at that moment. Subtle forms:

Using the day's high when the rule would execute intraday.
Using a closing price to enter at that close.
Using revised economic data — most releases are revised weeks later, and your backtest reads the final number.
Filtering by an outcome that had not happened yet.

The tell: results that look too clean. Equity curves that barely draw down usually contain lookahead.

Lie 3: ignoring costs

A strategy that averages 6 pips per trade before costs is a strategy that loses money after a 1.2-pip spread, commission, slippage and overnight swap.

Model all four. Slippage especially — backtests fill at exact prices, and reality does not, particularly around news and at the session open.

Lie 4: too few trades

Thirty trades tells you almost nothing. Random variation across thirty samples is large enough to make a coin flip look like an edge.

Aim for at least a few hundred trades across multiple market regimes: a trending period, a ranging period, a high-volatility period and a quiet one. A strategy tested only on 2023 has been tested on one regime.

My backtest shows 85% wins and a smooth equity curve. Good news?

Almost certainly bad news. That profile is far more consistent with lookahead bias or curve-fitting than with a real edge. Genuine edges are noisy, produce meaningful drawdowns, and win a lot less than 85% of the time. When a result looks too good, the correct first hypothesis is that the test is broken.

A workable sequence

Write the rules before touching the data.
Split the data: in-sample for development, out-of-sample untouched.
Test in-sample. Include full costs.
Check parameter stability — is it a plateau or a spike?
Test once out-of-sample.
If it survives, forward-test on a demo for a period. Live conditions surface problems no historical test can.
Go live at the smallest size that is meaningful, and scale slowly.

What to remember

A backtest can prove an idea bad; it cannot prove one good. It is a filter, not a validation.
Curve-fitting shows up as a spike rather than a plateau in parameter space — hold out data and test once.
Lookahead bias produces suspiciously clean equity curves; audit exactly what information each rule uses.
Model spread, commission, slippage and swap, and require hundreds of trades across several market regimes.

A backtest is a filter for rejecting bad ideas, not evidence that a good one works. The four ways it lies are curve-fitting, lookahead bias, ignored costs and too few trades — each with a specific tell and a specific defence, ending in out-of-sample and forward testing before any real money.

← All lessons