Your Best Backtest May Be the One You Should Trust Least
A 2015 paper explains why optimisation creates convincing false edges, and what systematic traders should do about it
I recently revisited a 2015 paper titled The Probability of Backtest Overfitting.
The title sounds academic. Its message is brutally practical:
The more strategy variations you test, the easier it becomes to find an exceptional result with no genuine edge behind it.
This is not a new warning. Every systematic trader understands that overfitting exists. Yet most discussions stop at familiar advice: use fewer parameters, keep the rules simple, run an out-of-sample test, and be sceptical of perfect equity curves.
All of that is useful. None of it fully captures the real problem.
The paper asks a better question:
What is the probability that the strategy we selected was simply the luckiest version in the test?
That question changes how we should think about optimisation.
Optimisation is not inherently dangerous. The danger comes from searching hundreds or thousands of alternatives, selecting the winner, and treating its performance as though it came from a single independent test.
It did not.
It came from a competition. And every competition produces a winner, even when every contestant is noise.
The Optimisation Trap
Imagine testing one moving-average strategy with a single entry rule, exit rule, and stop. The result is poor, so you reject it.
Now imagine testing:
50 entry lengths
30 exit lengths
20 stop sizes
10 filters
Five markets
Three bar intervals
You now have millions of possible combinations.
Some of them will look exceptional. That outcome may be guaranteed by the size of the search, not by the existence of a real edge.
You may find a smooth equity curve, a high profit factor, a small drawdown, consistent annual returns, and a parameter set that appears nearly perfect.
It can feel like discovery. In reality, you may have found the combination that fitted historical noise most precisely.
That is why the best backtest can sometimes be the least reliable. It won because it remembered the past better than every other candidate—not because it is more likely to survive the future.
A Backtest Does Not Show What Came Before It
This is what most strategy reports leave out.
They show the final model, but rarely reveal how many ideas were tested before reaching it.
A published result might contain one strategy, four parameters, one equity curve, and one performance report. What it does not show is the research path behind it.
Perhaps the researcher tested 500 rule combinations. Perhaps they repeatedly changed the market, timeframe, session, indicator length, stop, or exit. Perhaps they reviewed an out-of-sample result, returned to development, and adjusted the strategy again.
By the time the final version is presented, the entire dataset may have influenced the finished model.
The backtest report does not record that history. It presents the final result as if it were the original hypothesis.
That is why the number of trials matters.
A profit factor of 2.0 found on the first test is not equivalent to a profit factor of 2.0 selected from 100,000 tests. The number is the same. The evidence is not.
Why One Out-of-Sample Period Is Not Enough
The standard response to overfitting is to reserve part of the data for out-of-sample testing. That is better than optimising on the full history, but it is not bulletproof.
Suppose we develop a model from 2000 to 2018 and reserve 2019 to 2022 as the out-of-sample period. The test fails.
What usually happens next?
We investigate. We change a filter, adjust the stop, or alter the holding period. Then we run the test again.
The moment the out-of-sample result influences the strategy, that period is no longer truly unseen. It has become part of development.
This does not require deliberate cheating. It is simply how research works: we learn from results, form a new hypothesis, and test again. But after enough iterations, we can overfit the out-of-sample window as well.
The paper argues that a single holdout can be unreliable for investment backtests. Financial data is limited, noisy, and heavily reused. One unusually favourable or unfavourable market period can distort the conclusion.
That does not make out-of-sample testing useless. It means a single split should not carry the entire burden of validation.
What the Paper Proposes
The authors propose a method called Combinatorially Symmetric Cross Validation. The name is more complicated than the idea.
Instead of selecting one fixed development period and one fixed test period, divide the return history into several sections. Then repeatedly:
Use one combination of sections as the development sample.
Select the best strategy configuration within that sample.
Test the selected configuration on the remaining sections.
Record where that supposed winner ranks out of sample.
Repeat across many combinations.
The important question is not simply, “Did the selected strategy remain profitable?”
The stronger question is:
Did the configuration that ranked best in sample continue to rank well out of sample?
If the in-sample winner repeatedly falls below the middle of the pack out of sample, the selection process is probably overfit.
This is the Probability of Backtest Overfitting, or PBO.
In plain English: how often does our optimisation process choose a winner that becomes average or poor when tested on unseen data?
That is a far more useful question than staring at the final equity curve.
The Problem Is Strategy Selection
The paper is not concerned only with models that contain too many fitted coefficients. It focuses on the process used to select a strategy.
That distinction matters.
You can overfit with a very simple strategy. Imagine testing 500 two-rule systems and publishing the strongest one. The final strategy may contain only two parameters, yet still be overfit because it was chosen from a large field of alternatives.
This is why “my model only has three rules” is not enough.
Better questions include:
How many variations were tested?
How many markets were searched?
How many exits and filters were compared?
Was the selected result stable across different samples?
Did neighbouring parameter values also work?
Did the idea transfer to another market?
Was the final test genuinely untouched?
Complexity is not limited to the finished code. It also exists in the path used to find it.
How This Fits Our Research Process
This paper reinforces why I do not treat optimisation as final proof.
Optimisation is useful because it helps us understand the shape of an edge. It can reveal whether a result is isolated, whether nearby values behave similarly, whether a strategy depends on one exact setting, and whether the market responds consistently to the underlying idea.
But I am not looking for the single highest net profit. I am looking for a stable area.
If a holding period of 17 bars is exceptional but 16 and 18 collapse, I am not interested. If holding periods from 12 to 25 bars all behave reasonably well, that is much stronger evidence.
That is why our validation process uses several different attacks:
Parameter sensitivity: We want a plateau, not one perfect point.
Parameter extremes: We move inputs away from the chosen values to see whether the strategy still has a pulse.
Randomised and skipped trades: We test whether the equity curve depends too heavily on a few perfectly timed trades.
Multiple out-of-sample windows: We do not want the conclusion to rest on one fortunate split.
Walk-forward testing: We repeatedly select parameters using earlier data and measure what happens next.
Strategy parameter permutation: We evaluate the broader parameter family rather than celebrating one winner.
Alternative markets: A different market is effectively unseen data. The exact result does not need to match, but the underlying behaviour should still make sense.
None of these tests proves that a strategy will work. Together, they make it much harder for a false edge to survive.
That is the point.
A Good Result Should Be Hard to Kill
Most researchers approach validation backwards. They build a strategy and then try to prove it works.
I prefer to assume it is wrong and try to break it.
Increase costs. Change the start date. Shift the parameters. Remove random trades. Test another market. Use a different out-of-sample period. Separate the long and short sides. Check whether one year generated most of the profit. Ask whether one parameter controls the entire result.
If the strategy dies immediately, good. We saved ourselves from trading it.
If the edge remains visible after repeated attacks, it starts to become interesting.
The objective is not to protect the backtest. The objective is to expose it.
The Most Dangerous Phrase in Strategy Research
The most dangerous phrase is:
“This was the best-performing setting.”
Best according to what? Over what period? Selected from how many alternatives? Was it also strong elsewhere? Did nearby values work? Did it remain above average out of sample?
Being the best setting in a historical sample is not automatically positive. Sometimes it is evidence that the value is unusually attached to that sample.
The further a result stands above every neighbouring alternative, the more suspicious I become.
Real market effects are usually messy. They tend to appear across ranges. Noise tends to produce sharp, impressive peaks.
That is why I would often rather trade the middle of a broad plateau than the top result from an optimisation report. I am intentionally giving up some historical performance in exchange for less dependence on one precise version of the past.
What I Would Change After Reading This Paper
The paper does not make me want to stop backtesting. It makes me want to keep better records of the research process.
For every strategy, I want to know:
How many meaningful variations were tested?
Which metric was used to select the model?
How stable were rankings across different samples?
Did the best in-sample configuration remain competitive out of sample?
How much performance degradation occurred?
What was the probability of an out-of-sample loss?
Would randomly selecting from the robust parameter region have produced a similar result?
That final question is particularly powerful.
If only one exact configuration works, we may have selected noise. If many reasonable configurations work, the precise choice matters less.
That is what robustness should look like.
The Practical Lesson
You do not need to implement the paper’s full mathematics tomorrow to benefit from it.
Start with four changes.
Track your trials. Record how many strategy variations you tested. Do not judge the final result without considering the size of the search.
Stop worshipping the winner. Inspect the entire parameter region. The highest result matters less than the stability around it.
Use more than one validation method. One out-of-sample period is not enough. Combine sensitivity analysis, multiple windows, walk-forward testing, randomisation, and alternative markets.
Freeze a final period. Once the strategy is complete, run it on a period that has never influenced development. Run it once. Do not return to the model simply because you dislike the answer.
That final step is difficult. It is also where many false edges are exposed.
The Real Edge Is Not the Backtest
A strong backtest is not the edge. A disciplined research process is the edge.
Anyone can optimise until an equity curve looks good. The hard part is knowing when the result deserves capital.
This 2015 paper gives a formal framework to something experienced system developers eventually learn: historical performance becomes less trustworthy as the number of opportunities to discover it increases.
The solution is not to avoid research. It is to make a strategy survive more than one version of history.
Do not ask whether the backtest looks convincing. Ask whether the selection process continues to find something useful when the data is rearranged, divided, and moved forward.
That is a higher standard. It should be.
Because the market will not give us the historical sample again.
TL;DR
The more strategy variations you test, the more likely you are to find an impressive result by chance.
The paper’s PBO framework asks whether the best in-sample configuration continues to rank well across many out-of-sample combinations.
The practical takeaway is simple: do not trust the winner of one optimisation. Track the number of trials, look for broad parameter stability, use several independent validation methods, and protect one final untouched test.
A backtest should not be easy to prove. It should be hard to kill.
Disclaimer
This publication is for educational and research purposes only and is not financial advice.
Backtested and hypothetical results do not guarantee future performance. Futures and other leveraged products involve substantial risk. Validate all ideas using your own data, assumptions, costs, and execution infrastructure before considering live deployment.