Eighty Random Trading Rules and One Fake Discovery
Why the best backtest you have ever seen may be pure luck, and the one question that exposes it

Here is an experiment you can run on a laptop in under a minute. Make up 80 trading rules that have no edge at all. Each one is just a coin flip: every day it is either long or short, and the daily result is pure noise with an average of zero. Run all 80 for one year of trading days and look at the results.
I ran it 2,000 times. In a typical run, about 13 of the 80 rules finished the year with a Sharpe ratio above 1, which most people would call a good strategy. The best rule had a median Sharpe ratio of about 2.4. In almost nine runs out of ten, the best rule beat a Sharpe ratio of 2.
None of those rules knew anything. They were noise, and the winner still looked like a discovery.
The problem is not the rule. It is the search.
That is the central idea of my new book, Real Edge or Illusion? How to Test the Claims of Trading. A backtest answers a narrow question: how would this rule have done on this data? It does not tell you how many other rules were tried before this one was chosen. If you test enough ideas, some of them will look excellent by luck alone, and the best-looking one is the one you will publish, sell or trade.
Statisticians call this the multiple-testing problem. In trading it goes by data-mining bias, data snooping or backtest overfitting. David Aronson's Evidence-Based Technical Analysis built a whole book around it. Academic finance has been fighting the same battle in public: when researchers counted how many "factors" had been proposed to explain stock returns, the count ran into the hundreds, and some argued that the usual bar for calling a result significant should be raised accordingly.
What a chart cannot show you
Every trading claim comes with a chart. A rising equity curve is persuasive because it compresses a story into one picture. But three things almost never appear on the chart, and each can change the answer:
How many versions were tried. A rule chosen from 80 candidates needs far stronger results than a rule tested once.
Costs. Commissions, spreads and slippage are small per trade and large per year. A strategy that trades every day can lose its whole edge to them.
What happened afterwards. Many published patterns weaken once they are known. The honest test is how the rule performs on data it has never seen.
A win rate on its own is also less informative than it looks. A system that wins 80 percent of its trades can still lose money if the losing 20 percent are large enough.
Four possible verdicts
The book does not argue that trading cannot work. It argues that claims should be tested before money goes behind them, and that the test should end in one of four honest verdicts: supported, conditional (it works only under stated conditions), contradicted, or inconclusive. Most claims, examined honestly, land in the last group. That is not a failure. Knowing that you do not know is cheaper than finding out with real money.
The final chapter turns all of this into a practical Edge Audit: a short list of questions you can apply to any system, course, signal service or fund track record. How many variants were tested? What were the costs? What happened out of sample? How large was the worst drawdown, and would you really have sat through it?
A small habit with a large effect
The next time you see a strategy with a beautiful equity curve, ask one question before anything else: how many other curves did they throw away to find this one? If nobody can answer, treat the chart as an advertisement, not as evidence.
Kris Trufev is the author of Real Edge or Illusion? How to Test the Claims of Trading. This article is education, not financial advice.
About the Creator
Enjoyed the story? Support the Creator.
Subscribe for free to receive all their stories in your feed.
Comments
There are no comments for this story
Be the first to respond and start the conversation.