After ten winning trades, it's tempting to believe you've found something. You almost certainly haven't — ten trades can be pure luck. Telling a genuine edge from a lucky run is fundamentally a question of sample size and statistics, and getting it wrong is exactly how traders end up betting real money on noise. Understanding statistical significance is what separates evidence-based trading from wishful pattern-spotting. This guide explains it: why small samples prove nothing, how many trades you need, and why testing many strategies inflates false positives.
It's the rigorous companion to variance and luck, a defence against overfitting, and underpins honest walk-forward validation.
Key takeaways
Q: Why does a small number of trades prove nothing?
A: Because randomness easily produces winning (or losing) streaks over a small sample. With only a handful of trades, a profitable result can be pure luck — the same way a few coin flips can come up heads several times in a row without the coin being biased. A genuine edge only reveals itself over a large number of trades, where luck averages out and skill (if any) shows through. Concluding anything from ten or twenty trades is statistically meaningless.
Q: How many trades do you need to trust a result?
A: There's no single magic number — it depends on the size of the edge and the variability of returns — but the honest answer is 'many more than feels necessary,' typically at least a few hundred trades, and more if the edge is small or returns are volatile. A small edge buried in lots of noise needs a very large sample to detect reliably. As a rule of thumb, be deeply sceptical of any conclusion drawn from fewer than 100 trades, and prefer hundreds.
Q: What is the multiple-testing problem in trading?
A: If you test many strategies or variations, some will look profitable purely by chance — even if none has a real edge. Test enough random ideas and a few will produce great-looking results by luck alone, which you might then mistake for a discovery. This 'multiple-testing' or 'data-mining' problem inflates false positives, and it's why a strategy that emerged from trying hundreds of combinations deserves extra scepticism and out-of-sample validation.
Why small samples lie
A small number of trades proves nothing because randomness easily produces winning (or losing) streaks over a small sample. With only a handful of trades, a profitable result can be pure luck — in exactly the same way a few coin flips can come up heads several times in a row without the coin being biased. Flip a fair coin ten times and getting seven or eight heads is unremarkable; nobody would conclude the coin is "lucky." Yet traders routinely take ten profitable trades as proof their strategy works — making precisely the error they'd laugh at with the coin. The uncomfortable truth is that over a small sample, a strategy with no edge at all can easily look brilliant, and a genuinely good strategy can easily look terrible — because variance (luck) dominates short runs and swamps whatever real signal might exist. A genuine edge only reveals itself over a large number of trades, where luck averages out (the wins and losses you'd get by chance roughly cancel) and skill, if any, shows through. This is the law of large numbers in action, and ignoring it is one of the most expensive mistakes in trading: people add risk, increase size, or quit a good system based on tiny samples that contain no real information. Concluding anything — good or bad — from ten or twenty trades is, statistically, meaningless, no matter how convincing the run feels.
How many trades, and the multiple-testing trap
So how many trades do you need to trust a result? There's no single magic number — it genuinely depends on the size of the edge and the variability of returns — but the honest answer is "many more than feels necessary." A small edge buried in lots of noise needs a very large sample to detect reliably (the smaller the edge and the more volatile the returns, the more trades required); a large, consistent edge reveals itself sooner. As a practical rule of thumb: be deeply sceptical of any conclusion drawn from fewer than ~100 trades, and prefer several hundred or more before placing real confidence in a result — and recognise that even then, you're estimating, not proving. The intuition to carry: ask not just "did it make money?" but "over how many trades, and could that result plausibly be luck?" — and with few trades, the answer to the second question is almost always "yes, easily." Tools like Monte Carlo simulation help here by showing the range of outcomes luck alone could produce, making it concrete how much of a small-sample result might be noise.
There's a second, subtler trap that catches even careful traders: the multiple-testing problem (also called data-mining bias). If you test many strategies or variations, some will look profitable purely by chance — even if none has a real edge. This follows directly from the small-sample logic: each test has some probability of looking good by luck, so test enough random ideas, indicators, parameter combinations or pairs, and a few will produce great-looking results by luck alone — which you might then mistake for a discovery and trade. It's the statistical equivalent of flipping thousands of coins and then being amazed that one came up heads ten times in a row — that coin isn't special; you just looked at enough of them. This is especially dangerous in modern trading, where it's trivial to backtest hundreds of variations and cherry-pick the winner, which is almost guaranteed to be partly (or wholly) a fluke. The defences are exactly the rigorous-validation tools: a strategy that emerged from trying many combinations deserves extra scepticism and, above all, out-of-sample validation (walk-forward analysis) on data that played no part in the search — the only real way to check whether the "discovery" survives outside the data it was mined from. The overarching discipline: treat every promising result as guilty until proven innocent, ask how large the sample is and how many things you tested to find it, and demand out-of-sample confirmation before trusting it. Respecting statistical significance is what stops you from confidently risking money on what is, in truth, noise dressed up as signal. The honest framing: a handful of trades proves nothing — randomness produces streaks, so over a small sample a no-edge strategy can look brilliant and a good one terrible, because variance dominates; a real edge only emerges over a large sample (be sceptical below ~100 trades, prefer hundreds). And testing many strategies inflates false positives — some will look great by pure chance — so a result found by trying many combinations needs extra scepticism and out-of-sample validation. Always ask: over how many trades, how many things did I test, and could this be luck?
Thinking in confidence, not certainty
The deeper mindset behind statistical significance is to think in confidence rather than certainty. Any result you measure — a win rate, an expectancy, a profit factor — is an estimate with error bars around it, and those error bars shrink as the sample grows. With few trades the error bars are enormous (your "55% win rate" might really be anywhere from 35% to 75%); with many trades they tighten until you can be reasonably confident the measured edge is close to the true one. You don't need formal hypothesis tests and p-values to internalise this — though they formalise exactly it — you need only the habit of asking "how sure can I actually be, given this sample?" and answering honestly (usually: "less sure than I'd like"). This humility is the opposite of the typical trader's overconfidence after a good run.
A particular hazard to name is p-hacking (or data-dredging): consciously or not, searching through data, timeframes, indicators and parameters until you find a combination that looks "significant," then presenting that cherry-picked result as a discovery. Because you tried many things, finding one that looks good is expected by chance (the multiple-testing problem again) — so a result that emerged from extensive searching is far weaker evidence than one from a single, pre-specified test. The practical takeaways that follow from all this are concrete: track everything (so you have real samples to assess, not impressions); demand large samples before trusting any conclusion; insist on out-of-sample confirmation (walk-forward) for anything found by searching; prefer simple, robust edges over intricate ones (fewer parameters = less room to fool yourself); and treat live results as the ultimate test (real trading is the cleanest out-of-sample data there is). Above all, hold every edge with appropriate humility — confident enough to act when the evidence is strong, sceptical enough never to mistake a lucky run or a mined result for proof. That calibrated uncertainty, applied consistently, is what separates durable, evidence-based trading from confident gambling on noise. The honest reminder: think in confidence, not certainty — every measured result is an estimate with error bars that shrink only as the sample grows, so always ask honestly how sure you can be given the sample (usually less than you'd like); beware p-hacking (searching until something looks significant is expected by chance), and in practice track everything, demand large samples, insist on out-of-sample confirmation, prefer simple robust edges, treat live results as the real test, and hold every edge with calibrated humility.
A handful of trades proves nothing — randomness produces streaks, so over a small sample a no-edge strategy can look brilliant and a good one terrible, because variance (luck) dominates short runs (like a few coin flips). A real edge only emerges over a large sample as luck averages out — be deeply sceptical below ~100 trades, prefer several hundred+, and more if the edge is small or returns volatile. Beware the multiple-testing trap: test many strategies and some will look great by pure chance, so a result found by trying many combinations needs extra scepticism and out-of-sample validation. Always ask: over how many trades, how many things did I test, and could this be luck? — treat every promising result as guilty until proven innocent.
