Most trading ideas arrive as a feeling: "breakouts tend to keep going", "this pulls back after big days". The trouble is not that these ideas are wrong. It is that, as stated, they cannot be wrong, and a statement that no result can contradict cannot be tested. This article walks through how to rewrite an idea so a test can reject it, using one example from start to finish.
Why a vague idea cannot fail
The philosopher Karl Popper made falsifiability his test for separating scientific claims from non-scientific ones: a claim earns its place when some possible observation would contradict it. A theory that stays compatible with every outcome, often because it keeps being adjusted to fit, tells you nothing.
Trading ideas fall into this trap easily. "Breakouts tend to keep going" survives any result. If a breakout fails, it was a "false breakout". If it works after three weeks, the idea was "early". Nothing has been measured, so nothing has been learned. The same holds when the idea comes from a chart pattern, a forum post or an ai trading bot: a suggestion stays an idea until it has a condition and a way to fail.
A testable hypothesis fixes five things before you see any results:
- Condition — the exact event that triggers the idea.
- Inputs — the data, parameters and costs the test uses.
- Horizon — how long after the condition you measure.
- Evaluation — the metric and the comparison that counts as support.
- Failure criteria — the results that would make you drop or revise the idea.
A worked example: rewriting "breakouts keep going"
Here is the same idea rewritten in four passes. Every number below is illustrative, chosen to show the method, not taken from market data.
Pass 1: name the condition
"Breakout" has to become something a computer could detect without judgement:
That one sentence removes several hidden choices: intraday highs versus closes, the lookback length, and whether "near the high" counts. If you later want to test a 50-bar lookback, that is a separate hypothesis, recorded separately.
Pass 2: state the expected outcome and horizon
"Keeps going" becomes a direction, a size and a time window:
The comparison matters. A rising sample makes almost any long signal look good. Comparing against the unconditional 10-bar return asks whether the condition adds anything beyond the drift already in the data.
Pass 3: add costs and size
A difference that disappears after costs is not a result. Suppose you assume 10 basis points (bps) of fees and 5 bps of slippage on each side of a trade. A round trip then costs 2 × (10 + 5) = 30 bps, or 0.30%. The hypothesis has to clear that:
Fixing the position size keeps the test about the signal, not about leverage.
Pass 4: write the failure criteria
This is the step most people skip, and the one that makes the hypothesis honest. Before running anything, write down what would count against the idea:
Criterion (c) is a check on whether the idea depends on one kind of market. If it only works in a steady uptrend, that is useful knowledge, but it is a narrower claim than "breakouts keep going".
The hypothesis template
The four passes collapse into a template you can reuse for any idea. The right-hand column shows the completed breakout example.
| Field | Question it answers | Breakout example (illustrative) |
|---|---|---|
| Idea | What do you believe, in plain words? | Breakouts tend to continue |
| Condition | What exact event triggers it? | Close above the highest close of the prior 20 bars |
| Inputs | Which data, parameters and costs? | Sample scenario data; 20-bar lookback; 10 bps fees + 5 bps slippage per side; 10% position size |
| Horizon | When do you measure the outcome? | 10 bars after the condition |
| Expected outcome | What should happen if you are right? | Net excess 10-bar return above 0.30 points |
| Comparison | Compared with what? | Unconditional 10-bar return in the same sample |
| Minimum evidence | How many events before you judge? | At least 30 occurrences |
| Failure criteria | What result makes you drop or revise it? | Net excess ≤ 0.30 points, fewer than 30 events, or positive in only one of four scenarios |
| Decision rule | What will you do with each outcome? | Pass: test a second lookback as a new note. Fail: record and archive |
Run it, then read the result against what you wrote
With the template filled in, running the test is the easy part. The discipline is in how you read it.
Suppose the illustrative results come back like this: the condition fired 41 times; net excess return was +0.52 points in a steady trend, −0.18 in a range-bound market, −0.95 in a volatility shock and +0.07 in drawdown and recovery. Against the failure criteria, the hypothesis is rejected under (c). The evidence supports a narrower statement: "In a steady trend, this breakout rule beat the unconditional return after costs." That is a smaller claim, and a more useful one, because you now know where it stops working.
What you should not do is tune the lookback until all four scenarios turn positive. Research on backtest overfitting by Bailey, Borwein, López de Prado and Zhu explains why: when many configurations are tried on the same data and the best is kept, the chosen one often looks much better in the test than it will anywhere else. Each new parameter set is a new hypothesis, and it needs its own note.
Plan first, then look
The Center for Open Science describes preregistration as writing down your research plan before the study, and gives the reason plainly: the same data cannot be used both to generate a hypothesis and to test it. Self-directed traders do not need a formal registry, but the principle carries over directly. Write the template, date it, and only then press run.
A practical order of work:
- Write the idea in one plain sentence.
- Fill in condition, inputs and horizon.
- Calculate the round-trip cost and set the expected outcome above it.
- Write the failure criteria and minimum evidence.
- Run the test across every available scenario, not just the one you expect to favour the idea.
- Compare the results with what you wrote, and record the decision.
Limits of testing on sample scenarios
A hypothesis tested on synthetic scenarios tells you how a rule behaves under a defined market structure: a steady trend, a range, a volatility shock, a drawdown and recovery. That is valuable for understanding mechanics and for finding where a rule breaks. It does not tell you what a live market will do next, and a pass is not a forecast.
Treat each tested hypothesis as one dated entry in an ongoing record. The next step is keeping that record in a form you can come back to, which is the subject of building a research journal for strategy experiments. And if your results differ sharply between scenarios, why market context changes the meaning of a signal explains what that difference can tell you.