An AI trading bot, or any tool that summarises a strategy test in fluent sentences, can make every line sound equally certain. They are not. Some sentences describe what happened in the data, some estimate what is likely, and some assert things the system has no way of knowing. This article gives you a way to sort them, with examples of each.
Three kinds of sentence in a model's output
When you read a summary of a strategy test, almost every sentence falls into one of three groups.
- Explanation — a description of what the strategy did and why, traceable to the inputs and the rules. It can be checked.
- Probabilistic estimate — a statement about how often or how much, based on a sample. It can be useful, but it carries uncertainty that depends on how much evidence sits behind it.
- Unsupported claim — a statement about the future, about hidden causes, or about certainty that the inputs cannot back up.
The goal is not to discard everything except explanations. Estimates are often what you need. The goal is to give each sentence the weight it has actually earned.
Useful explanations: what can be traced
An explanation is useful when you could, in principle, reproduce it yourself from the run's inputs. Examples, using illustrative numbers from a moving-average crossover test:
- "A long position opened on bar 112 because the 20-bar average crossed above the 50-bar average."
- "Of the 22 trades, 13 were closed by the opposite crossover and 9 by the end of the sample."
- "Fees and slippage at 15 bps per side reduced the total return by 6.6 percentage points (22 round trips × 0.30%)."
- "The largest drawdown occurred during the volatility-shock segment, when four consecutive entries reversed within three bars."
Each of these points to a rule, a bar, a parameter or an arithmetic step. If you doubt one, you can open the trade list and check it. That is what makes it an explanation rather than a story.
The U.S. National Institute of Standards and Technology describes four principles for explainable AI. A system should give evidence or reasons for its output (Explanation), those reasons should be understandable to the person using them (Meaningful), they should correctly reflect how the output was produced (Explanation Accuracy), and the system should operate only within the conditions it was designed for and with sufficient confidence (Knowledge Limits). The last two are the ones to watch. A fluent sentence can be meaningful without being accurate.
Probabilistic estimates: useful, but read the sample size
Estimates are statements like:
- "41% of trades were profitable."
- "The average winning trade was about 1.8 times the size of the average losing trade."
- "In the range-bound scenario, the strategy had a lower win rate than in the steady trend."
These are fine as long as you read them with their evidence base. Take the first one. With 9 winning trades out of 22, the observed win rate is about 41%. But 22 trades is a small sample. A standard 95% interval for a proportion (the Wilson interval) at 9 out of 22 runs from roughly 23% to 61%. In plain terms, the data is consistent with a strategy that wins less than a quarter of the time and one that wins more than half the time.
A good estimate sentence tells you the sample behind it. A weaker one rounds the number and drops the context. When the sample is missing, add it yourself before you rely on the figure.
Unsupported claims: what no model can know
Unsupported claims are the sentences to catch. They usually take one of four forms:
- Forecasts — "This strategy should return around 18% next year." Nothing in a historical or simulated test measures next year.
- Hidden causes — "The model detected institutional accumulation." Unless institutional order data was an input, the system cannot have detected it. It has pattern-matched a phrase.
- Certainty — "This setup is highly reliable" or "can't lose." Reliability is a property of a sample, and the sample is always finite.
- Universality — "This works in all market conditions." A test covers the conditions it was run on, and no others.
Regulators flag the same pattern from the fraud side. The joint SEC, NASAA and FINRA investor alert lists promoters' claims such as "Our proprietary AI trading system can't lose!" as warning signs, and cautions that AI-generated information can rest on inaccurate, incomplete or outdated data, and can be faulty or even made up. The CFTC's advisory on AI trading bots states directly that AI technology cannot predict the future or sudden market changes.
Language models are particularly prone to the second form. They produce sentences that sound like market commentary because they have read a great deal of market commentary, not because they have checked the cause against data.
A quick sorting table
Here is the same strategy test described in six sentences, sorted. The numbers are illustrative.
| Sentence | Type | What to ask before trusting it |
|---|---|---|
| "Entries fired on 22 crossovers of the 20- and 50-bar averages." | Explanation | Does the trade list show 22 entries? |
| "Costs reduced the return by 6.6 points." | Explanation | Do 22 round trips × 0.30% match? |
| "41% of trades were profitable." | Estimate | How many trades? What is the range? |
| "Win rate was lower in the range-bound scenario." | Estimate | How many trades in that scenario alone? |
| "The strategy captures momentum driven by large buyers." | Unsupported claim | Was buyer data an input? If not, drop it. |
| "Expect similar results going forward." | Unsupported claim | No test measures the future. Drop it. |
How to read a model explanation in practice
A short routine works for both rule-based and AI-written summaries:
- Find the inputs. Which scenario, template, parameters, fees, slippage and position size produced this run? If you cannot see them, you cannot judge any sentence.
- Mark each sentence. E for explanation, P for probabilistic estimate, U for unsupported. Do it in the margin or in your notes.
- Check one explanation. Pick a single E sentence and verify it against the trade list or the metrics. If it fails, lower your trust in the whole summary.
- Attach sample sizes to every P. If the number of trades or events is missing, find it.
- Remove every U. Do not carry it into your conclusion or your research notes.
- Write your own one-line conclusion using only E and P sentences.
Why rule-based summaries are the default in Verdaxis
A rule-based summary can only say what its rules allow: it compares metrics against thresholds documented in the methodology and states the result. That makes it less fluent, and also less able to drift into unsupported claims. When an AI summary is enabled through a configured provider, it is labelled so you know which kind of text you are reading, and the same sorting routine applies.
The underlying habit matters more than the tool. A summary is only as good as the hypothesis behind it, so it helps to start with one that defines what success and failure mean; see from trading idea to testable hypothesis. And because many estimates change with the market regime, why market context changes the meaning of a signal is a useful next read.