How the agent decides
A quick read on what the research agent measures, why a good backtest isn't enough, and how a winning configuration is actually determined. Every number referenced here is visible on the corresponding run — this page explains what they mean.
The research loop
You state a testable idea and the success criteria (Sharpe, win rate, drawdown, trade count, recent out-of-sample Sharpe). Goals are evaluated deterministically — the AI never grades its own homework.
The agent writes an experiment plan: which levers to test, in what order, and what would falsify the hypothesis.
Each round runs a real action — backtest, parameter sweep, walk-forward validation, or a config change — and is judged by deterministic quality checks (overfit signals, bankrupt parameter regions, degenerate sweeps).
Before concluding, the trades are audited (exit mix, R-multiples, monthly consistency) and the best config is scored against the deployment-readiness gates below.
The run ends with an evidence-cited verdict and machine-readable follow-up suggestions — each must cite a concrete artifact (a failed gate, a quality flag, a fold result). Campaigns chain these automatically, pausing for your approval.
How a winner is picked
One number answering "how deployable is this config?" — not just how profitable it looked.
A composite that starts from stability-adjusted risk-adjusted return, is rewarded by profit factor and reward:risk, and is reduced by penalties for boundary-hugging parameters, unprofitable months, and churn-heavy exit profiles. Exact weights are proprietary.
Selection is constraint-first (configs that fail hard constraints are out regardless of score) and high-water-marked: a later round can never silently replace a better config found earlier.
Why a good backtest isn't enough
A single backtest shows one path through one history — it rewards configurations tuned to that exact path. Every promising config is therefore re-tested with walk-forward validation (train on one window, test on unseen data, repeat across windows) and a Monte-Carlo bootstrap (thousands of reshuffles of the actual trades to estimate how the strategy fares when luck runs against it). Deployment decisions are made on the unlucky tail of those simulations, not the average.
The training data the strategy was developed on — like studying for an exam with the textbook. Good IS results are expected but don't prove the strategy works on new data.
Fresh data the strategy has never seen — the actual exam. If OOS results hold up, the strategy likely isn't overfit to historical quirks.
Instead of one IS/OOS split, the data is divided into 4+ overlapping time windows. Each window trains on IS and tests on OOS independently. If the strategy works across all windows, it's robust across different market conditions — not just lucky timing.
The actual trade sequence is randomly reshuffled 1,000 times to build confidence intervals. Even if average performance is good, Monte Carlo shows the realistic range of outcomes — including worst-case scenarios. A strategy where the 5th-percentile Sharpe is still positive is genuinely robust.
Measures return relative to risk. A Sharpe of 1.0 means you earn 1 unit of return per unit of risk. Above 1.5 is strong; above 2.0 is excellent. Below 0.5 is generally not worth the risk.
The largest peak-to-trough decline during the test period. If your account hit $10,000 then dropped to $7,500 before recovering, that's a 25% max drawdown. Lower is better — high drawdown means high pain.
OOS Sharpe divided by IS Sharpe. A ratio of 1.0 means the strategy performs equally well on unseen data. Above 0.7 is healthy. Below 0.5 suggests the strategy may be overfit to the training period.
The deployment-readiness gates
A config is deployableonly when every gate passes. Current thresholds and each run's actual values are shown on the run's Deployment Checklist and the campaign Gate matrix.
| Gate | What it protects you from |
|---|---|
Worst-case tail wf_mc_p5 | Even in the unlucky bottom slice of simulated futures, the strategy should still earn a solid risk-adjusted return. |
No boundary-hugging no_boundary_hits | The best parameters shouldn't sit at the edge of the range that was searched — that usually means the optimizer wanted to go further and the result is fragile. |
Profit factor profit_factor | Gross profits should meaningfully outweigh gross losses. |
Reward : risk effective_rr | The average winner should be meaningfully larger than the average loser. |
Exit mix trailing_exit_share | Wins shouldn't depend almost entirely on one exit type — trail-dominated profiles tend to churn and break in live conditions. |
Consistency months_profitable | Most calendar months should be profitable, not one lucky streak carrying the year. |
Even activity fold_trade_cv | Trading activity should be spread across the whole period — a strategy that only fires in one regime is regime-dependent. |
Same data, same config wf_hash_match | The walk-forward validation must have run on exactly the config and data being judged — no swapped-in results. |
Circuit-breaker headroom breaker_headroom | The strategy should stay clear of its own safety limits. If a backtest already hits the loss streak or drawdown that halts live trading, the bot will stop on its own — so the limits and the strategy are mismatched, and one of the two needs changing. |
Sweep objectives
Sweeps search for the most deployable config overall.
The sampler and the selector both optimize Deploy Score, so search matches selection.
For hardening: after the normal sweep, the top finalists are re-tested with real walk-forward validation and re-ranked by their worst-case tail — the winner is the config that holds up when unlucky, not the one with the prettiest average.
Two-stage: Deploy Score screen, then a walk-forward Monte-Carlo re-rank of finalists keyed on tail Sharpe. Used when the worst-case-tail gate is the blocker.
Run verdicts
The hypothesis held: goals met and the result survived validation.
The run improved the config but didn't fully meet the goals — the most common productive outcome; learnings compound into the strategy's ledger.
The hypothesis was contradicted by the evidence. Recorded so future runs don't retry it.
Not enough evidence either way — usually too few trades or noisy results.
The run hit its round or LLM-call budget before reaching a conclusion.
Live vs backtest
Once a strategy is live, each config it ran is replayed over the exact window it ran, on the bars and funding the bot saw, and every live trade is matched to its backtest twin. Results are compared as return on the trade's size, net of fees and funding.
Slippage is read per round trip. The bot signals on one market and fills on another, so a single fill can look better or worse than the backtest just from the gap between the two prices; that gap cancels between entry and exit, and what is left is the real cost of getting filled. Backtests now charge each venue its measured cost, so research results include it.
The bot trades its own strategy: live trades line up with the backtest's over the same window, on the same bars and funding, and the results agree once fees and funding are counted.
The bot is not trading its strategy faithfully — it took trades the backtest didn't, missed ones it did, or its per-trade results fall short by more than costs explain. An execution problem: look now.
Execution is faithful, but the strategy is losing here and its own backtest loses over the same window too. The strategy, not the bot — a candidate to park or re-research.
Not enough to judge: too few live trades under that config, a config whose start isn't on record, or market data the replay can't reconstruct. A state, not a fail.
Live trust
Usable in backtests, sweeps and research. Not deployable live: it has not passed the automated live checks, or it reads data that has no live feed yet.
Passes every automated live check — the same values bar for bar live and in the backtest, after a restart too, and inside the real live strategy. It has not yet earned a live track record, so its first deploy asks you to acknowledge it. Risk is never changed for you.
Has traded live long enough, in enough trades it actually decided, and the live-vs-backtest check found live faithful. Changing its code sends it back to Live-ready until it proves itself again.
- Tiers are earned from evidence, never set by hand, and each is tied to the exact code that earned it.
- A strategy is as trusted as its least-trusted part — including the data each part reads.
- Deploying a strategy with a Backtest-only part is refused. Deploying one with Live-ready parts needs your acknowledgement; the bot then shows as on probation until a live-vs-backtest check agrees.
Exact scoring weights, threshold calibration, and indicator internals are proprietary and intentionally not documented here.
Last reviewed: 2026-10-27