How to evaluate backtest results before letting an AI agent trade real money

Backtesting an AI trading strategy means simulating how that strategy would have behaved using historical market data. It does not predict future profits, but reveals whether the agent's rules for entry, sizing, and exit produce mechanically consistent results over past conditions. A backtest is best understood as a diagnostic tool that surfaces logical flaws and hidden assumptions, not as a promise of returns. You should evaluate it by asking whether the simulation is honest about costs, constraints, and the limits of history.

What does a backtest actually measure?

A backtest measures the internal consistency of an agent's decision rules when applied to past prices, volumes, and other available data. It answers whether the logic is coherent and whether the agent would have followed its own instructions without contradiction. It does not answer whether the future will resemble the past. Many traders fixate on the final equity curve or a single Sharpe ratio, but these are just summaries. The real value lies in the path taken to reach those numbers. Look at the distribution of daily returns. Are they clustered around a small positive mean, or do they depend on a handful of outliers that skew the average? A strategy that makes most of its money from three unusually large trades is fragile, even if the annualized return looks attractive. Examine the depth and duration of drawdowns. A twenty percent decline that recovers in forty-eight hours is different from one that lingers for six months. The former suggests a volatile but bounded strategy. The latter suggests a strategy that may be structurally wrong during certain regimes. Check how often the agent was exposed to the market. A strategy that is flat most of the time and only enters during specific setups will have a different risk profile from one that is always carrying risk. You should also look at the sequence of returns. Consecutive losses tell you about the agent's behavior under stress. If the strategy assumes it can always get the next trade right, a string of losses may cause it to double down, freeze, or abandon its rules entirely. The backtest should reveal that pattern clearly. Separate the signal from the market regime. If the historical period was dominated by a single macro trend, the backtest may be measuring the trend, not the agent. Split the data into chunks, such as bull markets, bear markets, and sideways periods, and evaluate each one independently. A robust strategy should not collapse when the calendar changes. Finally, compare the agent's returns to a simple benchmark. If the agent underperforms a basic buy-and-hold portfolio after fees and stress, the complexity may not be justified.

How should you check for overfitting?

Overfitting happens when a strategy is optimized to capture noise rather than signal. With an AI agent, the risk is subtle and can hide in multiple layers. The model may learn spurious correlations in the training data, the prompt may embed implicit assumptions about how prices move, or the feature engineering may cherry-pick indicators that happened to align during the backtest period. To check for this, start with sensitivity analysis. Take the core parameters, whether they are thresholds, lookback windows, or model weights, and perturb them by ten or twenty percent. If the backtested returns fall apart, the edge was never stable. It was a tight fit to one historical path. Next, enforce strict out-of-sample testing. Reserve the most recent twenty to thirty percent of your data and never optimize against it. Train or tune the agent on the older slice, then run it exactly once on the reserved slice. If performance drops sharply, the agent memorized the training era rather than generalizing. Walk-forward testing adds another layer of rigor. Divide the history into sequential windows of equal length. Optimize on window one, test on window two. Then roll forward, optimize on window two, and test on window three. Repeat this process across the entire dataset. This mimics the reality of live trading, where the agent must adapt to new information without the benefit of hindsight. You should also inspect the trade list manually. Does the agent enter at suspiciously perfect local minima or exit at exact peaks? That is a sign of lookahead bias or data snooping. Make sure the backtest engine uses only information that would have been available at the exact timestamp of the trade, including any lag for data processing. If the agent uses dollar-based order sizing, ensure the backtest applies the same normalization logic that the live API will use. Dollar-based order sizing has its own risks if the simulation ignores rounding and minimum sizes. Be wary of strategies that look perfect. The goal is graceful degradation, not perfection. A slight decline in out-of-sample performance is normal. A cliff is a warning.

Why do execution assumptions matter more than returns?

The most dangerous backtest is one that assumes perfect execution. In reality, an AI agent generates a signal, sends an order, and the market responds. The price may move between signal generation and fill. This slippage is especially severe in fast or thin markets. A backtest that fills at the closing price of the signal candle ignores the bid-ask spread, queue position, and the possibility of partial fills. You must model these costs explicitly and conservatively. Start with the minimum realistic slippage per market type. Stocks, crypto, perps, options, and prediction markets all have different liquidity profiles. An agent trading across all five through one API still faces five distinct microstructures. Letting an agent trade every market through one API simplifies integration but does not homogenize liquidity. The API normalizes order sizing into plain US dollars, but the venue translates those dollars into contracts, lots, or shares. The backtest must replicate that translation, including rounding, minimum sizes, and margin requirements. If the agent targets one thousand dollars of exposure but the venue only fills eight hundred because of lot size constraints, the backtest should record the shortfall and the remaining uninvested cash. Latency matters too. An agent reacting to a signal may see the opportunity disappear before the order reaches the market. This is not a bug. It is a cost of doing business. If your backtest fills instantly at the signal price, you are measuring a theoretical agent, not the one you will deploy. You should also account for all fees. Venue fees, funding rates for perpetual futures, premium decay for options, and wide spreads in prediction markets all erode the edge. A strategy that shows a thin margin in a frictionless backtest will likely lose money once these costs are applied. Consider market impact as well. If the agent's size is large relative to average volume, its own orders may move the price. Even a small impact of a few basis points per trade compounds quickly over hundreds of trades. The return number itself is secondary. What matters is whether the edge survives after realistic execution assumptions are subtracted. If it does not, the strategy is not broken. The backtest was simply lying.

How do guardrails change the backtest?

A live Felix agent does not operate in an unconstrained environment. It trades within a hard budget, position limits, and under a kill switch that the owner can trigger at any time. How an agent trades within a hard budget describes the mechanics of these limits. These guardrails change the shape of the backtest dramatically. If your simulation lets the agent exceed its budget, accumulate unlimited positions, or ignore a stop loss, then the backtest is not testing the system you will actually run. You must code the same constraints into the simulation from the very first trade. The budget cap means the agent cannot average down forever. If the strategy relies on adding to losers until they turn, the backtest should show the exact point where the budget halts further entries. Without this cap, the simulation may show a miraculous recovery that the live agent is prevented from reaching. Position limits prevent concentration. A backtest that lets the agent put ninety percent of capital into one prediction market is not valid if the live guardrail caps any single position at ten percent. The difference in risk is enormous. Exit plans are equally critical. Automated stop losses, take profits, and time-based exits are part of the agent's logic. The backtest must apply them precisely. If the live agent will flatten at a five percent drawdown, the backtest should record the flattening, not a continued ride to a twenty percent recovery that never happens in reality. The kill switch is the ultimate guardrail. In a live environment, it revokes the agent's scoped key and flattens all positions. Your backtest should simulate this scenario. What happens if the switch is triggered mid-trade? Does the agent exit cleanly or leave orphaned orders that drift without oversight? Non-custodial infrastructure protects against theft and unauthorized withdrawals, but it does not protect against the agent losing its entire scoped budget. How self-custody works for algorithmic traders explains the relationship between owner-controlled wallets and agent permissions. The backtest should quantify that worst-case loss and confirm it matches the owner's risk tolerance. If the backtest ignores the guardrails, it is just a fantasy of an unconstrained trader.

What should you verify before going live?

Before an agent trades real money, it should pass through paper trading. This is the final out-of-sample test and the first time the agent's logic interacts with live data feeds, real order books, and actual API latency. Paper trading exercises the actual API, data feeds, and decision loop without exposing capital. Compare the paper results to the backtest. Expect some degradation. If the paper track record diverges wildly from the simulation, your execution assumptions were wrong or the agent is reacting to live data in ways the historical simulation did not capture. That is valuable information. It means the backtest needs to be rebuilt before any live capital is deployed. Next, verify the authorization chain. Live trading requires the owner to explicitly authorize a scoped API key. The agent cannot promote itself from paper to live. Confirm that the key has the correct permissions, budgets, and approved withdrawal addresses. Test the panic switch during paper trading. Trigger it and observe whether the agent flattens positions, cancels open orders, and revokes access as intended. A kill switch that fails in simulation will fail in production, and by then the cost is real. Monitor for drift. An agent that behaves one way in a backtest and another way in paper may be overfitting to historical noise, or the live market structure may have shifted. Finally, internalize the risk. Trading can lose money, including the entire amount allocated to the agent. The backtest does not change this fact. It only tells you how the agent behaved in a simulated past. Start with capital you can afford to lose. Increase allocation only after the agent demonstrates predictable, bounded behavior over a meaningful paper trading period. The transition from backtest to live is not a single decision. It is a sequence of escalating tests, each one designed to catch a different category of error before the next level of risk is introduced.