Recovering a Trading Agent After an Operational Incident
A step‑by‑step guide on how a trading agent should resume activity after an outage, data loss, or execution failure while preserving safety and compliance.
Produced with automation, then checked by deterministic quality rules and an independent source-grounded review before publication.
- 01The agent must verify market data freshness before any new order.
- 02Owner‑signed limits remain the final gate for any resumed activity.
- 03An emergency stop revokes the calling key but does not close existing positions automatically.
- 04Reconciliation of intended versus actual order states is required after a timeout.
- 05Permission reviews should be performed before the agent is re‑enabled.
After an operational incident the first priority is to confirm that the agent’s runtime environment is stable, that data sources are reliable, and that all security controls are intact. Only after these conditions are verified should the agent be allowed to place new orders or resume monitoring activities.
How does immediate post‑incident checks work?
The agent should pause all outbound activity, revoke any active agent keys, and run a health check that covers runtime status, venue availability, and data‑feed integrity. Missing timestamps or stale quotes must trigger a halt until they are verified.
- The runtime status endpoint confirms service health.
- The market data feed must include source, timestamp, and freshness warnings.
- The permission set should be re‑examined for any expired scopes.
Handling stale or missing market data
Stale data is a common cause of execution errors. The agent must compare multiple price sources when they are configured and reject any quote that fails freshness criteria. See When Should a Trading Agent Compare Multiple Price Sources? for guidance.
- 01Identify the freshest quote among configured sources.
- 02Validate the quote against a maximum age threshold.
- 03If no valid quote exists, log the condition and remain idle.
When is it safe to resume order placement?
Resumption is safe only after the agent has passed all verification steps and the owner has re‑authorized the appropriate limits. Owner‑signed limits such as daily notional, loss caps, and expiry remain enforceable throughout the recovery.
- Re‑enable the agent key with the same scope it had before the incident.
- Confirm that no pending orders are left in an ambiguous state.
- Run a dry‑run simulation that does not sign transactions to confirm logic.
Effect of an emergency stop on existing positions
An emergency stop revokes the calling key and cancels managed activity where possible, but it does not automatically close open positions or revoke token allowances. Those actions require a separate owner review and explicit intent.
The emergency stop is a protective measure, not a full liquidation tool.
Role of reconciliation in the recovery process
Because a timeout does not prove an order failed, the agent must reconcile intended orders with the venue’s confirmed state. This includes checking durable mutation identifiers and explicit error codes.
- Fetch the order status from the venue’s API.
- Match the venue’s order ID with the agent’s internal record.
- Record any mismatches for manual review before proceeding.
Should the agent automatically retry failed orders after an incident?
Automatic retries can duplicate trades or violate limits. The agent must wait for explicit owner approval after reconciliation.
How often should permission sets be reviewed during recovery?
Permission reviews should occur before the agent is re‑enabled and any time a scope is modified. See How Often Should AI Trading‑Agent Permissions Be Reviewed? for best practices.
Can the agent use a market order to speed up recovery?
Market orders may bypass price validation, increasing execution risk. The agent should prefer limit orders until data freshness is confirmed. Refer to When Should a Trading Agent Use a Market Order vs a Limit Order? for details.
What is the recommended way to handle repeated errors during recovery?
If errors persist, the agent should pause again, revoke its key, and alert the owner. Continuous error loops indicate deeper issues that need manual investigation.
Frequently asked questions
Automatic retries can duplicate trades or violate limits. The agent must wait for explicit owner approval after reconciliation.
Permission reviews should occur before the agent is re‑enabled and any time a scope is modified. See How Often Should AI Trading‑Agent Permissions Be Reviewed? for best practices.
Market orders may bypass price validation, increasing execution risk. The agent should prefer limit orders until data freshness is confirmed. Refer to When Should a Trading Agent Use a Market Order vs a Limit Order? for details.
If errors persist, the agent should pause again, revoke its key, and alert the owner. Continuous error loops indicate deeper issues that need manual investigation.
Sources and verification
Product claims in this article were checked against these first-party references. Runtime status remains authoritative for current availability.
- Felix documentationfirst party
- Felix machine referencefirst party
- Felix runtime statuslive status
Build with Felix now.
Felix infrastructure is live through MCP and the API. The full trading app launches September 17.
A decision log captures every step an AI trading agent takes, providing transparency, auditability, and a safety net for operators. This guide outlines the key fields and best practices for building a robust log.
A running trading agent depends on clear health signals to stay reliable. This article explains the most important signals, why they matter, and how to act on them.