How to set up audit logs and observability for trading agents with hard limits

Audit logs and observability do not prevent losses by themselves, but they let you verify that a trading agent is respecting hard limits and expose drift before it becomes costly. A hard limit is a boundary the agent cannot cross, such as a maximum position size or a daily spend cap enforced by the API, not merely a prompt instruction. Observability means collecting structured records of every decision, order, and limit check so that a human or a second system can confirm compliance without trusting the agent. When real money is at stake, the absence of observability means you are relying on hope, which is not a control.

What should audit logs capture for a trading agent?

An audit log for a trading agent must be complete enough to reconstruct intent, context, action, and outcome without ambiguity. If you omit any step, you cannot replay the decision later. A complete record should include these elements in sequence.

1. 01The agent's reasoning at the time of the decision. If the agent uses an LLM, capture the prompt context or the plan it generated, because a limit breach is often preceded by a subtle shift in reasoning.
2. 02The pre-trade state: the available budget, the current position, the open orders, and the market data snapshot the agent consumed.
3. 03The order request itself, including the intended size in plain US dollars, the instrument type, the direction, and the timestamp with millisecond precision.
4. 04The response from the limit controller. If the agent requested a trade that would have breached a cap, the rejection is as important as the fill.
5. 05The post-trade state: the new position, the remaining budget, the realized and unrealized exposure, and any fees.

This sequence creates an immutable chain that you can replay later. A missing step breaks the ability to audit. Many developers log only filled orders, which is insufficient. Suppose an agent attempts to open a position ten times in one minute because its logic is looping. The first request fills, but the next nine hit a rate or position limit and are rejected. If you only log fills, you miss the loop entirely. You also need to log changes to the limits themselves. If you raise a daily spend cap from five thousand to ten thousand dollars, that change should appear in the audit trail with the owner authorization that allowed it. Without this, a future review cannot distinguish between a limit that failed and a limit that was legitimately adjusted. Configuring these limits correctly is covered in our guide on how to set spend caps and drawdown limits.

How do hard limits change the observability requirements?

When an agent operates under hard limits, observability is not only about profit and loss. It is about proving that the enforcement layer did its job. The agent and the limit enforcer are two separate systems. The agent proposes trades; the enforcer approves or denies them based on scoped keys, budget caps, and position boundaries. Your observability must capture both sides of this conversation. If you only monitor the agent's internal state, you cannot prove that a rejected order was actually blocked by the infrastructure rather than by the agent's own logic. This distinction matters because an agent that starts censoring itself might be hiding a deeper malfunction. The enforcer's log is the ground truth. The agent's log is the hypothesis. Observability reconciles them.

Imagine a scenario where an agent is supposed to maintain a maximum position of one thousand dollars in a perps venue. The agent submits an order for twelve hundred dollars. The enforcer rejects it. The agent's internal log might say 'order skipped due to risk check,' while the enforcer's log says 'rejected: position limit exceeded.' Both logs are necessary. If the agent's log is missing or falsified, you still have the enforcer's record. If the enforcer's log is missing, you still have the agent's record. The goal is cross-system verification, not trust in a single source. Understanding the enforcement layer is part of how to build guardrails for a trading agent.

Because Felix is non-custodial by construction, the wallet owner controls the funds and the withdrawal addresses. The agent can spend within limits but cannot withdraw to itself. Observability should reflect this custody model. Log the wallet address, the approved withdrawal destinations, and any attempt to modify those destinations. If an agent ever requests a destination change, that event should trigger an immediate audit alert because the agent is not authorized to make such changes. The hard limit here is architectural, not just numerical.

What does a minimal observability stack look like?

You do not need enterprise software to start. You need three components: structured logging, time-series metrics, and a read-only review interface. Structured logging means every event is emitted as a JSON object with consistent fields: timestamp, agent_id, event_type, limit_id, requested_amount, approved_amount, remaining_budget, and error_code if applicable. Send these logs to an append-only store. Append-only is important because it prevents an attacker or a buggy agent from erasing its tracks. The store can be a hosted logging service, a database with write-once permissions, or even a simple file system with immutable snapshots if the volume is low. The schema should be versioned. If you add a new field six months later, such as a correlation ID that links an LLM reasoning call to an order, the version number lets you query across old and new records without breaking historical analysis.

Time-series metrics track the rate at which limits are consumed. A gauge might show the percentage of the daily budget spent in the last hour. A counter might show the number of orders rejected by the position limit. These metrics let you see drift visually. Suppose the agent normally spends two percent of its budget per hour, but over the last three hours it has spent fifteen percent. The time-series chart will show a slope change that is easy to spot, whereas the raw logs might require a query to reveal. The review interface should be read only for the operations team. If the person reviewing logs can also modify limits or approve orders, you lose the separation of duties that makes observability trustworthy.

How should you alert on limit breaches and drift?

Alerts should distinguish between a breach, a near-breach, and behavioral drift. A breach means the limit controller blocked a trade or the kill switch flattened a position. This is a hard failure and should page the owner immediately. A near-breach means the agent has consumed eighty percent of a budget cap or is within five percent of a position limit. This should send a high-priority notification but does not need to wake someone up unless it is part of a pattern. Behavioral drift is the hardest to detect. It might look like the agent doubling its average order frequency while halving its average order size, staying under the cap but changing its strategy in a way that was not intended.

To catch drift, define baseline metrics during a paper trading phase. Record the normal distribution of order sizes, the typical hours of activity, and the expected correlation between market volatility and position changes. Then set thresholds that flag deviations. For example, if the agent normally places orders between nine and eleven in the morning and suddenly starts trading at two in the morning for three consecutive days, that is worth reviewing. The alert itself should include a deep link to the relevant audit logs so the reviewer can see the exact orders and the agent's reasoning. Never let alerts become noise. If you alert on every order, you will ignore the signal that matters. The relationship between alerts and emergency stops is explained in how to build a kill switch your trading agent cannot override.

Trading can lose money, including everything, and observability does not change that fact. What it changes is the speed at which you can respond. An alert that fires within minutes of a strategy breakdown gives you the option to flatten, revoke the key, or update the limit. An alert that fires after the account is depleted is only a post-mortem. Design your alert hierarchy so that breaches are impossible to ignore, near-breaches are reviewed within hours, and drift is analyzed weekly.

How do audit logs relate to guardrails and kill switches?

Audit logs are the memory of your safety system. Guardrails are the rules, the kill switch is the emergency brake, and the logs are the record that proves whether the brake worked. When you build a kill switch that flattens positions and revokes the agent's key, the log should show the trigger event, the positions closed, the sequence of revocation, and the final account state. Without this, you cannot be certain that the kill switch executed correctly. The agent might report that it stopped, but only the enforcer's log can confirm that the scoped key was actually disabled and that no further orders were accepted.

After any kill switch event, review the preceding logs for early warning signs. Look for orders that were approved but should have been rejected, limit checks that took longer than usual, or market data that seemed stale. Suppose the agent was trading on a prediction market and the kill switch triggered because the agent reached its loss limit. The logs might reveal that the agent had been increasing position size gradually over six hours while the market moved against it. This is not a failure of the kill switch; it is a failure of the drift detection that should have fired earlier. Use the logs to tighten the guardrails, not just to blame the agent. For broader safety practices, see how to run an AI trading agent with real money, safely.

Paper trading should produce the same log format as live trading. If the logs differ between environments, you cannot rely on paper trading to validate your observability. The only difference should be a flag indicating the environment. When you promote an agent to live trading, the owner must explicitly authorize the key, and that authorization should itself be logged. The first live trade should be small, and the observability system should confirm that the live limit enforcer is active by showing a limit check in the log. This is your final integration test before larger capital is deployed. Treat this test as seriously as the live trade itself. If the log does not show the limit check, stop and investigate the API configuration before increasing position size.