How to architect audit logs and observability for trading agents from first principles

Audit logs and observability for trading agents exist to answer one question after the fact: what exactly happened, and who or what caused it. Because the agent operates with real money across stocks, crypto, perpetual futures, options, and prediction markets through a single API, the logging architecture must treat the agent as potentially compromised from the start. The owner needs an immutable, owner-controlled record that is independent of the agent's own state and memory. This article explains how to build that system from first principles without trusting the agent to report honestly on itself.

Why does observability matter for non-custodial trading agents?

Non-custodial architecture means the owner controls the wallet and the funds, while the agent receives only scoped permission to trade within limits. This separation is essential, but it also creates a visibility gap. The owner is not manually clicking buttons on each venue, so they cannot rely on memory or screen recordings to reconstruct a session. The agent, whether connected through MCP tools or the REST API, may misinterpret a prompt, hallucinate a parameter, or encounter a corrupted tool definition. When any of these happen, the owner needs to see the chain of events that led to a position or a loss. Observability closes that gap by providing a ground truth that is separate from the agent's internal reasoning. Without it, the owner is left guessing whether a drawdown was caused by market conditions, a bad strategy, or a bug in the execution layer. Trading can lose money, including everything, but logs should reveal the mechanism of failure rather than obscure it. This is why how the safety model for trading agents differs from trading bots matters for visibility design.

What should an audit log capture from first principles?

A useful audit log is one that allows an owner to reconstruct every decision without asking the agent what it was thinking. From first principles, this means each log entry must bind together intent, instruction, context, and outcome. Intent describes the goal or strategy identifier that initiated the action, such as a portfolio rebalance or a hedging instruction. Instruction captures the exact API call or MCP tool invocation, including parameters that the agent believed it was sending. Context records the market data or portfolio state the agent observed at the moment of decision, which helps distinguish between a bad strategy and a stale price. Outcome stores the venue response, including fill prices, fees, slippage, errors, and rejections. Because Felix normalizes orders in plain US dollars across all five market types, the log schema can remain uniform even when the underlying venues use different contract specifications. This normalization prevents the agent from hiding true exposure behind complex multipliers or decimal conventions. However, the log must still retain the venue-specific confirmation hash or trade identifier so the owner can reconcile the normalized record against the venue's own statement.

• ·Timestamp with monotonic ordering to prevent replay or reordering attacks.
• ·Identity of the agent, the specific scoped key used, and the owner-controlled wallet address.
• ·Intent summary, such as the prompt fragment or strategy identifier that triggered the action.
• ·Exact request parameters in plain-dollar terms, including the target market type and venue.
• ·The raw response payload, including confirmation identifiers, partial fill details, error codes, and latency measurements.
• ·A cryptographic link, such as a hash or signature, chaining the entry to the previous entry to detect tampering or deletion.

How do you prevent the agent from tampering with its own logs?

If the agent can write or rewrite its own audit history, the log is merely a diary, not evidence. The architecture must therefore separate log creation from log storage and deny the agent any ability to modify stored records. The correct place to generate a log entry is at the API gateway, not inside the agent's process. The gateway records the request on the way out and the response on the way back, writing both to an append-only store that the owner controls. The agent may read from this store to maintain context across sessions, but it has no write privileges. Scoped API keys enforce this boundary. The key that authorizes trades is distinct from the key that administers logs, and the trading key lacks any permission to alter the log store. This is consistent with the non-custodial design where withdrawal addresses are owner-approved and the agent can never move funds to itself. The same principle applies to data: the agent can spend within budget, but it cannot erase the record of spending. Append-only storage can be implemented with simple cryptographic chaining or with owner-hosted storage where the agent simply lacks the credentials. The critical requirement is not the specific technology, but the separation of privileges. If the agent is compromised, the attacker should gain the ability to trade within limits, but not the ability to hide that they did so.

How do you observe an agent's internal state without exposing it to attack?

Observability is not only about the immutable past. It also includes real-time telemetry about what the agent plans to do next. The agent should expose its current strategy, pending orders, open positions, and remaining budget through a read-only telemetry channel. This channel is useful for dashboards, alerts, and human oversight, but it is distinct from the audit log in one crucial way: telemetry is a hint, while the log is a fact. The owner can use telemetry to see that the agent is approaching a position limit or that it has misinterpreted a price feed before it acts. If the telemetry claims the agent is idle but the audit log shows active orders, that divergence itself is a sign of compromise or corruption. The architecture should therefore treat the two streams as cross-checks. Telemetry passes through the same API gateway so it cannot be spoofed by the agent, but it is stored in a separate, mutable cache optimized for fast queries rather than long-term proof. When the agent connects through MCP tools, the telemetry also helps the owner understand which tool calls are in flight and what context the LLM is holding. This separation of hints and facts is covered in how audit logs keep MCP agents accountable.

How do you query logs when you need to stop the agent?

In a crisis, observation must convert into action without friction. A panic or kill switch flattens positions and revokes the scoped key, but the owner still needs to query logs during and after the event to confirm what was closed and what remains. The logging system must remain available even when the trading key is revoked, because the revocation itself is an event that must be recorded. The architecture should support high-cardinality queries across time ranges, market types, key identities, and order statuses so the owner can answer critical questions under pressure. For example, the owner might need to know which orders were filled in the last hour, which perps positions are still open, or which API key was responsible for a specific drawdown. The exact request schema is in the docs; the shape looks like this:

curl -X POST https://api.example.com/v1/query \
  -H "Authorization: Bearer YOUR_KEY" \
  -d '{
    "owner_wallet": "YOUR_WALLET",
    "time_range": {"from": "2026-08-15T10:00:00Z", "to": "2026-08-15T11:00:00Z"},
    "scope": "read_logs",
    "filters": {"market_type": "perps", "status": "filled"}
  }'

How do safety controls interact with the audit trail?

Safety controls like budget caps, position limits, exit plans, and kill switches are not merely runtime constraints. They are decisions that must leave their own audit trail. When a cap is reached, the system should log the threshold value, the current consumption, the requested action, and the enforcement result. This allows the owner to distinguish between a safety system that fired correctly and one that fired due to a misconfiguration. If a beginner sets a drawdown limit too tight, the logs will show the limit, the market movement, and the flattening order that followed. This matters because trading losses can come from two sources: the market moving against a position, or the safety system failing to behave as expected. Immutable logs make the distinction visible. If the agent attempted to exceed a cap but the gateway rejected it, that rejection is proof that the safety model functioned. If the cap was never checked, the absence of a log entry is proof of a bug. The same principle applies to kill switches. When the owner triggers a panic, the log should record the command, the positions flattened, the venues contacted, and any failures to close. A complete audit trail includes both the trades the agent wanted to make and the trades the system prevented it from making.

How do you maintain logs across a multi-market portfolio?

Trading across stocks, crypto, perps, options, and prediction markets through a single API creates a single chronological timeline that should not be fragmented by venue. The architecture stores all events in one unified stream rather than splitting them into separate databases by market type. A unified timeline makes it easier to detect correlated mistakes, such as an agent taking a long position in a stock while simultaneously shorting a related perps contract, or doubling exposure through an option without accounting for the underlying. The log schema must be flexible enough to accommodate market-specific details without breaking the common structure. An options entry might include strike price and expiration, while a prediction market entry might include resolution date and outcome bin. These fields are stored as structured metadata within the same normalized envelope that records plain-dollar sizing and wallet identity. The owner queries one interface to see total exposure and one interface to see the full history. This reduces the cognitive load of managing multi-market portfolios with agents and prevents the agent from hiding complexity by shifting activity across market types. The log is the single source of truth for everything the agent touched, regardless of which venue actually filled the order.