How to build a market data pipeline that respects agent hard limits

A market data pipeline for an agent with hard limits must do more than deliver prices. It must verify that every quote, fill, and balance update is consistent with the owner’s budget caps, position limits, and kill switch before the agent acts. If the pipeline skips these checks, the agent can receive a signal it is technically allowed to trade on and breach a limit that was supposed to be absolute.

What does a market data pipeline do for a trading agent?

A trading agent does not have eyes. It has no intuition about whether a price is reasonable or whether a market is closed. Its entire model of the world is built from the data the pipeline feeds it. This means the pipeline is not a passive pipe. It is the agent’s perception layer, its accounting layer, and its first line of defense. The pipeline decides what the agent knows, when it knows it, and how much capital the agent believes it has left.

For an agent trading across multiple market types, the pipeline must normalize inputs that look nothing alike. A stock feed delivers last trade and bid ask. A perps venue delivers mark price, index price, and funding rate. An options venue delivers implied volatility, delta, and gamma. A prediction market delivers probability prices and resolution status. The pipeline must convert all of these into a single coherent schema where the agent can reason about total exposure in plain USD. The agent should not need to know that a perp contract has a multiplier or that an options lot covers one hundred shares. The one API for every market model handles this normalization, but the pipeline is where the raw feeds enter that unified model.

The pipeline also serves as the system of record for committed capital. The agent may think in terms of signals and entries, but the pipeline must track how much of the owner’s budget is already locked in open orders, working orders, and filled positions. If the owner sets a hard daily budget of one thousand dollars, the pipeline must know that three hundred dollars are already tied up in a limit order that has not yet filled. Sending the agent a price without this context is dangerous, because the agent may assume it has the full one thousand dollars available. This shadow ledger is the source of truth for every limit check.

How do you source prices without expanding the agent's authority?

The most common architectural mistake is giving the agent direct access to external data sources. If the agent can query a broker, a venue, or an aggregator on its own, it can bypass the internal limit checks. It can also consume data that has not been normalized, leading to errors in position sizing. The pipeline must sit between the agent and the outside world, and the agent must have no alternative path.

This separation requires distinct credentials. The pipeline holds read-oriented keys for fetching prices. The agent holds a scoped key that only permits trading actions within the owner’s limits. The agent should not be able to request new API keys, add data sources, or query raw balances from a venue. These privileges belong to the owner and the pipeline. Why AI agents need scoped API keys when trading real money explains this in detail, but the core idea is simple: the agent should be able to trade within a box, not redesign the box.

When the pipeline fetches data, it should pull from the same venues the agent will trade on, but through a sanitized interface. For stocks, this means a broker feed with proper market data entitlements. For crypto and perps, this means public order books and ticker feeds. For options, this means volatility surfaces and greeks. For prediction markets, this means probability feeds and resolution oracles. The pipeline should not mix testnet or simulated data with live data. It should tag every packet with its source and its ingestion timestamp so that downstream checks can verify freshness.

What checks keep the agent inside its hard limits?

Hard limits are owner-defined constraints that the agent cannot override. They include daily or weekly spend caps, maximum position sizes per market, total portfolio exposure limits, and maximum drawdown thresholds. These are not soft suggestions. They are boundaries that, if crossed, represent a failure of the system. The pipeline must therefore treat limit checking as a mandatory step in every data publication cycle.

It is important to remember that trading can lose money, including everything, and hard limits are designed to cap that risk, not eliminate it.

The pipeline maintains a real-time shadow ledger. This ledger tracks not only filled positions but also working orders. A working order is a commitment of capital. If the agent is long five hundred dollars of a stock and has a working buy order for another five hundred dollars, the pipeline must report that one thousand dollars are committed. The agent’s available envelope is zero, even though only half has filled. This prevents the agent from stacking orders that collectively exceed the budget.

Before each decision cycle, the pipeline should compute the agent’s current state and attach it to the data context. The agent receives a packet that includes the current price, the remaining budget, the current position size, and any halt flags. This lets the agent reason within constraints. For example, if the remaining budget is two hundred dollars and the position limit is one thousand dollars, the agent can size its next order accordingly. However, the agent’s reasoning is not the final enforcement. The execution layer must also validate every order against the same ledger. The pipeline provides the shared truth that both the agent and the execution layer rely on. How an AI agent trades within a hard budget it cannot exceed describes how this envelope is enforced end to end.

The pipeline must also account for the non-custodial architecture. Funds remain in the owner’s wallet. The pipeline tracks delegated spending authority, not wallet balances. This means the pipeline must detect wallet balance changes that reduce the agent’s effective budget and adjust the envelope immediately. If the owner withdraws capital or trades manually through another interface, the pipeline should see the reduced balance on the next update and tighten the agent’s available envelope. Failure to do this creates a gap between the hard limit and the real capital available, which can lead to overtrading.

How do you handle stale or missing data safely?

Stale data is one of the fastest ways to breach a hard limit. Suppose the agent sees a price of sixty thousand dollars for an asset and calculates that it can buy one thousand dollars worth within its budget. If the real price has moved to sixty-five thousand dollars, the same order might execute at a higher notional value than expected, or the order might fail and leave the agent without a position it thought it held. Either way, the agent’s internal model diverges from reality, and the limit check becomes unreliable.

The pipeline must timestamp every packet at ingestion and compare it to the system clock. If a stock tick is older than one minute, or a perp book is older than five seconds, the pipeline should mark it as stale. The policy should be simple: stale data is not actionable data. The pipeline can either withhold the packet entirely or append a trading_halted flag that the agent is instructed to respect. The agent should be configured so that a halt flag means do nothing. This is a prompt-level safeguard, but the pipeline makes it enforceable by providing the signal.

For missing data, the pipeline should not invent prices. Interpolation is dangerous for hard limits. If a feed drops, the pipeline should report an absence and wait. Fallback sources can be used, but they must be reconciled. If the primary and secondary sources for a prediction market diverge by more than a defined tolerance, the pipeline should treat both as suspect and pause. The agent should never be asked to arbitrate between two prices. That is the pipeline’s job, and if the pipeline cannot resolve the discrepancy, trading should stop.

In fast markets, the pipeline might need to batch updates to keep up. Batching is acceptable only if the batch is atomic and the timestamp reflects the newest data. Sending a batch with mixed ages can confuse the agent about current exposure. The pipeline should also monitor its own latency. If the end-to-end delay from venue to agent exceeds the strategy’s intended reaction time, the agent is effectively trading on old information even if the timestamps look recent. This awareness of its own state is critical.

Where does the kill switch fit in the data pipeline?

The kill switch is the final backstop. It flattens positions and revokes the agent’s access. Most owners think of it as an execution-layer tool, but the data pipeline plays a critical role in making it effective. The pipeline should listen to the kill switch state. If the owner triggers a panic, the pipeline must stop publishing new data immediately. Even if the execution layer is still unwinding positions, the agent should not receive new prices that might prompt it to enter again or adjust stops.

The pipeline can also act as a sensor for the kill switch. If the pipeline detects anomalous data, such as a price that implies the agent’s portfolio just breached a catastrophic limit, it can initiate a soft kill by cutting the feed. A soft kill is not a full flattening. It is a data cutoff. Without data, the agent cannot generate new orders. This is faster to deploy than market orders across five different market types, and it gives the owner time to assess before deciding whether to flatten.

The pipeline should maintain an immutable log of every data packet, every limit check, and every kill switch event. This log is essential for post incident review. It proves that the limits were checked and that the agent acted on a specific data context. The pipeline should also be designed so that a restart does not bypass the kill switch. If the pipeline reboots, it should query the current kill switch state before resuming publication. How to audit your trading agent guardrails before going live covers the full audit process, but the pipeline log is the primary evidence that the guardrails were active.

How do you test the pipeline before live trading?

Paper trading is the first test, but it is only useful if the pipeline is identical to the live version. Do not use a simplified mock data source for paper trading and a real feed for live. The pipeline should be the same in both environments, with only the execution layer switching between simulated and real fills. This ensures that the limit checks, normalization, and stale data handling are fully exercised.

Test the stale data path by introducing an artificial delay in the feed. Verify that the pipeline flags the delay and that the agent stops sending orders. Test the limit check path by simulating a hypothetical large fill that consumes ninety percent of the budget, then send a new price that would normally trigger a buy. Verify that the pipeline attaches a zero or negative remaining budget to the context and the agent does not send an order. Test the kill switch by triggering it during an active paper session. Confirm that the pipeline stops sending data within one agent cycle and that the agent does not attempt to trade while the switch is active.

The exact request schema is in the docs; the shape looks like this.

POST /v1/pipeline/context HTTP/1.1
Host: api.felix.trade
Authorization: Bearer YOUR_KEY
Content-Type: application/json

{
  "market_type": "perps",
  "symbol": "EXAMPLE",
  "require_fresh": true,
  "include_budget": true
}

Test normalization by feeding the pipeline an options delta, a stock price, and a perp funding rate. Verify that the agent’s context shows all three as USD exposures and that the agent can sum them correctly. Test prediction market resolution by simulating a resolved market and verifying that the pipeline updates the agent’s capital and removes the market from the active set. Test latency by measuring the time from a venue price change to the agent’s receipt. If the delay is too long for the strategy’s intended holding period, the pipeline is effectively stale by design.

Failure tests are also essential. Shut down one venue feed and verify the pipeline handles the absence without crashing or sending last known prices as current. Red team the agent by giving it a prompt that explicitly tries to trade despite a halt flag. Verify that the execution layer still rejects the order because the pipeline’s context is the source of truth. These tests prove that the hard limits are structural, not just behavioral.

Frequently asked questions