How to build a reliable market data pipeline for trading agents

A trading agent needs a clean, time-ordered stream of prices, volumes, and order book states that it can trust enough to act on. Without that, the best model will generate orders on stale or misaligned information and lose money. The pipeline must deliver a single schema across every market type so the agent reasons about the trade, not about parsing.

What does a trading agent actually need from market data?

Before you design storage, networking, or caching, decide what the agent actually needs to see. A trading agent that works across stocks, crypto, perps, options, and prediction markets does not need to understand every native primitive of every venue. It needs a curated snapshot that contains enough context to make a decision and nothing more. The contents of that snapshot depend on the strategy, but the delivery must be uniform. For most strategies, the minimum viable context includes a current price, available liquidity, recent volume, and any market specific parameter that directly changes the expected payoff. A stock broker provides bid, ask, and last trade. A crypto venue adds twenty four hour volume and perhaps chain specific confirmation depth. A perps venue includes funding rate and mark price, which affect holding cost. An options venue includes implied volatility and delta, which affect how the position behaves as the underlying moves. A prediction market provides probability and liquidity around that probability, plus the time remaining until resolution. The Felix API normalizes venue specific contract math into plain US dollars, so the agent receives a single object shape whether it is looking at a stock or a perpetual future. That means your pipeline should emit a common schema with a canonical symbol, a UTC timestamp, a mid or last price, a spread or depth metric, and a freshness score. The agent should not query venues directly. Direct queries expose the agent to rate limits, timeouts, and schema changes that can be misinterpreted as signals. Instead, the pipeline polls or streams on the agents behalf, then presents a curated snapshot that the agent consumes in one read. If the agent is an LLM based system, concise context reduces prompt length and lowers the chance of hallucination. If the agent is a quantitative model, uniform columns let you reuse the same feature engineering across every market type. How trading APIs let AI agents trade across markets explains why one abstraction layer matters for multi market agents.

How should you normalize data across multiple venues?

Every venue uses its own symbol format, timestamp precision, and book depth. A stock broker may quote AAPL in dollars with four decimal places, while a crypto venue may use a token identifier and eight decimal places. An options venue may quote by strike and expiration in a nested structure. A prediction market may use a question identifier and a yes or no outcome flag. Your pipeline must map these into a canonical registry so the agent asks for a single symbol and receives the same field layout regardless of the underlying source. You should maintain a translation layer that maps venue native identifiers to your canonical symbols. This layer should also handle corporate actions for stocks, such as splits and ticker changes, so the agent does not think a price has halved overnight. For crypto and perps, the mapping should account for different base and quote conventions. For options, the canonical symbol might encode underlying, expiration, strike, and side in a fixed string format. For prediction markets, it might encode the event identifier and the outcome. The goal is that the agent never sees a venue internal identifier. You should also decide how much depth the agent needs. Some venues provide fifty levels of an order book, others provide only the top of book. If the agent strategy depends on depth, normalize every source to a fixed number of levels or to an aggregated depth metric, such as the volume available within one percent of the mid price. If the agent only needs a price and a spread, do not stream extra data that increases noise and bandwidth. More data means more parsing, more memory, and more opportunities for malformed fields to crash the consumer. Timestamps are another source of silent bugs. One venue may send epoch milliseconds, another ISO strings, and another may omit timezone information. Normalize every tick to UTC milliseconds and attach a pipeline ingestion timestamp so you can measure internal lag. Sequence numbers are equally important. If a venue provides them, store them. If not, assign a monotonic sequence per symbol so you can detect gaps and out of order messages. The exact request schema is in the docs; the shape looks like this:

curl -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"symbol":"EXAMPLE","market":"a perps venue"}' \
  https://api.felix.trade/...

The response would contain normalized fields such as symbol, bid, ask, last price, volume, and a source timestamp. The agent then consumes this shape without knowing whether the origin was a stock broker or an options venue.

Why do latency and freshness budgets matter more than raw speed?

Most agentic strategies do not need microwave latency, but they do need to know how old their information is when they act. A freshness budget is the maximum age of data that your agent is allowed to use. For a crypto perps venue, that budget might be five hundred milliseconds. For a stock broker during market hours, it might be one second. For a prediction market, it might be five seconds. The budget depends on the volatility of the asset and the reaction time of the agent, not on the absolute speed of the venue. You should measure end to end latency, not just the network hop. End to end latency includes the venue internal processing time, the transit time to your pipeline, the normalization and validation time, and the transit time to the agent. If any of those stages grows, the agent is trading on older data than it realizes. You should compare the venue timestamp against the pipeline ingestion timestamp at the moment the agent consumes the tick. If the gap grows, you are building a silent queue. When the gap exceeds the freshness budget, the pipeline must mark the data stale and the agent must halt or switch to a fallback source. A heartbeat mechanism helps you distinguish a slow market from a broken feed. If a venue stops sending ticks for longer than your budget, the pipeline should treat it as a failure rather than a calm market. Some venues send explicit heartbeats. Others do not, so you must synthesize one by expecting at least one message per symbol within a defined window. If the heartbeat misses, raise a flag. Sending an order on stale data is a common cause of unexpected slippage and losses. Trading can lose money, including everything, and stale data makes that risk worse. How to run an AI trading agent with real money, safely covers how to wire these halts into kill switches and budget caps so the agent does not trade blind.

What failure modes should a pipeline expect and handle?

Market data feeds fail in predictable ways. A venue may publish a zero price, a negative spread, or a volume figure that is larger than the entire market. Messages may arrive out of order, or the same tick may be delivered twice after a websocket reconnect. Without guards, the agent will act on these anomalies as if they were real signals. Your pipeline should validate every tick before it reaches the agent. Check that prices are positive, that the bid is below the ask, and that the timestamp is within a reasonable window of the current time. Check that volume is non negative and that percentage changes from the previous tick are within a sane threshold. Deduplicate using a venue sequence number or a unique message identifier. If the venue does not provide one, generate a deterministic hash from the content and drop duplicates within a short window. You should also maintain a cache of the last known good state so that if a single bad tick arrives, you can reject it without clearing the agents context entirely. Backpressure and circuit breakers are essential. If the agent is slower than the feed, an unbounded buffer will eventually exhaust memory and crash the pipeline. Instead, apply backpressure to the source or drop old ticks while keeping the newest. If a venue throws errors repeatedly, open a circuit breaker and stop pulling for a short period. Repeated bad data is often a sign of maintenance or a schema drift on the venue side. When the circuit closes again, validate the first few ticks carefully before resuming full speed. When the data layer fails, the safety layer must respond. Felix provides safety controls including scoped keys, budget caps, position limits, and a panic and kill switch that flattens positions and revokes access. If the pipeline detects systemic bad data, it should trigger that switch rather than hoping the agent will notice. The agent can spend within limits but can never withdraw to itself or steal, yet bad data can still cause it to spend the full budget on bad trades. How to build guardrails for a trading agent discusses how to connect pipeline health checks to those controls.

How do you test a pipeline before the agent trades live?

Testing the agent in isolation is not enough. The pipeline must be tested as part of the system. Paper trading on Felix uses the same data path as live trading, which means you can validate the full loop without risking capital. Live trading requires explicit owner authorization of a key, so the pipeline should be hardened long before that key is activated. Suppose you record a week of normalized ticks from every venue you intend to trade. You can replay that recording through the pipeline at faster than real time to stress the agent, or at real time to verify that decisions match your expectations. During replay, inject latency spikes and missing heartbeats to see whether the agent halts or overtrades on stale data. You can also inject bad ticks, such as a zero price or an inverted spread, to verify that your validation layer catches them before the agent sees them. You should also run shadow mode. In this mode, the pipeline consumes live data and the agent generates orders, but the orders go to paper trading instead of a live venue. This lets you observe behavior during real market events without exposure. If the agent performs well in shadow mode but the pipeline was never stressed with a schema change or a silent feed, you are not ready to go live. Test schema changes by adding or removing optional fields in your replay data to see if the agent parser breaks. After you go live, monitor the pipeline with the same freshness and error metrics you used in testing. A dashboard that shows lag per venue, validation error rate, and circuit breaker state gives you early warning before the agent starts losing money. Keep logs immutable and time synchronized so that post trade analysis can reconstruct exactly what the agent saw. Finally, test the authorization boundary. The agent should be able to read data and send paper orders, but live trading should remain blocked until the owner explicitly authorizes the key. Test that the kill switch works from the pipeline side. Simulate a data failure and verify that the panic switch flattens positions and revokes access as expected. Keep the pipeline stateless between ticks and log every normalized message so that any decision can be replayed later for debugging. Your first automated multi market portfolio walks through a step by step approach to connecting data and paper trading before you authorize live keys.