How market data pipelines differ when agents run them

When a developer moves a market data pipeline from a manual script to an agent, control shifts from explicit code to prompt context and scoped permissions. The agent requests normalized data through a single API, but the developer no longer reviews every raw response before it reaches the trading logic. This changes where errors appear, how failures propagate, and what safety mechanisms actually prevent bad trades.

Most developers start with a script that polls a venue, transforms the response, and feeds a strategy. The transition to an agentic system is not just a swap of execution engine. It is a change in architecture, responsibility, and failure surface.

What does a manual pipeline look like for developers?

In a typical manual setup, a developer writes a script that authenticates directly to a market venue, fetches order book or ticker data, and normalizes the response into a local format. The script handles retries, rate limiting, timestamp alignment, and decimal precision in explicit code. When a venue returns an unexpected field or changes a schema, the script usually fails with a stack trace or a log entry that the developer can inspect. The human is the final filter: the developer sees the error, stops the process, and fixes the logic before any capital moves.

This model gives the developer total control over every byte of data. The script might store historical ticks in a local database, compute indicators in Pandas or NumPy, and only then emit a signal. The developer knows exactly which version of the code is running, which API key is active, and which IP addresses are whitelisted. The pipeline is deterministic: the same inputs produce the same outputs every time, assuming the venue has not changed its schema. The cost of this control is maintenance. The developer must manage authentication rotation, normalize contract specifications across multiple venues, and keep the execution loop running through network interruptions.

Manual pipelines also tend to couple data ingestion with strategy logic. A single Python file might fetch prices, compute a moving average, check a position table, and submit an order. This works for a single venue and a single asset class, but it becomes brittle when the developer wants to trade stocks, crypto, perps, options, and prediction markets through the same capital pool. Each venue uses different contract sizes, margin rules, and tick formats. The developer writes more normalization code, more error handling, and more state management. The complexity grows linearly with the number of markets.

What changes when an agent consumes the data directly?

When an agent takes over the pipeline, it does not run a developer's script. It uses an MCP tool or a REST API to request data, and the infrastructure returns normalized responses sized in plain US dollars. The developer no longer writes code to parse a perps venue's inverse contract or an options venue's Greek notation. The API abstracts that away. This removes a large class of normalization bugs, but it also removes the developer's direct visibility into the raw response.

The agent decides what to request based on its prompt context, not on a hardcoded cron schedule. Suppose the prompt tells the agent to monitor a portfolio across five market types. The agent might request stock prices, then crypto balances, then perps funding rates, all within the same session. It does not need a separate script for each venue. It uses one key and one API. This consistency is powerful, but it means the agent's reasoning about what data matters is now part of the pipeline. If the prompt is ambiguous, the agent might request the wrong data, misinterpret a field, or ignore a critical update because it fell out of the context window.

Because the agent is non-custodial by construction, it cannot withdraw funds to itself. It can only spend within owner-approved limits. This is a structural safety improvement over a manual script that holds an API key with broad permissions. However, the agent can still lose money by acting on stale or misinterpreted data. The developer must now design safety into the prompt and the scoped key, not just into the ingestion script. Why AI agents need scoped API keys when trading real money covers how those keys limit what the agent can do even if the data layer behaves unexpectedly.

The exact request schema is in the docs; the shape looks like this.

{
  "tool": "fetch_market_data",
  "arguments": {
    "market_type": "perps",
    "instrument": "example-pair",
    "fields": ["last_price", "funding_rate"],
    "max_budget_usd": 500
  }
}

Where do the failure modes shift?

Manual pipelines fail in ways that are usually easy to detect: a network timeout throws an exception, a JSON key is missing and the parser crashes, or a database connection drops and the script exits. These failures are noisy. They leave logs. They stop the system. Agentic pipelines fail differently. They can fail silently by misinterpreting a normalized value, hallucinating a parameter, or deciding that a stale price is still valid because the prompt did not explicitly tell them to check the timestamp.

Imagine a manual script that receives a price of 50000. The script knows this is the index price for a perps contract because the developer hardcoded that mapping. An agent receives the same normalized value and might interpret it as the mark price, the last trade, or the liquidation threshold, depending on how the prompt describes the field. The API normalizes the math, but it does not normalize semantics. The agent supplies the meaning, and that meaning can drift across sessions or model updates.

Another shift is in state management. A manual script usually stores state in a database or a flat file. The developer can query that state to see exactly what the system thought at 10:03 AM. An agent may hold state in its context window or in a lightweight session. If the context window truncates or the session resets, the agent might forget that it already opened a position, or it might double count a recent price move. The developer must now think about context retention and session boundaries as part of the data pipeline, not just as part of the trading strategy.

Error handling also changes. In a manual pipeline, the developer writes retry logic with exponential backoff. In an agentic pipeline, the retry is implicit in the tool call, but the agent might not know that a 503 error from a venue means it should wait five minutes. It might instead try a different endpoint, or it might decide the market is closed and trade anyway. The developer must now encode operational knowledge into prompts rather than into control flow. How to control risk when an AI agent trades through MCP explains how to build those boundaries so that a confused agent cannot amplify an error into a losing position.

How do safety controls move from code to keys and prompts?

In a manual pipeline, safety is often a series of if statements. If the price deviation is greater than five percent, abort. If the position size exceeds one thousand dollars, reject. If the volatility spike is unconfirmed, wait. These checks live in the same script as the data ingestion. They are explicit, testable, and version controlled. When the pipeline moves to an agent, some of these checks can still live in the strategy code, but many must move upstream into the infrastructure that hosts the agent.

Felix provides safety controls that are independent of the agent's reasoning. Scoped keys restrict which markets the agent can access. Budget caps prevent the agent from spending more than a fixed amount of owner capital. Position limits enforce maximum exposure. A panic switch flattens positions and revokes the key. These controls are not prompts. They are structural constraints that bind the agent even if the data pipeline feeds it garbage. This is a fundamental difference from a manual script, where a bug in the data layer can bypass every safety check because the checks are just more lines of code in the same file.

Prompt design becomes part of the safety architecture. The developer must tell the agent how to handle missing data, how to verify timestamps, and when to stop trading because the data looks suspicious. This is less precise than a unit test, but it is necessary because the agent has autonomy. The prompt is the interface between the developer's intent and the agent's behavior. If the prompt does not mention slippage, the agent might not account for it. If the prompt does not define stale data, the agent might trade on a price from twenty minutes ago. A practical checklist for non-custodial AI trading includes prompt patterns that help keep the agent aligned with the developer's intent.

The developer also loses the ability to step through the pipeline line by line. In a manual script, you can insert a print statement or a breakpoint to see exactly what the data looks like before the trade function runs. With an agent, the data is consumed by a model that does not expose its internal reasoning in a debugger. You can see the tool call and the response, but you cannot see every intermediate interpretation. This opacity means that safety controls must be enforced at the API layer, not assumed in the strategy logic.

What should developers validate before handing over the pipeline?

The transition from manual to agentic should not happen on a live account. Paper trading exists so that developers can observe how an agent interprets data without risking capital. This is not a formality. It is the only way to see whether the agent understands normalized fields the way the developer intends. A paper trading session might reveal that the agent confuses notional value with margin requirement, or that it treats a prediction market probability as a price in dollars. Trading can lose money, including everything, and these errors are far more costly when the agent is using live funds.

Developers should validate the kill switch and exit plan before the agent ever sees a live key. The panic switch must flatten positions and revoke access without depending on the agent's cooperation. The exit plan must define what the agent should do when data quality degrades. Suppose the agent normally trades based on a five minute moving average. If the data feed lags by fifteen minutes, what should the agent do? The answer must be in the prompt, and the answer must be tested in paper mode.

Developers should also review the scoped key permissions. A manual script might use a key that can read balances, fetch prices, and place orders across every market. An agent should start with the narrowest possible scope: only the markets it needs, only the order types it uses, and only the budget it was assigned. If the agent's pipeline is only supposed to trade prediction markets, the key should not have access to perps. This limits the blast radius when the data layer or the agent misbehaves. How market data pipelines stay safe for beginner trading agents offers a step by step guide to narrowing this surface.

Finally, developers should monitor the agent's tool calls as closely as they once monitored their script logs. The fact that the agent is autonomous does not mean it should be ignored. Log every request, compare the agent's interpretation against the raw normalized response, and watch for drift. If the agent starts requesting data at odd intervals or using parameters that do not match the prompt, that is a signal to revoke the key and debug. The pipeline is only as safe as the attention paid to it.

Frequently asked questions