Serving Prefill and Decode for Agentic AI

An AI agent rarely finishes a task in a single model call. Instead, it works in a loop: generate instructions, wait for tools, read their results, and continue.

Fast token generation decreases the time between steps, and storing context in memory saves the system from re-computing existing inputs. However, the balance between processing new input and generating output changes according to the task and the model.

Tenstorrent Galaxy® systems handle both phases on the same Blackhole® chip: processing new input (prefill) and generating output (decode). That lets operators shift capacity between the two as their workloads change, instead of being required to make an allocation decision when they buy the hardware.

Agents turn one task into many model calls

Every model call has two phases. Prefill processes the input tokens and builds their key-value cache (KV cache), the stored attention data the model uses as context. Decode then draws on that state to generate output, extending the cache as it goes.

In an agent workflow, that output often kicks off another operation. A coding agent, for example, might request a file, examine it, change some code, and run a test. The result then comes back to the model in a new call, along with the instructions, earlier messages, and prior outputs that came before it.

Figure 1. An agent repeats prefill and decode as tools return results. The bars show how earlier input and output become a reusable prefix on later calls. Schematic lengths, with ideal prefix reuse; real sessions also branch, retry, and shorten their history.

What the serving system sees, then, is a series of related requests with steadily growing context, where a long input is often mostly material the model has already processed. That matters both for latency and for how much fresh computation each request requires.

Faster generation reduces time spent waiting

When a tool can't run until it has the model's output, the next step in the flow has to wait for generation to finish. These delays add up along each sequence of dependent calls, so even work that runs in parallel only finishes when its longest chain of dependencies does.

Consider a task made up of eight sequential model calls, each generating 300 tokens. Allow 0.5 seconds per call before generation begins, plus seven tool waits of one second each. At 50 tokens per second, the task takes 59 seconds; at 200 tokens per second, it takes 23. The model is generating four times faster, but the task finishes only about 2.6 times faster, because the other waits stay the same.

Figure 2. Modeled task latency, not measured hardware performance. Eight calls generate 2,400 tokens in total. Other time is four seconds of startup plus seven seconds of tool execution. Queueing is excluded; output length and tool behavior stay fixed.

Speed alone isn't enough, though. Operators still need enough total throughput to make the service economical, so the useful target is the number of concurrent agents a deployment can support at the required response time.

Cached tokens change the amount of prefill work

Workloads are often described by their input sequence length (ISL), the number of tokens presented to the model, and their output sequence length (OSL), the number of tokens it generates. Neither number, however, tells us how much of that input has already been processed.

For an agent session, three counts are more useful:

1. Cached input, which reuses existing KV state,
2. New input, which needs fresh prefill, and
3. Generated output, which needs decode.

Prefix caching is what makes the first of these possible, skipping computation for an exact reusable prefix whenever its state is available to the serving worker.

Suppose a call presents 100,000 input tokens and generates 1,000 output tokens, giving it an ISL/OSL ratio of 100:1. If 99,000 of those input tokens are served from cache, only 1,000 need fresh prefill, and the new input/output ratio drops to 1:1. Both ratios describe the same model call, and neither is a hardware allocation rule.

Figure 3. Illustrative token and billing example. Prices are assumptions: $0.20 per million cached tokens, $2 per million new input tokens, and $8 per million generated tokens. Charges total $0.0298. Cache storage and write fees are excluded.

The same example shows why discounted cache reads still matter financially. At those assumed prices, cached tokens make up 66.4% of the token bill. That figure represents customer spending and provider revenue, not the cost of the infrastructure behind it.

Reusing the cache avoids recomputing the prefix, but it isn't free. The KV state still occupies memory and still has to be read by attention, moving it between workers uses bandwidth, and if it's evicted, later requests must rebuild whatever is missing. Preserving useful context is a key part of serving the agent efficiently, not just an accounting feature.

Separate serving pools need a changing balance

Prefill and decode put different kinds of pressure on hardware. A long fresh prompt supplies many token positions that can be processed in parallel, whereas decode generates each sequence one step at a time. At small batch sizes, reading the weights and KV state often takes longer than the arithmetic itself.

Disaggregated serving tackles this by assigning the two phases to separate worker pools, each with its own batching and parallel layout, and transferring KV state between them. DistServe studied this approach, and the split works with identical accelerators as well as with different chip types.

But separating the pools creates work of its own. The cache has to reach the decode worker, and requests have to be routed to available capacity without losing prefix reuse. In a fixed fleet, every worker added to one pool is capacity taken from somewhere else.

The right prefill/decode (P/D) allocation depends on the workload. Fresh documents increase prefill demand, reused prefixes reduce it, and longer reasoning adds decode work. Changing the model, the context length, or the target response time also changes how quickly each worker can serve requests. No single P/D ratio fits all of these conditions.

Hardware that can serve either phase keeps that allocation adjustable after purchase. Reassignment isn't instant, since active requests must drain and weights or parallel layouts may need to change, but it gives operators a choice of roles for the capacity they already have. Whether that turns into savings depends on measured utilization and transition costs.

Blackhole combines compute with two kinds of memory

Tenstorrent uses the same Blackhole chips in Galaxy systems for both prefill and decode. Each 32-chip Galaxy provides 6.2 GB of on-chip SRAM and 1 TB of external GDDR6 memory, with a peak of 23 PFLOPS in Block FP8. Those are aggregate hardware specifications rather than application throughput. Each system also ships in a 6U air-cooled chassis with no direct liquid-cooling connection, which extends that flexibility to the facility: it can go into any site equipped for air cooling, provided the racks meet its power, airflow, and weight requirements.

The two memory types do different jobs. SRAM keeps active data close to the compute engines, while GDDR6 holds the model weights and KV state that exceed on-chip capacity. The interconnect then carries data between cores, chips, and servers so that larger models can run across systems and superclusters. In our inference architecture, prefill distributes work across servers, while decode pipelines work using both SRAM and DRAM.

A single matrix operation shows why the two phases call for different execution layouts. DeepSeek-V3's attention output projection contains about 117 million weights, so at one byte per weight, a full read comes to about 117 MB. Under an idealized assumption of 512 GB/s memory bandwidth, that read takes about 229 microseconds.

Figure 4. Modeled work for one projection

Figure 4. Modeled single-operation example, not a benchmark. The matrix is 16,384 × 7,168. Arithmetic assumes 719 trillion operations per second, derived from 23 PFLOPS across 32 chips. One multiply-add counts as two operations. The model assumes one full weight read and excludes metadata, activation traffic, synchronization, and other layers.

In the long prefill, each weight is applied to thousands of input positions, so arithmetic dominates. In the small decode batch, there is far less computation per weight read, and keeping memory busy matters more. That contrast is why neither peak compute nor peak bandwidth alone can predict serving performance.

Data placement determines how the hardware is used

Our kernels explicitly control where tensors reside and when they move. Reader code brings data into SRAM, compute code operates on it, and writer code sends the results onward, with all three overlapping in time. When several cores need the same data, the on-chip network distributes a shared copy rather than having each core fetch it separately from DRAM.

In prefill, reusing inputs and keeping intermediate results on chip cuts external memory traffic while token positions are processed. In decode, spreading weights across memory banks and coordinating reads keeps data flowing to the compute engines. In both cases, SRAM provides local working space and GDDR6 provides capacity for weights and reusable context (though weights, KV state, and temporary buffers all have to share that finite capacity).

Attention is a good illustration of how the layout changes. During prefill, the cores divide up the input queries. During decode, a single sequence supplies only one new query at a time, so the cores instead divide the existing KV context and combine their partial results. The hardware is the same in both cases; only the work assignment changes.

Figure 5. Simplified attention layouts. Prefill distributes query positions; decode distributes KV positions and combines partial results. KV is read in chunks, not assumed to fit in full on each core. Batch and attention-head parallelism are omitted.

A published measurement shows why this control matters. In our FlashAttention report, we tested on Wormhole, our previous-generation processor, using a single n150 card. The FlashAttention kernel ran 9 to 44 times faster than a basic attention implementation on the same hardware. The baseline wrote intermediate results to DRAM, whereas the optimized version kept them in SRAM and skipped unnecessary causal-attention work. The tests covered BF16 and BFP8 formats, sequence lengths from 512 to 16,384, and batches totaling 16,384 tokens. This is a measured kernel comparison on Wormhole, not a Blackhole service result or a comparison with another vendor.

Agentic traffic keeps the P/D balance moving

Agentic traffic pulls on both phases at once. Dependent calls make decode speed show up in every task, context reuse cuts fresh prefill but makes KV state worth keeping, and the mix between them shifts with the task, the model, and the response target. No fixed prefill/decode split fits all of that, so hardware that can serve either phase lets operators follow the balance as it moves.

It also means tests should be run on the whole deployment, not on a single kernel or phase. Agent-focused benchmarks run coding-agent traffic under concurrent load and show how many agents a system can serve at a given response time. A fair comparison holds the model, workload, cache conditions, and response targets constant, states its power boundary, and keeps customer token prices separate from the operator's serving cost.

Tenstorrent Galaxy gives operators one hardware platform for both phases, local control over data movement, and an air-cooled deployment option. For agent workloads, you should judge the whole system: what it delivers, how fast, and at what cost.

If you're planning inference capacity for agentic workloads, talk to our team about how Galaxy fits your deployment, or explore Galaxy.

-----------------