Throughput vs Latency: How Transformer Design Impacts LLM Inference Speed

Imagine you are running a customer support chatbot. A user types a question. If the system takes five seconds to show even the first word of the answer, the user feels ignored. They might leave. But if you tune that same system to respond instantly, you might only be able to handle ten users at once before the server crashes under the load. This is the core tension in Large Language Model (LLM) serving: throughput vs latency. You cannot simply have both maxed out simultaneously without understanding the underlying hardware and software mechanics.

This tradeoff isn't just a minor inconvenience; it defines your operational costs and your user experience. Throughput measures how many tokens your system generates per second across all active requests. Latency measures how long a single user waits for a response. Optimizing one usually hurts the other. To break this deadlock, we need to look under the hood of transformer architectures and see exactly where the bottlenecks live.

The Anatomy of LLM Inference: Prefill vs Decode

To understand why speed varies so wildly, you have to split inference into two distinct phases. Most people treat "inference" as a single block of computation, but it is actually two very different jobs with opposite hardware needs.

Prefill is the initial phase where the model processes the entire input prompt to generate the first token. During prefill, the GPU is compute-bound. It has plenty of data (the prompt tokens) to keep its cores busy. Because the work is heavy on matrix multiplication, adding more requests to the batch doesn't help much. The GPU is already saturated. Whether you process one prompt or ten, the time to finish the prefill step stays roughly the same because the hardware is working at full capacity.

Decode is the iterative phase where the model generates subsequent tokens one by one based on previous outputs. Here, the situation flips. The decode phase is memory-bound. For each new token, the system must fetch the massive model weights from High Bandwidth Memory (HBM). This fetch operation is slow compared to the actual calculation. If you only generate one token at a time, the GPU spends most of its time waiting for data. By batching multiple requests together, you amortize that memory fetch cost. You pay the price of fetching the weights once, but you use them to calculate the next token for ten different users. This makes decoding scale linearly with batch size-up to a point.

This distinction explains why small batches feel fast (low latency) but waste money (low throughput), while large batches save money (high throughput) but make users wait (high latency).

Decoding the Metrics: TTFT, TPOT, and Goodput

When engineers argue about performance, they often talk past each other because they are measuring different things. You need three specific metrics to get the full picture.

1. Time To First Token (TTFT): This is the delay between when a user hits "Enter" and when the first character appears. TTFT is dominated by the prefill phase. Users care deeply about TTFT because it signals responsiveness. A high TTFT feels like a frozen app.
2. Time Per Output Token (TPOT): This measures how long it takes to generate each subsequent token after the first one. TPOT is dominated by the decode phase. Low TPOT means the text streams quickly once it starts.
3. Goodput: Raw throughput can be misleading. If your system processes 1,000 tokens per second, but half of those responses take too long and time out, your effective value is lower. Goodput counts only the requests that meet your Service Level Objectives (SLOs). It is the metric that actually correlates with business success.

A common mistake is optimizing for average latency. Averages hide the "long tail." If 90% of your users get a response in 200ms, but 10% wait 5 seconds due to a bad scheduling decision, your average looks fine, but those 10% will churn. You must optimize for percentiles, typically the p95 or p99 latency, to ensure a consistent experience.

The Batch Size Dilemma

Batch size is the primary knob you turn to balance these metrics. Let's look at a concrete example using an NVIDIA A100 GPU. If you run a small batch size of 4, the latency is low, but the GPU utilization might sit at 30%. You are paying for 70% idle hardware. If you increase the batch size to 64, utilization jumps to nearly 100%, and throughput increases by 14x. However, latency also increases by 4x. That four-second delay might be unacceptable for a real-time chat interface.

The relationship isn't linear forever. Once you hit the "compute-bound" regime in decoding, doubling the batch size no longer increases throughput proportionally. Instead, it just adds more queueing delay. Every additional request competes for the same limited memory bandwidth. At this stage, you are throwing money at the problem without gaining speed.

Scheduling Strategies: How Systems Manage the Tradeoff

Hardware limits are fixed, but software scheduling can mitigate them. The way an inference engine orders requests determines whether you prioritize speed or volume.

Traditional systems used Request-Level Batching is a method where new requests wait until all currently running requests complete their decode phase before starting. This approach keeps batches clean and simple. However, it creates a bottleneck. If one user asks a complex question that requires generating 2,000 tokens, every new user arriving during that time must wait. Their prefill phase is stalled. This leads to high TTFT for latecomers.

Modern frameworks like vLLM is an open-source library that uses PagedAttention to manage memory efficiently and enable continuous batching. introduced Continuous Batching is a technique that allows new requests to join the batch immediately after their prefill phase completes, without waiting for others to finish. With continuous batching, the system interleaves prefills and decodes. When a request finishes decoding, a new request enters the batch instantly. This keeps the GPU fed with work, boosting throughput significantly. Research shows iteration-level batching can achieve an order of magnitude higher throughput than request-level batching.

However, continuous batching introduces complexity. Interleaving prefills (which are compute-heavy) with decodes (which are memory-heavy) can cause interference. If too many prefills happen at once, they steal compute cycles from the ongoing decodes, causing the streaming tokens to stutter. Systems like Sarathi-Serve address this by dynamically prioritizing either prefill or decode based on current load, improving throughput by up to 6.9x for large models like Falcon-180B while maintaining latency SLOs.

The Cost of Scaling: Tensor Parallelism

What happens when the model is too big for one GPU? You use Tensor Parallelism (TP) is a distributed computing technique that splits model layers across multiple GPUs to reduce memory and compute load per device. This seems like a free win, but it comes with a hidden tax: communication overhead.

In tensor parallelism, each GPU holds a slice of the model's weights. After each layer, the partial results must be combined using an "all-reduce" operation. This happens twice per transformer layer: once after the attention mechanism and once after the MLP. The critical insight is that the volume of data communicated depends on the sequence length and hidden dimension, not on the number of GPUs.

As you add more GPUs, the computation per GPU drops, but the communication volume stays constant. This increases the communication-to-compute ratio. Eventually, the GPUs spend more time talking to each other than calculating. This degrades latency scalability. For example, splitting a model across 8 GPUs might cut latency slightly compared to 4 GPUs, but the throughput per dollar often drops because you are paying for interconnect bandwidth that isn't fully utilized. Topology matters here; NVLink connections between GPUs are far faster than standard PCIe, making the choice of server architecture crucial for TP efficiency.

Choosing Your Strategy Based on Use Case

There is no universal best setting. Your architecture should follow your application's needs. Here is how to decide:

• Interactive Applications (Chatbots, Copilots): Prioritize low TTFT. Users expect immediate feedback. Use smaller batch sizes or systems that prioritize prefill completion. Accept lower throughput to ensure the first token arrives in under 200ms. Quadrant II performance (low TTFT, moderate throughput) is ideal here.
• Bulk Processing (Summarization, Translation): Prioritize throughput. The user submits a document and checks back later. Latency doesn't matter as much as cost-per-token. Maximize batch size to saturate GPU memory bandwidth. Aim for high TPS (Tokens Per Second).
• Hybrid Workloads: If you serve both types, consider separating traffic. Route interactive queries to a dedicated cluster optimized for low latency, and send batch jobs to a separate cluster optimized for high throughput. Mixing them in one pool often leads to suboptimal performance for both.

Remember that high TPS numbers in marketing materials can be deceptive. Always ask for TTFT and p99 latency figures. A system claiming 10,000 TPS might be achieving that by letting some requests wait 10 seconds, which is useless for a live conversation.