What Is AI Inference, and Why Is Scaling It So Hard?

What is Inferencing in AI?

AI Inference is when an artificial model provides you an answer based on data. What most of us generally call "AI" is really the success of AI inference. Call it the final step in the long and complex procedure of machine learning.

Let me take an example. Think of an AI model trained on insects, on their differences and similarities, their typical health and behavior. It needs a large collection of data to make connections and identify patterns.

After successful training, the model is able to make inferences such as identifying the species of a spider, recognizing its movement, and so on. Even though the AI has never seen those exact insects outside its training data before, the extensive data it was trained on allows it to make inferences in a new environment in real time.

Our own neurons make connections like this too. We learn about insects from books, documentaries, and online resources. When we see an insect, we can make an inference like that is a spider, even if we've never been to a zoo, because we researched beforehand. The same philosophy applies to AI models during inference.

Why is it important?

Inference is the operational phase, where the model applies what it learned during training to real scenarios. Its ability to identify patterns and reach conclusions is what sets it apart from other technologies.

But inference comes under a lot of pressure from models that keep growing bigger. As models get more complex, inference gets slower.

For inference to be fast, AI models need to do a lot of math in a short period of time. Factors like model size, user volume, and latency can all limit performance in one way or another. As models grow in size, they require more memory (part of why GPUs have gotten so hard to get your hands on), which in turn makes it harder for accelerators to keep up.

That's why the hardware and software that support inference can make or break your AI strategy.

I'll explore the types of AI inference in more depth in another post, but to give you an idea for now, there are four:

• real-time (online) inference
• batch inference
• streaming inference
• edge inference

How do we scale inference?

The deployment of LLMs has moved from basic capability demos to the bedrock of industrial computing. The emergence of reasoning-centric architectures, such as DeepSeek-R1 and OpenAI-o1, has proven that the next generation of capability isn't solely a product of larger training runs. Instead, we're entering a new phase of test-time compute scaling, where models are given the computational resources to "think," explore multiple paths, and check their work before generating a final answer.

Inference Has a Split Personality: Bandwidth-Bound and Compute-Bound

To understand why reasoning models put so much load on infrastructure, I find it helpful to look at something called the Roofline Model. It's a simple way of asking one question about any piece of computation: are you limited by how fast the chip can do math, or by how fast you can move data in and out of memory? You plot throughput against a number called arithmetic intensity, which is just how many FLOPs of math you get done per byte of data you move. High arithmetic intensity means you're doing lots of math per byte moved, so you're compute-bound. Low arithmetic intensity means you're moving lots of data for very little math, so you're bandwidth-bound. Every chip has a point called the ridge point where it flips from one regime to the other. Once I look at inference through this lens, it becomes obvious that inference isn't a single workload, it's two very different ones stitched together.

The first is prefill, which happens the moment a prompt comes in. The model reads the entire input in one go and runs it through the network in parallel. Because the same weights get reused across every token in that input, you get a lot of math done per byte of weight loaded. That's high arithmetic intensity, so prefill sits right above the ridge point. It's compute-bound, meaning the bottleneck here is raw GPU horsepower, not memory speed.

The second is decode, and it works completely differently. Once the model starts generating its response, it produces one token at a time, and to generate each single token it has to read through the entire weight set again plus the KV cache, all from high bandwidth memory (HBM). You're moving a huge amount of data to do a comparatively tiny amount of math per step. Arithmetic intensity collapses to something like 1 FLOP per byte, hundreds of times below the ridge point. Decode is bandwidth-bound. The GPU is sitting there waiting on memory, not crunching numbers.

For a normal chatbot this split barely matters because outputs are short, so decode doesn't last long. Reasoning models are a different story. Their traces can run past 10,000 tokens before they even generate an answer, and telemetry from real deployments shows these workloads spending upwards of 99% of their wall clock time stuck in that slow, bandwidth-starved decode phase.

Falling Off the Cliff

As a reasoning model works through a long Chain of Thought (CoT), it can't just forget what it already generated. To produce the next token, the model needs to attend back over every token that came before it, and to do that cheaply it stores a compressed representation of each past token, called its key and value, in the KV cache. Every new token adds more entries to this cache. The longer the reasoning trace gets, the bigger the cache gets, and it grows linearly with the number of tokens already generated. There's no way around this, it's just how attention works.

Now think about what a normal operator would do to get more throughput out of a GPU. The usual move is to run bigger batches, packing in more concurrent requests so the hardware stays busy. For reasoning workloads, this backfires badly. The cumulative KV cache footprint eats through available HBM fast, and once memory hits saturation, you've hit the Reasoning Cliff.

This is where it gets painful. The scheduler now has no good options. To free up memory it has to preempt some active requests, kicking them out of active generation and back into a waiting queue. But when those requests eventually get rescheduled, the system usually can't just pick up where it left off. It has to redo the entire prefill computation from scratch to rebuild the context the model needs. That recomputation wrecks tail latency and tanks overall throughput, which defeats the entire point of running a bigger batch in the first place.

So you end up with a genuinely inverse scaling law. Pushing batch size higher does shave a bit off time to first token, since requests spend less time waiting to start. But it absolutely destroys time per output token, since the system is now constantly thrashing between generating and recomputing. You gain a little on one latency metric and lose a lot on the other. Net, you're worse off.

Where the Bytes Actually Go

Surviving the cliff comes down to controlling three separate data flows, and each needs its own set of tricks:

• weight I/O
• KV cache I/O
• activation I/O

Weight I/O is dominated by sheer model size, and that size drives decode-phase cost, especially at smaller batch sizes. Quantization, dropping from FP16 down to INT4, cuts weight memory by roughly 4x and gives decode throughput a real boost. Structured sparsity, like the 2:4 pattern Nvidia's Ampere and Hopper chips support natively, zeroes out weights in a hardware-friendly way and roughly halves weight I/O on top of that.

KV Cache I/O is where things get interesting for me, because for long reasoning traces the KV cache can actually outgrow the model weights themselves. A few approaches have stood out. Grouped Query Attention (GQA) shares KV heads across multiple query heads, shrinking the KV footprint by 4x to 8x versus standard multi-head attention (MHA). It's more or less table stakes now for anything with a long context window, Llama 3 included. DeepSeek-R1 goes further with Multi-Head Latent Attention (MLA), compressing the KV cache into a low-rank latent representation. DeepSeek's own published numbers put the reduction at around 93% versus a dense MHA baseline, something like 14 to 15x. That's a big part of how a 671B parameter model manages to sustain massive reasoning traces without falling over.

Then there's PagedAttention (PA), the method vLLM popularized. It chops the KV cache into fixed size, non-contiguous blocks, the same way an OS handles virtual memory paging. Before this, KV caches were stored in large contiguous chunks, and the original vLLM paper found that this wasted somewhere between 60 and 80% of allocated memory to fragmentation. PagedAttention gets that waste down under 4%, which means a lot more concurrent requests fit in the same physical memory.

I also think eviction deserves more attention than it gets. Not every token in a long context window is worth keeping around. As the trace grows, irrelevant tokens start diluting attention, and useful evidence has to compete with noise for the model's focus. Methods like DBTrimKV, H2O, and SnapKV drop low-utility tokens based on predicted relevance or attention scores. That frees memory and, somewhat counterintuitively, often sharpens long-horizon reasoning too.

Speculative Decoding

One of my favorite ways around the memory bandwidth wall is speculative decoding. Rather than the big model generating one token per weight load, a small, fast "drafter" model proposes several candidate tokens at once, and the target model verifies them all in a single parallel pass. That single move raises the arithmetic intensity of decode enough to push it back toward compute-bound territory. Newer variants even let the drafter and target use completely different vocabularies, matching at the string level instead of the token level, so I don't need to train a custom drafter for every target model.

Parallelism Needs a Rethink Too

Data Parallelism (DP), replicating the full model across GPUs, used to be the default scaling strategy. For reasoning workloads I've come to see it as a bad fit. Every replica needs to hold a full copy of the weights, which leaves very little room for KV cache. I end up with stranded capacity: one replica thrashing under load while another sits mostly idle.

I'd rather match parallelism strategy to model shape. Dense models like Llama 3.1 405B lean on high-degree Tensor Parallelism (TP), sharding weights across GPUs to pool memory and bandwidth and free up HBM for the KV cache. That pushes the preemption cliff further out. Sparse MoE models like DeepSeek-R1 671B behave differently. Since they only activate a fraction of parameters per token, their compute-to-communication ratio is low, and heavy TP just adds synchronization overhead. These models do better with hybrid Pipeline Parallelism (PP) plus tensor parallelism, using PP to split the model into sequential stages across GPUs, handling memory staging while keeping TP light.

Where This Is Headed

Prefill wants compute. Decode wants bandwidth. Trying to serve both from the same monolithic GPU is starting to look like the wrong architecture to me.

I think the direction things are heading is disaggregation, routing compute-heavy prefill to dense accelerators built for raw TFLOPS, while decode gets handled by memory-centric clusters with tiered pooling, HBM backed by CXL expansion and NVMe, built to absorb the huge KV caches that long reasoning chains generate.

On the edge side, unified memory architectures, Apple's M-series being the obvious example, sidestep a lot of this by sharing one big pool of LPDDR across CPU and GPU, avoiding the offload penalties that plague discrete GPU setups. That alone lets consumer devices run reasoning models with KV caches that would otherwise choke a typical GPU.

Where This Leaves Me

Honestly, I picked this niche subject just to learn how AI inference works and scales, and I think the move from fluent text generation to genuine multi-step reasoning is quietly rewriting a lot of my assumptions about AI infrastructure. The Reasoning Cliff tells me raw compute isn't the constraint it used to be. Memory bandwidth, KV cache design, and compression are the real first-class problems now. Between smarter parallelism, aggressive KV eviction, and disaggregated hardware, I think the industry is finding its way toward inference that doesn't fall apart the moment a model decides to think longer.

References

Arif, M., Maurya, A., Vazhkudai, S., & Nicolae, B. (2026). Understanding Inference Scaling for LLMs: Bottlenecks, Trade-offs, and Performance Principles. arXiv:2605.19775

Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP '23). arXiv:2309.06180

DeepSeek-AI. (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434