The question nobody answers properly
OK, so you use ChatGPT or Claude every day. You ask it things, it answers in seconds, and meanwhile another three million people are doing exactly the same. How is that even possible?
The short answer: an obscene amount of hardware, a network that looks like science fiction, and serving software, which is where the real magic lives.
The long answer is this article. Hold on tight. You are going to love it.
Where can I tell you more?
I am 100% sure you are going to enjoy this article. And you will want more. Before you forget, subscribe to the newsletter so you don’t miss anything:
1. The hardware: it’s not about having lots of GPUs, it’s about having the right ones
Datacenter GPUs: a different league
When we talk about inference at industrial scale, we are not talking about your RTX 4090 for playing Cyberpunk. We are talking about NVIDIA datacenter GPUs: H100, H200, and the new Blackwell generation (GB200/GB300).
Why these and not others? Two reasons:
- Raw compute power for Transformer operations
- HBM (High Bandwidth Memory): memory with absurd bandwidth
HBM in one sentence: memory that sits physically stacked on top of the GPU die, connected by thousands of microscopic wires. Result: you can move data between memory and processor way faster than with regular memory.
And here comes the plot twist:
In large LLM inference, what limits you is not compute, it’s memory. You are constantly moving model weights and the KV cache around (more on that in a moment). If your memory is slow, it doesn’t matter that your GPU is a rocket.
The trick: “one giant GPU” instead of many separate ones
For very large models, the problem is not only having powerful GPUs. It’s how you wire them together.
Imagine you have a 400 billion parameter model. It doesn’t fit in a single GPU. You have to split it across several. And every time you generate a token, those GPUs need to talk to each other. If communication is slow, everything falls apart.
The fix: systems like the GB200 NVL72, which connect 72 GPUs into a single NVLink domain so they behave like one monstrous GPU.
NVLink in one sentence: a proprietary NVIDIA interconnect that links GPUs to each other with brutal bandwidth (up to 900 GB/s per link), skipping the bottleneck of going through the CPU or PCIe.
That lets you spread the model and the KV cache across many GPUs without latency going through the roof.
2. The network: two completely different worlds
When we talk about networking in these clusters, there are two levels that have nothing to do with each other:
Inside the rack: NVLink and NVSwitch
Here we are talking about communication between GPUs that live in the same server or rack. That runs over NVLink (the fat cable between GPUs) and NVSwitch (a switch that lets every GPU talk to every other GPU).
The characteristics: microsecond latency, bandwidth in the hundreds of GB/s. Basically, the GPUs see each other as if they were a single machine.
Between racks: InfiniBand or Ethernet
Here things change. When you need to connect GPUs sitting in different racks (or different buildings), you use:
- InfiniBand: the de facto standard for HPC and AI. Very low latency, enormous bandwidth.
- High-speed Ethernet: some deployments use heavily tuned Ethernet (100/400 Gbps with RDMA).
InfiniBand in one sentence: a network technology designed so data goes straight from one machine’s memory into another machine’s memory, without going through the operating system. Microseconds instead of milliseconds.
The goal of all this network engineering is simple: generating a token must not depend on waiting for data from another rack. Because if it does, latency multiplies and the user watches the cursor blink like an idiot.
3. Memory: it’s not just “the model”
Here comes a concept a lot of people are fuzzy about. When you load an LLM into memory, the model weights are not the only thing taking up space. There are three tenants:
The model weights
The model parameters, which can live in several formats:
FP16(16 bits per parameter)FP8(8 bits, more compressed)INT4/FP4(4 bits, aggressive quantization)
Quantization in one sentence: reducing the precision of the numbers that represent the model weights. Instead of 16 bits per number, you use 8 or 4. You lose some quality, but the model takes less space and runs faster.
The KV cache
This is the big one. The KV cache stores information about previous tokens so you don’t have to recompute it at every step.
KV cache in one sentence: when the model processes a token, it stores some vectors (keys and values) that it will need to process the following tokens. Without this cache, every new token would require reprocessing the whole conversation from scratch.
The problem: the KV cache grows with context length and with the number of concurrent users. If you have a 100,000 token context and 1,000 simultaneous users, you are talking about terabytes of memory just for caches.
Buffers and runtime state
Temporary space for intermediate operations, scheduler state, request queues, and so on. Not the biggest chunk, but it adds up.
4. Power and cooling: the real physical limit
This is where it gets serious. You can buy all the GPUs you want, but:
Electrical power
A rack with latest generation GPUs can draw 40-100 kW. A large AI datacenter can need hundreds of megawatts. To give you an idea, that is the consumption of a small city.
AI companies are signing contracts directly with energy providers and even building their own plants.
Cooling
Here is the physical wall. At 40-100 kW per rack:
- Air cooling: forget it. Air cannot carry away that much heat.
- Liquid cooling: the standard for dense deployments. Pipes with water (or coolant) running directly across the GPUs.
Direct liquid cooling in one sentence: instead of blowing cold air over the chips, you run a coolant through channels that touch the hot components directly. Far more efficient for dissipating concentrated heat.
NVL72 style systems already ship designed for liquid cooling because, at that density, there is no other option.
5. The serving software: where the real magic lives
This is the part people miss, and it is the difference between a service that works and one that dies.
A service like Claude or ChatGPT does not do “one inference per user” in isolation. It does something far more sophisticated.
Smart batching
The concept is simple: instead of processing one request, we process many at once.
Batching in one sentence: grouping multiple requests to process them together in a single pass through the GPU. GPUs are very good at processing lots of things in parallel, so a batch of 32 requests can take almost the same time as a single one.
The trick is batching without wrecking latency. You cannot wait 10 seconds to gather requests. There are algorithms that build batches dynamically, adding requests on the fly.
Separating prefill and decode
This is key and few people get it. LLM inference has two completely different phases:
Prefill: processing the initial prompt. Lots of compute in one go, but only once.
Decode: generating tokens one by one. Little compute per step, but many steps, and latency is critical because the user is sitting there waiting.
Prefill vs decode in one sentence: prefill is reading your whole message at once (compute intensive). Decode is writing the answer word by word (memory bandwidth intensive and latency sensitive).
Modern systems optimize these two phases separately and scale them differently. You can have GPU pools specialized in prefill and others in decode.
Disaggregated serving: decoupling everything
The most advanced architecture breaks the system into independent components:
| Component | What it does |
|---|---|
| Prefill nodes | Process prompts |
| Decode nodes | Generate tokens |
| KV cache servers | Store and serve caches |
| Routers | Distribute requests based on load |
Disaggregated serving in one sentence: instead of every server doing everything, you specialize servers in specific tasks and wire them together. Like an assembly line rather than a craftsman doing the whole thing alone.
This matters especially for MoE (Mixture of Experts) models.
MoE in one sentence: a model with multiple expert “submodels” and a router that decides which one to use for each token. You only activate part of the model at each step, so a 1 trillion parameter model can have the computational cost of a 100 billion one.
Tricks to cut latency
On top of all of the above, there are specific techniques to make the answer arrive faster:
Speculative decoding: you use a small, fast model to propose several tokens at once. The big model validates them in parallel. If it gets them right, you just saved several steps.
Speculative decoding in one sentence: a small model places “bets” on what the big model is going to say. The big one validates the bets in a single pass. If 4 out of 5 tokens are right, you saved 4 decode steps.
Caching frequent prompts: if lots of people ask the same question, you can cache part of the processing.
KV cache reuse: if a conversation continues, you don’t recompute the KV cache for the previous messages.
6. The numbers: how much hardware is “this”?
There is no fixed number because it depends on:
- Model size
- Whether it is dense or MoE (and how many active parameters per token)
- Average conversation length
- Target tokens per second
- Concurrent users
What we do know:
- Inference spending is gigantic. We are talking billions of dollars a year in compute.
- A big chunk is rented from clouds (Azure, AWS, Google Cloud, Oracle).
- Typical deployments for OpenAI/Anthropic workloads are associated with clusters of thousands of GPUs in NVL72 configurations.
To give you an idea: independent analyses estimate that OpenAI spends a 10 digit figure per year on compute, and a very significant part of that is just serving inference.
7. How OpenAI and Anthropic get their capacity
The typical pattern:
- Use public clouds (Microsoft for OpenAI, AWS and Google for Anthropic) to get elastic capacity
- Reserve huge blocks of GPUs long term with contracts worth millions
- Build their own serving layer on top of the cloud infrastructure
Recent news talks about:
- Massive capacity agreements (OpenAI diversifying providers)
- Infrastructure investments specifically for Anthropic
- Construction of dedicated datacenters
Basically, these companies are among the biggest cloud infrastructure customers in the world.
The summary
To serve an LLM to millions of users “in real time” you need:
- Datacenter GPUs with HBM (H100/H200/Blackwell) because what limits you is memory bandwidth
- NVLink interconnect inside the rack so the GPUs behave like a single one
- A fast network between racks (InfiniBand) to coordinate without adding latency
- Power and cooling at scale because we are talking megawatts and liquid cooling
- A serving system that does magic: smart batching, prefill/decode separation, disaggregated serving, speculative decoding, caches…
The hardware is impressive, but the software is what makes the difference. You can own every GPU in the world; if your serving system is mediocre, you will serve a fraction of the users you could.
And that, folks, is how the factory works that lets you ask an LLM stupid questions at 3 in the morning and get an answer in two seconds.
