The new shape of intelligent experience

For last few years, every product team building on top of language models for voice experience has been forced to pick a corner of an awkward triangle: you could have fast, you could have smart or you could have cheap; but not all three at once. Real-time voice assistants were small and dumb because anything bigger could not respond inside a human’s patience window. Frontier reasoning models were astonishing in benchmark tables, but their real time voice versions felt lacking. If you found something that was intelligent and fast, it was also very expensive. In addition to experimenting with natively multimodal models, developers also tried cascading state of the art models together: ASR, LLM, TTS with each layer adding latency, each handoff losing context, each silence between turns reminding you that you were talking to a machine.

The interaction-model and background-model split, as proposed by Thinking Machines (TML), is the first credible answer to that triangle. By assigning the two halves of intelligence to different hardwares on different latency budgets, the architecture sidesteps the trade-off rather than negotiating with it. The half that has to be present is allowed to be small, fast, multimodal, and continuously attentive — five times a second, no exceptions. The half that has to be smart is free to be enormous, slow, deliberative, and tool-using, because it is never on the critical path of the 200ms cadence. Neither side is asked to compromise on what it is good at. The user experiences both, fused.

What does that unlock concretely? Consider the experiences that have been technically promised for a decade and have never quite worked:

1. A language tutor who can listen to you stumble through Spanish, gently correct your pronunciation mid-sentence, and simultaneously plan a six-month curriculum tailored to where you actually struggle. The interaction model handles the listening and the correcting in real time. The background model is somewhere else, watching your error patterns accumulate, deciding when to introduce subjunctive mood, drafting tomorrow’s lesson while you finish today’s.

2. A cooking companion that watches your hands through your phone camera, calls out that the onions are about to burn, and pulls up a substitute when you realise you’re out of cumin — without any of those interactions feeling like separate apps. The visual perception is handled in the 200ms loop. The recipe knowledge, the substitution reasoning, the meal-planning context lives in the background.

3. A therapy or coaching presence that can simply be there in long sessions, patient, responsive, never lagging; while the background model does the genuine work of remembering what was said three weeks ago, noticing a pattern across months, deciding when a gentle reframe is appropriate.

4. A workplace assistant that is actually in the meeting with you, not transcribing it for later. It can interject with the relevant figure when someone mis-cites a number, surface the Slack thread that supersedes a decision being re-debated, and do all of that without the awkward “OK assistant, are you there?” preamble that makes today’s tools feel like talking to a Roomba.

The pattern is the same in every case: the experience becomes natural when being there and being smart stop competing for the same compute budget. Once you stop forcing one model to do both, the entire UX problem dissolves. The model that’s there is allowed to be specialised for presence. The model that’s smart is allowed to be specialised for thought. They cooperate over a thin interface, and the user perceives a single attentive intelligence that, finally, does not punish them for talking the way humans talk.

This is what people mean when they say voice is the next interface. It was never really about voice. It was about whether intelligence could show up in the room without making you wait for it. The split architecture is the first proof that it can.

Finally we can have an ever present intelligent, responsive compassionate assistant like that showed in 2013 Science Fiction romance movie “Her”. This blog will discuss key technological milestones that led us to this point.

Welcome Thinking Machine’s Interaction Model

I have been mesmerised by the demos coming out of Thinking Machines interaction model. It is the same kind of mesmerisation I felt the first time I saw Kyutai’s Moshi, and then again when Sesame dropped (try the preview on their site — it is worth the click). We are getting closer to “Her”. Top tier intelligence, with high interactivity and compassion.

What is striking is that both Sesame and Thinking Machines’ new interaction model lean heavily on ideas pioneered by Kyutai’s Moshi. If you want to understand how Thinking Machines manages to combine frontier-grade intelligence with sub-second responsiveness, the most useful prerequisite is Moshi and its retrieval-augmented successor MoshiRAG.

The Thinking Machines (also referred to as TML) blog post is however sparse on how their interaction model interacts with their background model that does the asynchronous heavy lifting, and that is the gap this blog tries to fill.

I also call out differences between TML’s approach and that of Moshi’s on input side.

I also want to take a moment to thank Kyutai lab who build and open sourced Moshi and also to thank Thinking Machines team for taking the torch further and bringing us closer to “Her”. As discussed, the kind of applications it is going to enable are going to be profound. This one is one cool example:

Major milestones on the way

Most language models still treat conversation as a turn-based protocol: you finish, then I speak, then I finish, then you speak. That works for chat boxes. It falls apart the moment you want a model to translate live, count your push-ups, or stop you mid-sentence when you have made a factual error. Real conversation is full-duplex, because both parties produce and perceive at the same time, and silence and overlap carry as much information as the words.

Three lineages have been pushing at this problem from different angles. Moshi rebuilt the speech LM stack around a dual-channel architecture. Sesame scaled it to a level where it started feeling magical. MoshiRAG bolted on asynchronous retrieval to fix factuality without sacrificing real-time flow. And Thinking Machines’ interaction model takes both ideas and scales them to a 276B MoE backbone, to video, and to a deliberately tiny 200ms micro-turn cadence.

1. Moshi: the dual-channel foundation

Kyutai’s Moshi (2024) is a 7B parameter speech-text foundation model built on an RQ-Transformer (the paper). There are two transformers stacked together:

• A “temporal” Transformer running at 12.5 Hz, processing one frame every 80ms.

• A “depth” Transformer that, at each temporal step, predicts 8 audio codebook tokens — Moshi’s actual speech output, encoded by the Mimi neural codec.

The key trick is that Moshi is dual-channel: it ingests the user’s speech tokens and predicts its own speech tokens simultaneously, in interleaved streams. There is no voice-activity-detection harness deciding when it is “the model’s turn” — at every 80ms frame the model is free to speak, stay silent, backchannel, or interrupt. This is the architectural foundation that everyone in the modern full-duplex space, including Sesame builds on. Thinking Machines changes it a bit that we will see in detail.

2. Sesame: refining the experience

Sesame’s Conversational Speech Model takes the same full-duplex spirit and pushes hard on voice quality and emotional naturalness. The public technical details are slimmer than Moshi’s (which has an open paper), but the design lineage — continuous bidirectional audio streams, no separate VAD harness, low-latency codec output — is unmistakably Moshi-style. If you have tried the preview, the demo experience speaks for itself.

3. MoshiRAG: thinking while talking

Moshi has one weakness that is hard to engineer around at 7B parameters: factuality & intelligence. Speech LMs are trained on vastly less text (in word count) than text-only LMs, and it shows on QA benchmarks. The obvious fix is to make the model bigger, but a bigger model cannot run in real time. Kyutai’s answer, published in MoshiRAG in April 2026 (the paper), is to split the system in two:

• A front end that stays small and real-time (this is similar to TML’s interaction model) — Moshi 7B, a 1B streaming ASR, and a frozen reference text encoder (ARC-Encoder).

• A back end that handles knowledge retrieval asynchronously (this is similar to use of a background model by TML) — a local LLM like Gemma 3 27B or an API call to GPT-4.1.

The two are connected by a single new token. When Moshi sees a knowledge-intensive question coming, it predicts a <ret> token in its text channel. That fires off a retrieval call. Crucially, Moshi does not wait for the answer — it keeps speaking, producing a “pre-RAG” lead like “Let me think about that...” or “In the Netflix series...” that buys the back end roughly two seconds of slack. By the time Moshi reaches the body of its answer, the retrieved reference is ready.

An ASR model transcribes the user speech and that with useful context is provided to the back end model to do its work.

How the retrieved answer gets back into the stream

This is the part most people get wrong on first reading. The retrieved reference is not injected as user text, nor as model text, nor as audio. It comes back as a sequence of continuous embeddings — produced by ARC-Encoder, compressed 4× so the sequence is short enough to fit Moshi’s 12.5 Hz frame budget, then projected through a single trainable linear layer.

Those projected embeddings are then additively summed into Moshi’s input vector at the temporal Transformer level, over a short window. Mathematically, the input that was originally:

becomes:

for the duration of the reference’s compressed embedding sequence, and reverts to plain hi​ afterwards. The reference is a transient side-channel that biases Moshi toward grounded facts during the body of its turn — nothing more, nothing less.

The authors did try “insertive” injection (treating the reference as extra tokens spliced into the input sequence) and found it scored higher on accuracy. They rejected it anyway, because lengthening Moshi’s input sequence hurts the model’s ability to hold long conversations. Additive injection was the better engineering trade-off.

What changed in Moshi to make Moshi-RAG work

The architecture changes are surprisingly minimal from orginal Moshi. The paper says it outright: “the only modifications from the original Moshi model are the introduction of a special retrieval trigger token <ret> and a reference text encoder.” In practice there are four small additions:

1. A new <ret> token in the text vocabulary, placed in training data immediately before the “lead” portion of every knowledge-intensive turn (using TTS forced alignment).

2. A frozen ARC-Encoder that compresses retrieved text 4× into Moshi’s embedding space.

3. A one-layer trainable linear projection sitting between ARC-Encoder and Moshi.

4. A learnable h_dropout vector that replaces the reference 20% of training time, so Moshi learns to behave gracefully when retrieval fails or is late.

The base Moshi backbone — the RQ-Transformer, Mimi codec, dual-channel design — is untouched. MoshiRAG is fine-tuned from the original Moshi checkpoint with 100k updates at a tiny 2 × 10⁻⁶ learning rate.

4. Thinking Machines’ Interaction Model

The TML blog post is up-front about the lineage: “This approach builds upon prior work like Qwen-omni, KAME, MoshiRAG.” But the scale and the design choices have moved on substantially. The demoes are incredible, to say the least. Here is on more example. You can see more in the blog.

Time-aligned micro-turns. Where Moshi works at 12.5 Hz (80ms frames), TML works in 200ms chunks. Each chunk interleaves input and output across all modalities — audio, video, and text. There are no turn boundaries, no VAD, no harness. We will talk more about it in the next section.

Encoder-free early fusion. On the input side to Interactive Model, the blog has following details: Audio enters as dMel features through a light embedding layer. Video enters as 40×40 patches through a small hMLP. Audio output uses a flow head rather than a discrete codec. All of it is co-trained from scratch with the transformer. The 12B active parameters deal with all the modalities.

The background model split. This is the part that most directly echoes MoshiRAG. The interaction model handles real-time presence. When a task needs deeper reasoning, planning, or tool use, the interaction model delegates to a background model that runs asynchronously, then weaves results back into the conversation as they arrive — at a moment “appropriate to what the user is currently doing, rather than as an abrupt context switch.” That last phrase is exactly the MoshiRAG philosophy: hide the retrieval latency inside the natural rhythm of speech. One call out: where MoshiRAG bolts on a separate Whisper-class streaming ASR (1B parameters, frozen), TML does not explain whether they use an ASR model or not to provide input to the background model, but it is quite likely they do.

What the Thinking Machines blog does not tell us is how results come back into the interaction stream. MoshiRAG’s answer — additive embedding injection at the temporal Transformer level — is documented in the paper. TML’s mechanism is not. But the surface-level analogy is striking enough that it is reasonable to guess the implementation belongs to the same family.

Thinking Machine’s approach to model input:

Here there is a difference from Moshi

Moshi: combined at every frame. As we discussed earlier, Moshi’s temporal Transformer receives, at each 12.5 Hz time step, one input vector that is the element-wise sum of three streams: model text, model speech, and user speech embeddings (Equation 1 in the paper). MoshiRAG adds a fourth optional term for the reference embedding during the injection window. There is no notion of “input now, output later” within a frame. At every 80 ms tick the model simultaneously sees the user’s incoming audio and predicts its own next text and audio tokens. Three concurrent streams collapse into one combined vector per step.

TML Interaction Model: interleaved at chunk granularity, combined within a chunk: Thinking Machines runs at 200 ms chunks (5 Hz). The blog illustration is explicit about the macro structure — “the model receives a single interleaved token sequence” — and shows it as input_0 → output_0 → input_1 → output_1 → input_2 → output_2 → …. So at the between-chunk level, the model alternates: it consumes 200 ms of input, then emits 200 ms of output, then consumes the next 200 ms of input, and so on. That is sequence-based, not parallel.

But within a single 200 ms chunk, multiple modalities are still combined, just in a different way. The architecture diagram in the blog shows text frames, audio dMel features, and 40×40 video patches (encoded via hMLP) all flowing into model input after bring embedded.

Size comparison

TML Interaction Model’s Inference optimisations

The blog is pretty brief on this. Here is my read: the 200ms micro-turn design fundamentally changes what inference looks like. Standard LLM serving is optimised for big prefills followed by long decodes; TML produces a continuous stream of tiny prefill+decode rounds, five per second per active session. Per-turn overhead dominates the math, so most of the optimisation work is about killing that overhead.

Streaming sessions. Instead of treating each 200ms chunk as a fresh request that re-allocates buffers and rebuilds metadata, the inference server keeps a persistent sequence in GPU memory and appends incoming chunks to it. A version of this is upstreamed to SGLang.

Custom MoE kernels. TML-Interaction-Small is a 276B MoE with only 12B active. At interaction time the per-step batch is tiny, so the standard grouped-GEMM kernel that MoE inference usually relies on wastes most of its throughput. Thinking Machines replaced it with a GATHER+GEMV strategy — gather small per-token activations to the right experts, then run GEMV rather than GEMM — drawing on similar work in PyTorch’s gpt-fast and Cursor’s “warp decode.”

Latency-tuned kernel shapes. Beyond MoE, the kernels are explicitly tuned for the unusual shapes of bidirectional serving — where prefill and decode happen continuously at small sizes rather than in the canonical “big prefill, long decode” pattern.

The post is notably quiet on quantisation, speculative decoding, and KV-cache compression.

TML Background Model’s Inference optimisations & Hybrid Architecture

While the blog does not state what background model TML team used, it is probably a state of the art model. If you try to use an off the shelf LLM API like Sonnet 4.6 or Opus 4.7 you typically get 50-100 tokens per second per user. However, that is not enough for generating tool calls, executing them, preparing a response and handing it back to the interaction model. In such scenarios Cerebras like performance can make a material difference to serve SOTA intelligence in the real time. Cerebras can serve 1500+ tokens per second for SOTA open source models.

Furthermore, one could deploy a hybrid architecture wherein interaction model runs on an edge server or on a server in a datacenter, and it connects to the background model over a fast network (within the same location or over the internet). Even if the connection to the background model is lost, the system will continue to work as interaction model has enough intelligence to carry on a conversation. The TML blog is not 100% clear on how the models interact. If it is text out, text in, it is easy. But if there is tighter integration involving activations, then it may require higher bandwidth (though I don’t think this is what is happening. It is most likely text out, text in). Also, for now Cerebras does not have built in web search from what I know, so there will need to be an intermediate agent - which would also need to be fast.

Ambient AI

This is a privacy preserving setup and may be desired by many as interaction model is exposed to voice and video - which may haev sensitive content. This approach can safely enable “Ambient AI”. Ambient AI, often referred to as ambient intelligence (AmI) or ambient computing, represents a shift in artificial intelligence from reactive (waiting for prompts) to proactive and invisible (acting on context).

In my mental model if coding is 1X of token use, AI co-working is going to be 10X of token use, and Ambient AI is going to be 100X of token use. This two model architecture pattern and chipset like Cerebras are vital to enable it.

Trivia

One of my favourite blogger Stratecherry believes Cerebras main use case is not Coding, but such real time interaction use cases in voice (Ambient AI) and possibly robotics domain. I believe Cerebras - in addition to such performance sensitive applications - is also very important for coding, as well as, personal AI agents use cases when users are in driving seats. Stratcherry believe background agents will replace all the coding and personal agent use cases. I believe, users will want to be in the loop forever, not because they have to, but because they want to - just like people want to browse their social media themselves. It is immensely satisfying - plus it can compound the output! But, that is a debate for another time…
You can also read this excellent deep dive on Cerebras by Semianalysis.

Kyutai, take a bow!

Image Source: Techcrunch

Kyutai was launched in November 2023 at Station F in Paris, a non-profit AI lab seeded with roughly €300 million from Xavier Niel (Iliad), Rodolphe Saadé (CMA CGM), and Eric Schmidt's foundation, with a mandate that is still rare in the field: do frontier research, publish everything, release the weights, and treat open science as the default rather than a marketing posture. Six scientists took the stage that morning, and their CVs read like a quiet history of European generative AI.

1. Patrick Pérez, who became CEO, had spent decades at Inria, Microsoft Research, Technicolor, and Valeo, where he ran the valeo.ai lab.

2. Hervé Jégou had helped found Meta's FAIR Paris office and is the mind behind product quantization and the FAISS vector-search library that quietly underpins much of modern retrieval.

3. Edouard Grave came from FAIR's language modelling group.

4. Laurent Mazaré arrived from DeepMind by way of Jane Street.

5. Neil Zeghidour and Alexandre Défossez, both veterans of Google Brain, DeepMind, and Meta, had between them already invented many of the neural audio codecs the field now takes for granted — SoundStream, EnCodec, MusicGen, AudioLM.

When this team turned its attention to full-duplex speech, the result was Moshi: a system that did not just compete with the cascaded pipelines of the day but made them look like a category error. MoshiRAG followed in 2026, extending the same architecture with asynchronous retrieval while keeping the open-science commitment intact — every detail down to the training-time retrieval-delay sampling distribution is in the paper. In late 2025, Zeghidour, Mazaré, and Défossez spun out Gradium to commercialise the voice work while keeping the lab close, a quietly elegant solution to the hard economic question of how a non-profit funds frontier compute.

While Thinking Machine should get massive well deserved credit for bringing us this far, Moshi paved the way for this revolution. The lineage matters. So does the lab that chose to publish it. Many thanks to the team behind Kyutai!!!

Closing thoughts

Moshi proved you could ditch the cascaded ASR–LLM–TTS pipeline and get a real full-duplex speech model. MoshiRAG proved you could keep that real-time front end and still get factuality by farming knowledge work out asynchronously.

Thinking Machines is now proving you can scale the same two-tier idea to a 276B/12B Active MoE backbone, add video, and still hit sub-second turn-taking latency and basically build your HER. You could add more capable agents between the interaction model and the background model and create phenomenal applications that will feel much more pleasant to interact with. This will also create a market for Cerebras style inference accelerators. And it is a big market.

TML’s interaction model is incredibly impressive, and as they say: “this is the worst it will ever be”!

Also, if you haven’t watched the movie Her, let me tempt you by showing this trailer.

Moore’s Law — and why it’s over - and what comes after

In 1965, Intel co-founder Gordon Moore observed that the number of transistors on a chip doubles roughly every two years at constant cost, a trend that held for five decades and powered the entire digital revolution. But Moore’s Law was never a law of physics; it was an economic observation, and it depended on two things: transistors getting smaller, and smaller transistors being cheaper. Both have stopped. On the cost side, the price per transistor plateaued at the 28nm node around 2014 and is now rising: TSMC’s 2nm node, shipping in 2026, will be the first major node where cost per transistor goes up, not down. On the physics side, we’ve hit the atomic wall: at the 2nm process, critical transistor features are roughly 10 atoms wide. At TSMC’s upcoming 1.4nm node (2028), that shrinks to about 7 atoms. You cannot build a switch smaller than an atom; this isn’t an engineering problem to be solved, it’s a fundamental limit of matter. Quantum tunneling causes electrons to leak through barriers that should be impenetrable, blurring the distinction between “on” and “off” that all digital computing depends on. The golden era where each generation was simultaneously denser, faster, and cheaper is definitively over.

That is why Jensen Huang (CEO Nvidia) has been saying in his every interview that Moore’s Law is dead! We can no longer shrink transistors at the pace Gordon Moore predicted in 1965. The doubling of transistor density every two years on a single chip has stalled. The number of transistors per GPU tile (we will talk more about tiles later) only increase slightly: 80 billion in Nvidia Hopper tiles, 104 billion in Nvidia Blackwell tiles, and 168 billion in Nvidia Rubin tiles. And yet, performance improved much more!

How? The answer isn’t one trick — it’s a system of engineering innovations across five axes (there is a name to it: extreme co-design):

· More tiles per GPU: tiling multiple dies into one GPU package

· Lower precision: FP8 → FP4 doubles effective FLOPS per tensor core

· More memory bandwidth: advancing HBM generations and adding more stacks

· Superchips: fusing CPU + dual GPUs with coherent chip-to-chip interconnects

· More GPU-to-GPU bandwidth: scaling NVLink and building rack-scale fabrics like NVL72

The Nvidia GPU Generations at a Glance

(To keep the table simple, I haven’t added H200, B300 and Rubin Ultra in the table. B300 offers for example much higher Peak AI TFLOPs for FP4 around 14,000 FP4. Rubin Ultra offers more performance and memory.)

For Rubin , Nvidia claims up to an effective 50,000 TFLOPS of FP4 performance can be achieved with an updated 3^rd generation Transformer Engine that replaces 2:4 structured sparsity from prior generations. Without use of it, it would be around 35,000 TFLOPs.

Rubin has another variant Rubin Ultra, which doubles the silicon to 4 reticle-limit chiplets, pairs them with 16 HBM4e stacks totalling 1 TB of memory, and delivers ~100,000 TFLOPS of FP4 compute with the 3^rd generation Transformer Engine .

Axis 1: More Silicon — Tiling Past the Reticle Limit

The Reticle Wall

Every chip is printed using a photolithography mask called a reticle. The maximum area a single reticle can expose on a wafer is roughly 800 mm² — a hard physical limit of the lithography equipment. The H100, at 814 mm², was already pushing right up against this boundary. You simply cannot make a single chip bigger. This is the wall that killed the naive version of Moore’s Law for GPUs.

Blackwell: Two Dies, One GPU

NVIDIA’s B200 broke through this wall with a dual-die design. Two reticle-limit dies (each ~104 billion transistors on TSMC 4NP) are placed side by side and connected via NV-HBI (NVIDIA High-Bandwidth Interface) — a custom die-to-die interconnect running at 10 TB/s. The two dies appear as a single GPU to software. This is how NVIDIA jumped from 80B to 208B transistors without a dramatic process shrink.

Rubin: Two Chiplets on 3nm

The Rubin R100 GPU also uses a 2-die design, but on TSMC’s N3P (3nm-class) process. Two reticle-limit compute chiplets are connected on a CoWoS-L advanced package, reaching approximately 336 billion transistors. The density improvement from 4nm → 3nm, combined with architectural improvements, is how NVIDIA achieves ~1.6× more transistors than Blackwell with the same number of tiles.

Rubin Ultra (2027): Four Chiplets

Rubin Ultra doubles the silicon again — four reticle-limit chiplets on a single package, using die-to-die bonding with a fast coherent interconnect. Combined with 16 HBM4e stacks (1 TB total), Rubin Ultra roughly doubles the compute of standard Rubin to ~100 PFLOPS FP4.

Key insight: NVIDIA isn’t relying on Moore’s Law. They’re using packaging innovation — putting more dies on a single substrate — to scale transistor count far beyond what any single die could hold.

Below images can help you understand this point visually!

For the Blackwell GPU, you can notice two tiles: one on the left and one on the right. Area is split approximately equally between the two.

For the Rubin GPU as well, you can notice two tiles: one on the left and one on the right. Area is split approximately equally between the two.

Axis 2: Lower Precision — How FP4 Doubles Effective FLOPS

One of the most powerful — and least understood — levers NVIDIA pulls each generation is reducing numerical precision. This is not about making calculations less accurate. It’s about doing more useful work per clock cycle by using smaller numbers where full precision isn’t needed.

The Precision Ladder

Neural networks are remarkably tolerant of reduced precision. The weights and activations in a trained model don’t need 32-bit or even 16-bit floating point to produce correct results. NVIDIA has exploited this insight aggressively:

Why Halving Precision Doubles FLOPS

The math is straightforward: when you halve the number of bits per value, the tensor cores can process twice as many values per clock cycle through the same silicon. A matrix multiply that operates on 8-bit values can crunch exactly twice as many elements per cycle as one operating on 16-bit values, using the same physical hardware.

NVFP4: Not Just Smaller Numbers

NVIDIA’s NVFP4 format isn’t a naive truncation. It uses a sophisticated dual-level scaling scheme:

· Values are grouped into small blocks (e.g., 32 elements)

· Each block gets its own FP8 scaling factor that captures the local dynamic range

· Individual values within the block are stored as 4-bit floats (1 sign bit, 2 exponent bits, 1 mantissa bit)

· The combination of per-block scaling + 4-bit values preserves accuracy within ~1% of FP8 on most LLM tasks

This is why NVFP4 works for both inference AND training — the scaling factors compensate for the limited range of 4-bit values.

The Double Benefit: FLOPS + Memory

FP4 doesn’t just double compute throughput — it also halves the memory footprint of model weights compared to FP8:

· A 70B parameter model in FP16: ~140 GB (doesn’t fit on one H100)

· Same model in FP8: ~70 GB (fits on one H100 with 10 GB headroom)

· Same model in FP4: ~35 GB (fits on one B200 with 157 GB free for KV-cache and batching)

Smaller weights mean more of the GPU’s HBM is available for KV-cache (critical for long-context inference) and larger batch sizes (critical for throughput). This is why NVIDIA claims up to 15× inference performance improvement from Blackwell over Hopper at system scale — it’s the compound effect of more FLOPS, more memory, more bandwidth, AND smaller model footprint.

Key insight: Hopper’s lowest native precision is FP8. Blackwell and Rubin add native FP4 support, which effectively doubles the useful FLOPS per tensor core without adding any transistors. This is a “free” 2× multiplier on top of all the other scaling axes.

Axis 3: More Memory Bandwidth — The HBM Scaling Playbook

AI workloads are memory-bandwidth-bound. A GPU can have all the FLOPS in the world, but if it can’t feed data to its compute cores fast enough, those cores sit idle. NVIDIA attacks this memory wall by advancing along three dimensions of HBM.

How HBM Works (30-Second Primer)

HBM (High Bandwidth Memory) is a 3D-stacked DRAM technology. Multiple DRAM dies are stacked vertically and connected using through-silicon vias (TSVs) — tiny vertical copper pillars that punch through each layer. The entire stack sits on a silicon interposer right next to the GPU die, connected via thousands of parallel wires.

Three Levers NVIDIA Pulls Each Generation

1. More stacks on the package

Each generation adds more HBM stacks, requiring a larger silicon interposer (CoWoS-L for Rubin).

2. More layers per stack (taller stacks)

· HBM3: 8 DRAM layers per stack

· HBM3e: 8–12 layers per stack

· HBM4: 12–16 layers per stack

· HBM4e: 16-20 (rumoured) layers per stack

3. Wider interface per stack

· HBM3: 1,024-bit interface → up to ~819 GB/s per stack

· HBM3e: 1,024-bit interface → up to ~1.2 TB/s per stack (higher data rate)

· HBM4: 2,048-bit interface → up to ~2.0 TB/s per stack (doubled bus width)

· HBM4e: 2,048-bit interface → up to ~3.0-4.0 TB/s per stack (doubled bus width and throughput)

HBM4 & HBM4e literally doubles the bus width — a fundamental architectural change in the JEDEC spec.

The Compound Effect

This is packaging engineering — more chips, taller stacks, wider bus. Not Moore’s Law.

Axis 4: The Superchip — Fusing CPU + Dual GPUs with NVLink-C2C

Starting with Grace Hopper, NVIDIA introduced the Superchip: a single module that fuses a CPU and GPU(s) together using NVLink-C2C (Chip-to-Chip) — a coherent, high-bandwidth interconnect fundamentally different from PCIe.

In this how Grace Hopper SuperChip looks. On the left you see Grace CPU and on the right you see Hopper GPU.

What is NVLink-C2C?

NVLink-C2C is a memory-coherent chip-to-chip interconnect that physically bonds the CPU and GPU dies on the same module. Unlike PCIe (which requires explicit data copies), NVLink-C2C creates a unified memory address space — CPU and GPU threads can access each other’s memory transparently.

Key properties of NVLink-C2C:

· Memory coherent — CPU and GPU see a single unified address space

· 900 GB/s bidirectional bandwidth (Grace Hopper and Grace Blackwell superchips) — 7× faster than PCIe Gen5

· 1.8 TB/s bidirectional bandwidth (Vera Rubin) — 7× faster than PCIe Gen6

· 5× more energy-efficient than PCIe per byte transferred (~1.3 pJ/bit)

· Enables KV-cache offloading from GPU HBM to CPU LPDDR without performance penalty

Why this matters: In LLM inference, the KV-cache can consume most of the GPU’s HBM. With NVLink-C2C, the KV-cache can spill into the CPU’s large LPDDR memory (480 GB–1.5 TB) at near-HBM speeds, dramatically increasing the effective context window without adding more GPUs. Let us see how these SuperChips look like.

Superchip Architecture for Grace Blackwell and Vera Rubin:
1 CPU + 2 GPUs

Grace Hopper SuperChip had 1 Grace CPU and 1 Hopper GPU, but now with Grace Blackwell and Vera Rubin Architectures there are 2 GPUs with 1 CPU.

Each Superchip connects one CPU to two GPUs via NVLink-C2C. The two GPUs also connect to each other via NVLink. This creates a tightly-coupled compute unit that serves as the building block for larger systems.

Here is a schematic diagram for Vera Rubin SuperChip. Following that we will see how the real SuperChips look like:

GB200 SuperChip: Notice two Blackwell (B200) GPUs in the upper part and one Grace CPU in the lower part.

GB300 SuperChip is a performance advancement over GB200: Notice two Blackwell (B300) GPUs in the upper part and one Grace CPU in the lower part.

Vera Rubin SuperChip: Notice two Rubin GPUs in the upper part and one Vera CPU in the lower part.

Axis 5: GPU-to-GPU Bandwidth — NVLink and the NVL72 Rack

A single GPU cannot train a trillion-parameter model alone. The speed at which GPUs exchange data — gradients during training, KV-cache during inference — directly determines system-level performance.

How NVLink Scales

NVLink bandwidth per GPU = number of links (ports) × bandwidth per link.

Each generation keeps the same 18 NVLink ports per GPU but increases the per-link speed through faster SerDes circuits. From H100 to B200: 2×. From B200 to Rubin: another 2×.

The NVL72: When 72 GPUs Become One

The Blackwell Generation Innovation

Perhaps the most architecturally significant innovation in the Blackwell generation isn’t the GPU itself — it’s the GB200 NVL72 rack system. It connects 72 B200 GPUs and 36 Grace CPUs (36 GB200 Superchips) into a single, liquid-cooled rack where all 72 GPUs communicate at full NVLink 5 speed. The entire rack forms a single NVLink domain with 130 TB/s of aggregate bisection bandwidth. There is a GB300 version of this NVL72 system.

Each rack has 18 trays with 2 SuperChips (4 GPUs and 2 CPUs), this total of 36 CPUs and 72 GPUs. Note however that 18 trays present you with 18 nodes/servers. You can split them for prefill and decode functions. This is how GB200 NVL72 racks looks. As an exercise can you find 9 NVSwitch trays and 18 compute trays?

How GPU-GPU Communication Works In NVL72 system

The rack contains 9 NVLink Switch trays, each housing 2 NVLink Switch ASICs. These 18 switch chips create a non-blocking, all-to-all fabric. Every GPU has 18 NVLink ports wired through the copper backplane to the switch trays, which route traffic to any destination GPU.

Comparing With H100 Based 8 GPU Servers Baseline

Consider this popular topology developed by SGLang team for DeepSeek inference on 12 servers with 8 H100 GPUs each. DeepSeek model has multiple layers (61 to be precise) and each has Attention and 58 have Mixture of Expert (MoE) experts. 90%+ model parameters are in MoE experts.

They reserve 3 nodes for Prefill (prompt processing and KV cache generation) and 9 for Decode (token by token generation). As you can see from the diagram, MoE experts are divided into subgroups across nodes, whereas each node (in fact each GPU) has full attention weights.

When tokens are processed by an attention layer, each of them gets sent to a bunch of experts for that layer. Some of those experts may be on the same server GPUs and some of them will be on GPUs on different servers. Communication for the former happens over NVLink within respective servers, communication for the latter happens over Infiniband which is much slower than NVLink. This introduces a performance penalty!

Source: LMSYS

How NVL72 Based Systems Solve This Problem?

Traditional GPU clusters connect 8 GPUs per node via NVLink, then connect nodes via InfiniBand. Inter-node bandwidth is 10–50× lower, creating a bandwidth cliff. The NVL72 eliminates this cliff for up to 72 GPUs. For mixture-of-experts LLM inference with GB200 based NVL72:

· Expert parallelism across 72 GPUs without bandwidth penalties

· 13.5 TB of unified GPU memory (72 × 192 GB) at NVLink speed

· 30× faster real-time inference on trillion-parameter models vs H100 (NVIDIA’s claim, we will see real world performance numbers later)

Grace Blackwell NVL72: Where It All Began!

If my memory serves right NVL72 was announced in GTC 2024. I highly recommend watching the following video:

Vera Rubin NVL72: The Next Step!

The Vera Rubin NVL72 packs 72 Rubin GPUs and 36 Vera CPUs (36 Vera Rubin Superchips) into a rack delivering 3.6 EFLOPS of FP4 inference. With NVLink 6 at 3.6 TB/s per GPU, the aggregate rack bandwidth reaches 260 TB/s — 2× the Blackwell NVL72. NVIDIA also introduces silicon photonics for rack-to-rack links, enabling POD configurations of up to 576 GPUs.

I highly recommend watching the following video:

Role of GPU-HBM & GPU-GPU Bandwidths

In both training and inference, GPU compute cores spend most of their time waiting for data — the bottleneck is almost never arithmetic, it’s how fast you can feed weights, activations, and gradients to the tensor cores. During training, each GPU must read the full model weights from HBM every forward and backward pass, exchange gradients with every other GPU after each step, and write updated weights back — so HBM bandwidth determines how fast a single GPU can churn through batches, while NVLink bandwidth determines how fast GPUs can synchronise with each other.

The H100’s 3.35 TB/s HBM bandwidth meant a 70B-parameter model in FP16 (~140 GB, H100 has only 80GB memory, but let us ignore that for now) took about 42 microseconds just to read once from memory; the B200’s 8 TB/s cuts that to 17.5 µs, and Rubin’s 22 TB/s brings it under 6.4 µs — directly translating to faster iteration.
NOTE: All weights are not read from HBM at once, but they are loaded by GPUs layer by layer. This exercise tells you cumulative time lost in loading weights.

During inference, the constraint is even starker: autoregressive token generation is almost entirely memory-bandwidth-bound because each token requires reading the full KV-cache and model weights but performs very little compute per byte read — this is why inference throughput scales nearly linearly with HBM bandwidth. On the NVLink side, large models that don’t fit on a single GPU must be split across multiple GPUs using tensor parallelism (splitting individual matrix multiplies) or pipeline parallelism (splitting layers), both of which require GPUs to exchange intermediate activations every few milliseconds — if NVLink bandwidth is 2× higher, you can split across 2× more GPUs before communication becomes the bottleneck, which is exactly why the GB200 NVL72’s 130 TB/s aggregate bandwidth enables mixture-of-experts models with hundreds of experts distributed across 72 GPUs without the communication penalty that would cripple the same workload on an 8-GPU H100 based node connected by InfiniBand. In short: HBM bandwidth sets the ceiling on single-GPU performance, and NVLink bandwidth sets the ceiling on how many GPUs you can scale to before returns diminish — every generation that raises both ceilings directly unlocks larger models, longer contexts, and lower latency.

Are These Innovations Helping?

One of the best places to see the performance difference caused by these innovations is checking out results of InferenceMax benchmark maintained by SemiAnalysis. Scaling to larger number of GPUs in a single domain and change of GPU generation can cause dramatic improvement in performance. Below we compare performance of H200 (8 GPU servers) with that of GB300 (NVL72 server with 72 GPUs) for inference of DeepSeek R1 model.

Quote by SemiAnalysis: At GTC 2024, Jensen said that GB200 NVL72 was 35x faster than Hopper. Nobody believed it and thought it was classic fake Jensen Math. When we tested the performance of it, it wasn't just 35x faster, it was over 50x times faster even against an strong Hopper baseline with all of the inference optimisation composed together like MTP, Disagg prefill, wideEP, etc. View the nuanced results at InferenceX dot com.

Source: SemiAnalysis

2026: The Era of Extreme Co-design

In the latest Nvidia GTC conference (GTC 2026), Nvidia announced five rack types. We will see in upcoming articles the roles of other racks. But for now remember Groq3 LPX helps accelerate decode performance, Vera CPU racks are useful for deploying large number of agents (during training with RL or inference), STX Storage racks are for KV Cache and Spectrum-6 SPX are for networking.

Nvidia truely understands that “the whole can be greater than the sum of parts”.

SUMMARY: Why Do GPUs Keep Getting Faster?

The era of “free” scaling from Moore’s Law is over. What we’re in now is the era of system-level scaling — where performance comes from co-designing the chip, the precision format, the package, the memory, the interconnect, and the rack as a single integrated system. And NVIDIA has mastered this playbook.

Moore’s Law gave us exponential gains for free. The post-Moore era demands exponential engineering effort for each generation. NVIDIA’s bet is that AI workloads are valuable enough to justify that effort — and so far, the market agrees.

APPENDIX

Approach that Huawei is following:

While Chinese chipmakers can not get EUV machine needed for chips below 7nm, they can still follow this extreme co-design approach and build efficient systems for AI training and inference that compensate for not having chips denser than 7nm. That is exactly what Huawei is doing with their Cloud Matrix multi rack solution. You can read more from this SemiAnalysis report. Off-course these systems would need more power, but power is abundant in China.

Nvidia’s western competition like AMD is following this approach as well. So this is the race where the fastest one wins. However, what benefits Nvidia is also being able to work with thousands of engineers from frontier labs that informs them how the next generation of systems need to be built. Otherwise they have no way of knowing. In return engineers build model architectures and inference engines that take advantage of Nvidia ecosystem capabilities. In Jensen’s view China has abundant AI talent. If Nvidia is not active in China, they miss out on all these learnings that are possible by closely working with that talent. Also labs from the region won’t build for Nvidia ecosystem - thus a great American company stands to lose its position.

Dedication:

Dedicating this article to the giant who inspired the industry at an exponential speed Gordon Moore. He has a true successor in form of Jensen Huang (Nvidia CEO) who continue to inspire the industry for exponential improvements.

We want to understand how much compute and data are needed to train the next generation of models which are going to be in 2T, 4T and 10T parameter count classes (rumours!). Anyways, Elon announced today that they are training their models in this class, so this would be a very timely exercise.

Methodology: We use formulas and approaches used in estimating training costs for DeepSeek V3, ARCEE, KIMI 2.5 etc. We will use GPU performance numbers from Nvidia Data-sheets for B200/B300 and NVL72 racks, and statistics on data availability from Epoch AI. When we say B200/B300 we are actually referring them in GB200/GB300 superchips, as a part of NVL72 racks. NVL72 has world size of 72GPUs, it makes it easier to train models in this class as those 72 GPUs can access each others HBM at high throughputs through NVLink/NVSwitch connectivity. That said data parallelism alone is not enough. You still need other parallelism (pipeline, tensor, sequence). To account for all the bottlenecks, we assume MFU (Model Flop Utilisation) of only 20%.

1. Assumptions

2. Training Data Requirements

Formula:

D = tokens_per_parameter × N_total

D = 40 × N_total

Token count is based on total parameters — all expert weights must be trained across the full run, even though only 5% activate per token.

The publicly crawlable internet provides ~100T tokens (Epoch AI analysis. I have highlighted 100T tokens mark below). Models requiring more must rely on synthetic data generation, rephrasing, or multi-epoch training. As we will see later, synthetic data generation becomes very costly as we approach 10T parameter class.

3. Sparsity Ratios of Frontier MoE Models

To justify the 5% active ratio, we surveyed the latest frontier models:

Why 5% Is a Sound Assumption

· DeepSeek V3/V3.2 (5.4%), GLM-5 (5.4%), and MiniMax M2.5 (4.3%) all cluster around 4–6%. 5% is the center of this range.

· Kimi K2.5 already achieves 3.1% — the trend is toward more sparsity. 5% is conservative for a next-gen model.

· Older models (Mixtral, DBRX) used ~27% active. In 18 months the field moved to 3–6%, a decisive shift.

· DeepSeek V3 trained successfully at 5.5% active on 14.8T tokens with no quality loss vs. dense baselines.

· 5% active yields practical sizes: 2T→100B, 4T→200B, 10T→500B active per token — all proven architecture scales.

4. Training Compute (FLOPs)

With MoE, only the active parameters compute per token:

N_active = N_total × 0.05

C = 6 × N_active × D

C = 6 × (N_total × 0.05) × (40 × N_total)

C = 12 × N_total²

5. GPU Specifications

B300 only outperforms B200 at FP4. At BF16, they are identical (2.25 PFLOPS).

6. Calculation Chain

Step 1: Effective_FLOPS = Peak_FLOPS × 0.20

Step 2: GPU_seconds = C ÷ Effective_FLOPS

Step 3: GPU_hours = GPU_seconds ÷ 3,600

Step 4: GPUs_needed = GPU_hours ÷ 2,160 (= 90 days × 24 hrs)

Step 5: NVL72_Racks = GPUs_needed ÷ 72

Step 6: Total_Cost = GPU_hours × $/GPU/hr

7. FP4 Scenario

Theoretical best case — all FLOPs at FP4 throughput.

From what I know it is not possible today, or not proven today at this scale. Nvidia has done it for smaller models like 12B with 4 bit precision training. But, even there they maintained many other things in higher precision. To quote: “Attention, embedding, non-linear layers, and other tensors: To ensure numerical stability during training, we retain the original precision (e.g., BF16 or FP32) for embeddings, the output projection head, normalization layers, non-linearities, and attention components, including softmax and the query-key and attention score-value batched GEMMs. The main weights (stored by the optimizer), weight gradients (used for gradient accumulation across microbatches and across data-parallel replicas), and optimizer states are also kept in FP32. Tensor parallel reductions are performed in BF16 precision.”

B200 (FP4) — $5/GPU/hr

B300 (FP4) — $6/GPU/hr

8. BF16 Scenario

This is a more realistic scenario that they did all their training in BF16.

B200 (BF16) — $5/GPU/hr

B300 (BF16) — $6/GPU/hr

9. Cost Comparison Summary

Total Training Cost

NVL72 Racks Required

10. Training Cost Visualisations

Figure 1: Training Cost by Model Size and GPU Configuration

Figure 2: NVL72 Racks Required

11. Synthetic Data Generation Cost

The 10T model requires 400T tokens but only ~100T are available from the internet. The remaining 300T must be generated synthetically. We analyze the cost using Claude Sonnet 4.6 as a representative frontier API model.

Sonnet 4.6 Pricing

Cost to Generate 300T Synthetic Tokens

Assuming a 1:10 input-to-output ratio (30T prompt tokens → 300T generated tokens):

Standard: (30T ÷ 1M × $3) + (300T ÷ 1M × $15) = $90M + $4,500M = $4.6B

Batch: (30T ÷ 1M × $1.50) + (300T ÷ 1M × $7.50) = $45M + $2,250M = $2.3B

(off-course in case of Anthropic - it would cost them much lesser than outsiders due to them not having to pay their own margins :) )

12. Total Cost: Training + Synthetic Data (10T class)

Figure 3: 10T class Cost Breakdown — Training vs Synthetic Data

Practical alternatives to save on synthetic data costs

· Self-hosted inference: Running a Llama-class (MiniMax etc would be good - as they have fewer active parameters and robust performance) model on owned GPUs costs ~$50-100M for 300T tokens — 20-40× cheaper than Sonnet 4.6 API.

· Multi-epoch training: Train on the same 100T tokens 4× with curriculum learning. Zero data generation cost, though diminishing returns after 2-3 epochs.

· Rephrasing/augmentation: Take 100T internet tokens and rephrase 3-4× using a cheap model. Much cheaper than generating from scratch.

· Hybrid approach: 100T internet + 50T high-quality synthetic + 3 epochs approximately 400T effective tokens at minimal generation cost.

13. What about post training?

We mostly focus on pre-training and synthetic data generation cost. Post training (SFT + RL) can cost additional 20% (Epoch AI’s analysis of DeepSeek R1). xAI have reported spent almost as much compute on post training for Grok 4 as on pre-training of the model.

Key Findings of this thought exercise

2T class is very achievable: 191 NVL72 racks, $148M at BF16. NeoLabs can still manage it. Data requirement (80T) is within internet availability.

4T class is feasible: 762 racks, $593M at BF16. Needs ~60T synthetic tokens to reach total of 160T tokens — manageable but starts getting very expensive.

10T class is within reach with hyper-scaler partners and truck load of cash: 4,763 racks, $3.7B at BF16. But 300T synthetic tokens add $2.3B (API) or ~$75M (self-hosted cheaper models).

Synthetic data can cost as much as training: At API pricing, 300T tokens via Sonnet 4.6 costs $2.3B — it is almost 60% of $3.7B training cost for 10T class model on B200 with BF16 precision. Self-hosted generation may become essential at this scale.

B300 only wins at FP4: At BF16, identical throughput to B200 but 20% more expensive. As I understand, Nvidia positions B300 for inference, rather than training.

Data is a major bottleneck: Not compute or GPUs alone— the 400T token requirement for 10T models is probably the hardest constraint to satisfy!!!!

Disclaimer

This is just a thought exercise to discuss order of magnitude compute needed for next generation of models. I have used all the public formulas and performance numbers. Thanks!