Most software engineers treat the chip as an abstract boundary. You ship your PyTorch model, pray to the CUDA gods, and hope the silicon does its job. But as AI workloads strain against the limits of Moore’s Law and memory bandwidth, the abstraction is leaking. Understanding what happens below the compiler has become a practical advantage, not just intellectual curiosity.
Reiner Pope, CEO of MatX and a former Google TPU architect, delivers one of the clearest explanations we’ve seen of how chips actually execute computation. He starts with AND and OR gates and builds, layer by layer, to a multiply-accumulate unit. From there, he explains why data movement—not arithmetic—dominates cost, how systolic arrays amortize that cost, and why a GPU is essentially a fleet of tiny TPUs. The lecture is dense but never indulgent. Pope connects each architectural decision to a concrete tradeoff: determinism versus flexibility, area versus throughput, cache versus scratchpad. If you’ve ever wondered why an H100 looks the way it does, this is your answer.
Key Takeaways
- A multiply-accumulate (MAC) unit—the atomic operation of AI chips—can be built from scratch using only basic logic gates, making the full hardware stack legible.
- Data movement dominates chip cost and energy, not computation; multiplexers and register files are where the real silicon area and latency hide.
- Systolic arrays achieve efficiency by minimizing register-file reads: data flows through adjacent MAC units directly, amortizing memory access across many operations.
- FPGAs beat CPUs for real-time control because they offer deterministic clock-cycle behavior, while CPUs suffer non-determinism from caches, branch prediction, and OS interrupts.
- A GPU is architecturally a collection of many small TPU-like compute units, which explains why GPU cores are much smaller than CPU cores—they sacrifice single-thread latency for massively parallel throughput.
Who should watch: ML infrastructure engineers and systems architects who want to understand the hardware tradeoffs shaping GPU, TPU, and FPGA design—and why those tradeoffs directly impact model performance and cost.
Why This Matters
Pope’s lecture makes the case that the next wave of AI infrastructure differentiation won’t come from higher-level frameworks but from co-designing models with the silicon they run on. As frontier labs hit the limits of GPU supply and power density, the engineers who can reason about systolic arrays and scratchpad memory will have an edge.
