This episode is a rare artifact: two published researchers who build and ship models arguing specific technical constraints rather than gesturing at trends. Tim Dettmers (Ai2) and Dan Fu (Together AI) go deep on what's actually breaking in GPU scaling and what's replacing it.
The core tension: Dettmers argues from physical constraints — the von Neumann bottleneck means memory bandwidth isn't improving, which caps model utilization (MFU) regardless of how many FLOPs a chip advertises. You can buy more compute, but you can't feed it fast enough. Dan Fu counters from a systems perspective — kernel optimization, alternative architectures like state-space models, and agent-driven automation are squeezing more useful work out of existing hardware. Both agree the easy scaling era is over; they disagree on how much headroom remains.
The conversation lands on concrete, actionable territory. Agents writing GPU kernels isn't speculative — it's happening in production, and the economics already favor it for many workloads. State-space models are shipping, not just appearing in papers. Hardware decisions are fragmenting: NVIDIA still owns training, but inference workloads are splintering across AMD, Cerebras, Groq, and custom silicon. The practical takeaway is that infrastructure teams need to stop treating GPU scaling as a single curve and start optimizing for specific workload profiles — training, inference, and agent orchestration each have different bottlenecks and different hardware matches.
There's no hype here. Just two experts disagreeing productively about where the real constraints are and what to do about them.
Key Insights
- GPU scaling is hitting a von Neumann bottleneck where memory bandwidth improvements have stalled, meaning raw compute will continue increasing but data movement between memory and compute units won't keep pace — this fundamentally caps utilization (MFU) on large models regardless of FLOPs on paper.
- Agents have crossed a practical capability threshold: Dan Fu confirms that agent-generated CUDA kernels now match or exceed hand-tuned implementations in production, and the economics already favor agent-driven kernel optimization over manual engineering for many workloads.
- State-space models (SSMs) like Mamba are not research curiosities — they are in production at Together AI and other shops specifically because they sidestep the quadratic attention bottleneck, offering linear scaling in sequence length that transformers cannot match without architectural compromise.
- Hardware diversification is accelerating along functional lines: NVIDIA dominates training, but inference is fragmenting across AMD, Cerebras, Groq, and custom ASICs — and the decision of which chip to use is increasingly workload-specific rather than a default to NVIDIA.
- Small specialized models fine-tuned for narrow agent tasks are outperforming large general-purpose models on cost-per-task metrics, and this trend is accelerating as agent orchestration layers improve at routing to the right model for each subtask.
- The 'scaling is dead' narrative is wrong but the 'scaling continues unimpeded' narrative is also wrong — the reality is a bifurcation where some dimensions (raw compute) still scale while others (memory bandwidth, interconnects) have plateaued, forcing architectural creativity rather than brute force.
Who should listen: Infrastructure engineers and ML platform leads making hardware procurement or model architecture decisions for production inference and agent workloads.
Why This Matters
This episode validates a pattern we're tracking: the infrastructure stack is fracturing along workload lines. Training, inference, and agent orchestration are becoming distinct engineering problems with different hardware and software solutions — the era of one-stack-fits-all is ending. Practitioners who treat GPU scaling as a monolith will make expensive procurement mistakes.
