By Q2 2027, Samsung and Xiaomi will ship flagship devices running a 7-billion-parameter multimodal model entirely on the NPU, sustaining over 30 tokens per second for core assistant tasks without a cloud round-trip.

The silicon is no longer the bottleneck. Qualcomm disclosed sustained NPU throughput above 45 TOPS on the Snapdragon 8 Gen 4, a figure that moves past burst peak performance into the thermal envelope required for continuous inference. MediaTek’s Dimensity 9400 lands in the same zip code. These are not theoretical peaks measured in isolation. They represent throughput available while the modem, ISP, and CPU are active, a constraint that killed earlier attempts at persistent on-device models. The heat dissipation curve now allows a 7B model to run for minutes, not seconds, before throttling.

The model side has shifted faster than the silicon. Google I/O 2025 demonstrated Gemma-2B running locally on a Pixel, but the more telling demo was a quantized 7B model running on a reference design tablet with the screen off, processing voice and camera input in a continuous loop. Microsoft’s Phi-3 family has shown that aggressive 4-bit quantization of a 7B model preserves enough reasoning capability to handle the long tail of assistant requests: multi-step calendar negotiation, context-aware photo search, ambiguous voice commands that require disambiguation. The model does not need to write poetry. It needs to parse intent, retrieve context, and chain API calls. A 7B model at 4-bit precision fits inside a 4GB memory allocation, a carve-out that leaves the Android OS with ample headroom.

The Incentive to Disconnect

Cloud inference costs money on every query. A user who asks their assistant 40 times a day, a plausible number once latency drops and trust rises, generates a monthly inference bill that either hits the OEM’s margin or gets passed to the user in a subscription. Samsung and Xiaomi sell hardware, not tokens. Every query that runs on the NPU is a query they do not pay Amazon, Google Cloud, or Microsoft Azure to process. The marginal cost of an on-device token is the electricity drawn from a battery already sized for screen-on time. The marginal cost of a cloud token is a GPU cluster that never stops depreciating.

Privacy is the marketing lever, but margin is the mechanism. An on-device assistant that never ships raw audio, text, or images off the phone eliminates an entire class of legal liability around data handling. It also eliminates the latency budget spent on TLS handshakes, server queuing, and streaming responses. The assistant feels instantaneous in a way that a cloud model cannot match, even on a fiber connection, because the speed of light to a data center and back adds a floor that on-device processing removes.

Why the Analyst Models Lag

Current consensus models still project mobile AI spend flowing predominantly to cloud providers through 2027. Those models extrapolate from the current state, where on-device models are toys running on repurposed GPU shaders. They miss the discontinuity created by dedicated NPU pipelines that handle the entire transformer forward pass in a single scheduling cycle. When the hardware abstraction layer no longer splits attention layers across heterogeneous cores, throughput jumps by a factor of three to five. That jump has already happened in the 2025 silicon. The 2026 and 2027 tape-outs will refine it, but the threshold was crossed this year.

The OEMs have no reason to announce their intentions early. Samsung’s Galaxy AI marketing currently leans on cloud models because the installed base of Snapdragon 8 Gen 3 and earlier devices cannot run a 7B model locally. The moment the premium tier ships with Gen 4 or its successor, the software update that enables fully local processing becomes a competitive weapon against Apple, which is still navigating its own on-device strategy around a vertically integrated but smaller model architecture.

What Changes

When a flagship phone processes all assistant interactions locally, the data stays on the device by default. The user’s message history, photo library, calendar, and contacts become a private corpus that the model indexes without exfiltration. The assistant shifts from a server-side service that a user accesses to a local capability that the user owns. This inverts the current power dynamic between platform and user. It also resets user expectations. A generation of smartphone buyers will experience an assistant that responds before their finger lifts from the screen. After that, any latency becomes unacceptable. The cloud-dependent competitors will have to explain why their assistant is slower and requires a privacy policy.

What is driving this

  • NPU throughput on premium mobile SoCs has crossed the 40 TOPS sustained threshold required for 7B-parameter models at conversational speed.
  • Model compression techniques have reduced a capable 7B multimodal model to a 4-bit quantized footprint under 4GB, fitting within a reserved memory partition without crippling the OS.
  • OEMs face a zero-marginal-cost incentive to market infinite, private AI as a differentiating feature against cloud-dependent competitors who bear recurring GPU inference overhead.
  • Regulatory pressure in the EU and India is making on-device processing a legal safe harbor against data localization and consent mandates.

What would prove this wrong

A sustained collapse in DRAM density improvements that prevents a 4GB reserved memory partition from fitting inside a flagship BOM cost target, or an unforeseen thermal runaway problem in the 3nm node that forces NPU throttling below 15 TOPS under concurrent workloads.

The signal

Qualcomm Snapdragon 8 Gen 4 and MediaTek Dimensity 9400 benchmarks showing 45+ TOPS NPU sustained throughput plus Google’s on-device Gemma-2B/Phi-3 optimizations already demoed at I/O 2025.