NPUs, GPUs and the Hardware Driving the AI Boom

Every chatbot answer and every generated image is the product of silicon doing trillions of simple arithmetic operations. Understanding a little about that silicon explains why AI is expensive, why it is getting cheaper, and what to look for in a new laptop.

Why GPUs won

Neural networks are giant matrix multiplications — the same operation repeated billions of times. CPUs are generalists that do a few things very fast. GPUs are specialists that do thousands of operations in parallel. A training run that would take years on a CPU takes weeks on a GPU farm.

That is why the AI boom is inseparable from GPU supply. The chips most people name — NVIDIA's data-centre parts — were designed for graphics and turned out to be perfect for matrix maths. The industry has been riding that coincidence ever since.

Training versus inference

There are two different workloads:

• Training: adjusting billions of parameters over months. Requires enormous compute and memory bandwidth.
• Inference: running the finished model to generate text or images. Lighter, but still demanding — and far more common.

As AI adoption grows, inference dominates total compute. This is why the hardware industry is shifting: specialised inference chips are cheaper to make than training chips, and the market for them is enormous.

Enter the NPU

An NPU (neural processing unit) is a specialised chip for running models efficiently. It is why your phone can translate, denoise photos and run a small language model on-device. NPUs trade flexibility for efficiency: they do a narrow set of operations extremely well, with far less power than a GPU.

Every new laptop and phone now ships with an NPU, and the pitch is "AI features on your device, privately and free". It is real, and it is also a marketing term. The useful question is not whether a chip has an NPU, but what models it can actually run at usable speed.

What to look for when buying

For most people, the practical advice is unglamorous:

• Memory matters most. Local models live in RAM/VRAM. 16GB is workable for small models; 32GB or more is comfortable for 7–14B models.
• Look for an NPU if you want efficient on-device AI — but verify what the software ecosystem supports, not just the hardware spec.
• Do not buy a GPU for hobby prompts. If you just use chat apps, your NPU and the cloud are enough.

Where it is all heading

The trend is unmistakable: compute per dollar keeps climbing, and the marginal cost of AI inference is collapsing. In a few years, running useful models locally on a laptop will be as unremarkable as rendering a webpage. The hardware story is not about bigger chips — it is about the same capability becoming cheap enough to be everywhere.

AI hardware is the story of a capability becoming infrastructure: exotic and scarce, then boring and everywhere.