大语言模型推理加速:硬件视角的 (English)

Generated: 2026-06-20 09:04:58

• -

The Computing Power Anxiety Hides a Breakthrough You Haven't Noticed!

You've probably heard too many stories about "computing power anxiety" by now.

Engineers pacing back and forth in data centers, listening to GPU fans screaming, watching VRAM spike into the red—and yet those damn Tokens still come out one sluggish word at a time. They dream of making large models "speak human" more fluently. Think about it: trillion parameters, million-token context windows—if you're only squeezing out a few words per second, how are you supposed to do real-time interaction?

And here's the thing. By 2025, a quiet consensus has formed in the industry—a deeply counterintuitive insight:

You think the winner is the one with the most computing power? Wrong.

What really separates the pack is no longer how many billions of parameters you have. It's whether the three brothers—compute, memory, and network—can work together at the system level. Otherwise, no matter how expensive your GPU is, it's just spinning its wheels, waiting for that "slow variable" it can never catch up to.

Look, the hardware world is shifting from "one god to rule them all" to a rising tide of many contenders. GPU, NPU, LPU, FPGA... Who's the one that will make you exclaim, "So that's the answer"?

• -

GPU: The Ultimate Powerhouse That Still Hits a Wall

Let's start with the reigning champion—NVIDIA's GPU.

It's strong. How strong? It practically monopolizes the world of large model inference. But think again. This thing everyone calls "amazing" has a fatal weakness: memory bandwidth bottleneck.

Why? Because large model inference is autoregressive decoding—it spits out one Token at a time. Every single time a Token is generated, it has to read the entire model's hundreds of billions of weights—along with the historical KV Cache—out of that HBM memory. Doesn't that sound like some glutton who only needs one grain of rice, but insists on opening the whole granary door and hauling it all inside? Clunky, inefficient, and ready to explode at any moment!

The compute cores sit there staring into space, utilization pathetically low.

Of course, the industry is scrambling to do damage control. vLLM came up with Graph Capture. SGLang introduced CPU overhead hiding at the end of 2024—basically, hiding that sluggish GPU kernel launch time under the rug. But let's be honest—these are tactics. The physical ceiling of memory bandwidth? No one has managed to break through it yet.

• -

NPU and LPU: Two Extremes—One That Amazes You, One That Makes You Think

Since GPUs have this flaw, are there any specialists out there?

Yes, there are.

First up: NPU (Neural Processing Unit). Think Apple's Neural Engine, AMD's AI Engine. These things are born purely for matrix multiplication. How impressive? At just 7 watts of power, their performance-per-watt can leave GPUs in the dust!

But guess what? They have a headache-inducing problem: too rigid.

The programming model isn't flexible enough. When it hits dynamic logic control or complex multimodal reasoning, it starts to struggle. Like an athlete who can only do one move—explosive power, sure, but ask them to play a full game, and they run out of tricks.

The second option is much more radical—Groq's LPU (Language Processing Unit).

When it came out in 2024, it shook the entire field. What did it do? It completely ditched traditional memory.

Think about it. Everyone else is scrambling to use HBM, DDR... but Groq? It abandoned DRAM entirely and crammed all the model parameters into tightly coupled SRAM. This is a deterministic computing architecture—no waiting, no cache misses. The result? Single-Token latency? Under 1 millisecond. Lightning fast!

The cost? SRAM is expensive. And capacity is small. For now, it can only run medium-sized models. Fast, small, and painfully expensive—how would you choose in the real world? That's the question it has to face.

• -

FPGA: The Magician Everyone Has Underestimated

By now, you might think the race has already been carved up between GPUs and LPUs.

Wrong. Dead wrong!

There's another player hiding in the corner—FPGA.

See, while everyone is chasing extreme parameters and extreme computing power, FPGA took a completely different path: flexibility and energy efficiency.

In 2024, a paper at the FCCM conference dropped like a bombshell. It systematically analyzed the potential of FPGA for accelerating LLMs. It can tailor a custom dataflow pipeline for your specific model, like building with LEGO bricks. What does that mean? Low power, high throughput. It doesn't use brute force—it uses cleverness.

A Real Case That Gets You Excited: Qwen2.5 on FPGA

This isn't theory. This is real practice!

Someone put together a project called "Onboard Qwen2.5." On what platform? A tiny, sub-20-watt, palm-sized Xilinx Kria KV260 edge platform (that Zynq chip with FPGA logic plus ARM processor).

The result? Running the Qwen-0.5B model, it generated 5.1 Tokens per second.

5.1? Compared to a GPU, that's slow—yes, it is. But look at it from another angle: a 20-watt box, offline, on the edge, mobile—can you imagine what you could do with that? An inference server powered by a power bank? That energy efficiency ratio, in specific scenarios, becomes a moat that competitors simply can't cross.

Even Wilder: TeLLMe—The Ultimate Ternary Logic Breakthrough

If Qwen2.5 was impressive, TeLLMe is a revolution.

Introduced in 2024, this thing is humanity's first ternary LLM accelerator running on a low-power FPGA. It did something insanely aggressive: weights use only 1.58 bits, activations use only 8 bits! What does that mean? It's fully compatible with both the prefill and decode phases.

When the test data came out, I was stunned. On a total power budget of 7 watts, for an input length of 1024, it achieved real-time interactive latency!

Think about it: 7 watts! That used to be enough for a light bulb. Now it can smoothly run inference on a ten-billion-parameter large model. This isn't optimization—it's practically cheating.

Extreme quantization + programmable logic is turning the impossible into reality.

• -

System Synergy: Hardware Without Software Is Just Scrap Metal

At this point, I want you to take a breath.

No matter how impressive the hardware is, if the software is a deadweight teammate, none of it matters.

Look at vLLM's PageAttention. It's like a memory magician—by dynamically mapping logical and physical KV blocks, it cleans up memory fragmentation and sends system throughput through the roof.

And frameworks like TensorRT-LLM, MindIE-LLM—they orchestrate various attention patterns (MHA, MQA, GQA) and implement pipeline parallelism. And that CPU overhead hiding that SGLang emphasized in October 2024? Just overlapping kernel launch with data transfer—one small trick, and decode stage efficiency jumps 20% straight up!

Then there's something even smarter: Speculative Decoding.

Let a small model act as a "drafter," quickly writing a few candidate Tokens. Then the large model only has to play "approver," verifying them in parallel. Quality stays high, GPU batching capacity gets utilized, and latency gets crushed.

See? Every