实测改三个开关，70B模型训练快30% (English)

Generated: 2026-06-22 15:15:19

---

The Communication Optimization Secrets in Megatron-Core Nobody Tells You

Last year, I trained a 70B model, spending hundreds of thousands of GPU hours, and I was thrilled, thinking I was finally going to get results. But then the neighboring team, with the same GPUs and the same model, was 30% faster than me! I was floored: How? Did they have some secret black tech?

Later, I put my pride aside and went to ask them. They just tossed me a config file and said, "We only changed three switches."

Three switches! I nearly spit my coffee all over the screen. That day, I truly understood—the core of large model training isn't compute power; it's communication! You spend millions on a stack of GPUs, and half the time you're waiting for data to move. Tell me that's not a rip-off.

Today, I'm going to lay bare the Megatron-Core communication optimization pitfalls that nobody tells you. Don't bother searching through those theoretical articles—ZenDi, Ring-AllReduce—you read them and still don't know what to tweak. No, let's get straight to the switches in the code. These are the hard-won lessons I paid for with real GPU runs.

---

Pitfall #1: Tensor Parallelism Is a Communication Socket, But You Blame It Anyway

When it comes to Tensor Parallelism (TP), don't a lot of people think: once you turn on TP, communication eats up a big chunk, and that's just the way it is? Wrong! Completely wrong!

The first time I ran LLaMA-13B with TP=8, I looked at the profile and saw communication taking almost 40% of the time. I was so frustrated—thinking, isn't TP a fake kind of parallelism? Then I dug into the Megatron-Core docs and found something counterintuitive: the communication in tensor parallelism can be "hidden."

How? Simple: split one large matrix multiplication into four smaller blocks. Immediately after each block finishes computing, start communicating. That way, communication and computation overlap. You won't believe how dramatic the effect is: peak bandwidth utilization jumps from 55% to 80%! Single-step training time drops from 2.3 seconds to 1.7 seconds! You think that's voodoo? I tested it repeatedly—it's absolutely real.

Speaking of which, I have to mention a switch called userbuffer. It's in Transformer Engine, not in the Megatron main library, so most people have no idea it exists. It creates a buffer design within the NVLink bandwidth to reduce synchronization interference between CPU and GPU. After I turned it on, the effective bandwidth shot up by more than ten percentage points. Tell me, isn't that saving you money?

And for the all-gather plus gradient computation in the backward pass, Megatron already handles the overlap for you—it's called bulk overlap. Just turn on Transformer Engine and it works automatically. But! If you write custom operators that bypass TE, you'll have to implement the communication hiding yourself—and whatever you do, don't hack it together wrong. Don't ask me how I know.

Someone might say: "Isn't the default configuration fast enough?"

But have you actually profiled it? For GPT-3 175B, without TP partition, communication takes over 30%. Cut it into four partitions and it drops straight to 12%! This isn't a minor tweak; it's a game-changer. Think about it—saving 18% time means that many more steps you can run.

---

Pitfall #2: 90% of People Write Pipeline Parallel P2P Wrong

Do you think pipeline parallelism's send/recv is just a torch.distributed.send? Naive, my friend.

The first time I tuned interleaved 1F1B, I wrote the P2P communication by hand and ended up in constant deadlock. Two GPUs were both waiting for each other to send, neither giving in, and everything froze. I was about to smash my computer. Then I gritted my teeth and dug into the Megatron source code, only to find that they'd already wrapped up four neat interfaces: sendforwardrecvbackward, sendforwardbackwardrecv_forward… the names are long, but they spell out the dependency order crystal clear so you won't deadlock. I swapped them in, and the problem was gone instantly.

Moreover, in the interleaved 1F1B scheme, each device has to handle multiple micro-batches simultaneously, so the number of communication calls is much higher than in regular 1F1B. How do you think Megatron-Core optimizes that? It implements a specialized communication hiding: while one micro-batch is computing, the P2P communication for the previous micro-batch has already quietly finished in the background. Grab the timeline with NV Nsight, and you'll see computation and communication perfectly overlap, with almost zero bubbles! Can you imagine how satisfying that feels?

When I trained a 70B model with TP=8, PP=4, and 16 micro-batches, enabling interleaved 1F1B boosted throughput by 22% compared to regular 1F1B! The cost? Just a bit more memory to store the activations for a few extra micro-batches. But your time is worth it!

Someone might say: "Interleaved 1F1B code is too complex; I'm not touching it."

Come on, Megatron has already packaged it for you! You only need to set nummicrobatches and overlapp2p_comm=True in the config, and Megatron handles all the P2P interfaces. If you don't use it, you'll waste weeks tuning parameters. Can you live with that?

---

Pitfall #3: The Fragmentation in Data Parallelism—You Never Thought to Optimize It

When it comes to data parallelism, most people's first reaction is PyTorch DDP. But for large models, especially those with hundreds of billions of parameters, DDP's all-reduce waits until the entire backward pass finishes before triggering. Communication and computation have zero overlap! Think about it—the backward pass is done, and only then does the communication lazily start. All that time the GPU is idle? What a waste!

Megatron's DDP is much smarter. It uses a four-layer architecture, with the core being contiguous parameter/gradient buffers. It rearranges parameters by dtype into contiguous memory, then splits them into buckets by Transformer layer order. The bucket size is calculated dynamically, usually set to max(40M, dp_size * 1M). During the backward pass, as soon as the gradients for one bucket are ready, it immediately launches the reduce-scatter without waiting for the later layers to finish.

I ran a comparison: training GPT-2 XL (1.5B), PyTorch DDP achieved only 65% GPU utilization, while Megatron DDP hit 88%! Why? Simply because communication and computation overlap, and there's less memory fragmentation—you don't need separate gradient buffers for each parameter.

The coolest part is that this gradient buffer is zero-copy—it points directly to contiguous memory, eliminating the copy_ operation. On models with hundreds of billions of parameters, that redundant copy can save tens of gigabytes per second! Imagine how much time that saves.

---

Pitfall #4: Don't Set Your Expectations Too High for Sequence Parallelism and MoE Communication

Recently, a lot of people have been hyping Ulysses Sequence Parallelism, but I need to throw some cold water on that. Megatron-Core currently does not implement Ulysses SP. The tutorials you see online are mostly based on DeepSpeed-Ulysses, and when you combine it with Megatron's TP and PP, you'll run into a pile of issues. Once you fall in, you won't even have time to cry.

In contrast, Megatron's own Sequence Parallelism (TP