仅用3%参数微调大模型，效果几乎不变 (English)

Generated: 2026-06-22 01:09:51

---

That Summer of 2021, I Almost Thought LoRA Was an Anime Character

Speaking of which, I still laugh about it now.

The first time I saw the acronym LoRA, my brain immediately conjured up some cute girl in a sailor uniform from a Japanese anime. But when I looked it up—holy cow, this thing later blew the lid off AI art and model fine-tuning!

In 2021, a researcher at Microsoft named Edward J. Hu led a team to publish a paper: LoRA: Low-Rank Adaptation of Large Language Models.

Guess what happened then?

GPT-3 had just blown everyone away—175 billion parameters! Want to fine-tune it? You'd better ask your graphics card if it's up for the task. I tried it myself. Even with multiple A100s running in parallel, full fine-tuning of GPT-3's memory easily exceeded 350GB.

350GB! Who the hell can afford that?

That summer, LoRA was like an unsung tinkerer, standing in front of a behemoth and saying just one thing: "You don't need to move the whole model—just slap a few 'little patches' on it."

---

Can You Believe It? One Patch Did the Trick?

Don't rush me to throw formulas at you. Let me tell you a story.

You have a suit—expensive, perfectly tailored. Now you want to add a pocket on the chest for your phone.

What does full fine-tuning do? It rips the whole suit apart, recuts it, and resews everything. Time-consuming, labor-intensive, and easy to ruin the fit.

What does LoRA do? It simply sews a small pocket onto the suit. Nothing else moves.

That's it?

That's it!

Mathematically, LoRA's core assumption is a mind-blowingly counterintuitive idea: During model fine-tuning, the part of the weight changes that actually carries useful information is inherently "thin and small."

What does that mean? Let me do the math for you—

Original matrix: 4096×4096, totaling 16,777,216 parameters. Dense as a skyscraper of information.

But the effective information that truly needs updating? It might only need a few dozen dimensions to express.

LoRA uses a trick: it splits a big matrix into two smaller ones—one for dimensionality reduction, one for dimensionality increase. If we set r=64, how many parameters do we get? 64×(4096+4096) = 524,288.

Compression ratio: 3.1%.

Think about it: using only 3% of the original parameters, you can achieve roughly the same effect. A few years ago, who would have believed that?

I calculated it three times in a row just to make sure I wasn't dreaming.

---

How Does LoRA Work? Let Me Explain in Plain English

By now, you're probably thinking this must be super complicated.

Actually, it's dead simple—four steps, and you'll be able to explain it to someone else after reading.

Step 1: Freeze the original model

Don't touch a single pre-trained parameter. It's like that suit—you leave it alone, it stays perfect.

Step 2: Insert two small matrices

Right next to the model's key layers—usually the attention layers—quietly insert matrix A and matrix B. A compresses the information, B restores it.

Step 3: Train only these two small matrices

During training, the entire big model takes a nap. Only these two small matrices do the work. After training, they still add up to very few parameters.

Step 4: Merge and done!

During inference, multiply A and B together and add the result directly to the original weights. It looks like nothing ever happened.

Here's the kicker—inference speed is exactly the same as the original model, with zero extra latency!

I tested it with Stable Diffusion: after loading a LoRA module, generating a 512×512 image took almost the same time as without LoRA. That's way better than Adapter, which increases network depth and slows down inference.

Think about it: better results, drastically fewer parameters, same speed—isn't this the AI world's "have your cake and eat it too"?

---

Step-by-Step: Run a LoRA Training (All Practical Tips)

I've stepped on enough landmines to feed you the hard-earned lessons directly.

Environment Setup

I use kohya_ss, version v21.8.0. Strongly recommend Python 3.10 or higher, PyTorch 2.0+.

git clone https://github.com/bmaltais/kohya_ss.git
cd kohya_ss
pip install -r requirements.txt

Done.

Data Preparation—This Is the Pitfall I Must Emphasize

I trained a Pokémon-style LoRA with 100 Pokémon images. You think that's enough?

Wrong! I started with 50 Pikachu images, and the model went crazy—it could only draw Pikachu. Everything it drew had a Pikachu face. The horror, I can't even look at it.

Lessons learned the hard way:

• Image sizes must be uniform: 512×512
• Each image needs a text description, e.g., "a pikachu, pokemon style, anime"
• Variety is crucial! 50 Pikachu? That's basically training just one character.

Training Parameters—Just Copy Mine

learning_rate: 1e-4
train_batch_size: 4
max_train_steps: 1000
lora_rank: 64
lora_alpha: 128
network_module: networks.lora

Here's a counterintuitive point—bigger rank isn't always better.

I ran a comparison: rank=128 actually gave worse results than rank=64. The original paper also says that for most tasks, a rank between 4 and 64 is sufficient. Don't just crank up the numbers—AI isn't just about having better hardware.

Training Time

On a single RTX 3090, 1000 steps takes about 20 minutes.

Compare that to full fine-tuning: with the same data, it would take at least 2 to 3 hours.

20 minutes vs. 3 hours—with the former, you can scroll through TikTok; with the latter, you're stuck staring at a progress bar.

---

Wait, There Are So Many Variants of LoRA?

When something gets popular, all sorts of spin-offs pop up.

First up: LoCon—short for LoRA-Convolution

This one specifically targets convolutional layers. I tested it on Stable Diffusion, and LoCon preserves image details much better than vanilla LoRA. How much better? The difference between "meh" and "holy crap."

Second: LoHa—Low-Rank Hadamard Product

It uses Hadamard product instead of matrix multiplication, requiring even fewer parameters. I ran some tests—LoHa is indeed strong on style transfer tasks, but when drawing faces, the details look like you've overused a beauty filter—smooth to the point of losing texture.

Third: LyCORIS

This is a big mashup that integrates LoRA, LoCon, and LoHa all together. I recommend you just use this one directly—saves you the hassle of juggling multiple variants.

Fourth: LCM-LoRA—I Must Highlight This One

The paper is titled LCM-LoRA: A Universal Stable-Diffusion Acceleration Module.

It combines LoRA with Latent Consistency Models to do something insane: reduce Stable Diffusion's inference steps from 50 down to 4!

You read that right—4 steps!

I tested it: generating an image with LCM-LoRA takes only 0.5 seconds, 10 times faster than before.

10 times faster, with similar quality—where you used to wait for an image like cooking a steak, now it's as quick as making instant noodles.

---

What Can LoRA Do?

When it comes to applications, I have to say—this thing has liberated so many people!

First scenario: Keep characters consistent

Want AI to draw the same character in different scenes? Before, AI would change faces mid-generation, like a blind date who looks different every time you meet.

Train a LoRA on a few images of that character, then load that LoRA during generation. The character's features stay consistent—eyes are eyes, nose is nose, no sudden transformations.

Second scenario: Style transfer

Want all generated images to have a Hayao Miyazaki vibe? Train a Miyazaki-style LoRA and load it every time you generate.

Third scenario: E-commerce images

When I generate product images for e-commerce, I use LoRA to train the product's features. The generated images keep product details accurate. This is crucial for e-commerce—you're selling this bag, so the image better show that bag, or your return rate will skyrocket