I Let GPT-5.1-Codex-Max Refactor My Payment Module at 2 AM — Here's What Happened

Last Tuesday, at 2 AM, I did something reckless. I pasted our three-year-old payment module into GPT-5.1-Codex-Max and said, "Refactor this."

47 unit tests. Zero failures. First run.

It also fixed a concurrency bug I'd known about for two years but never dared touch.

Honestly? My spine went cold.

I've spent every spare hour these past two weeks stress-testing this thing. CRUD operations. Algorithm challenges. Frontend components. Database optimisations. Every scenario I could think of, I threw at it. Here's my unfiltered experience — the good, the bad, and the genuinely unsettling.

What Makes It Different From the Last Generation

GPT-5.1-Codex-Max isn't just "better at writing code." It's something else entirely — it's starting to understand engineering context.

Last Wednesday afternoon, I fed it an 800-line Python file. Business logic, database operations, and caching strategies all tangled together. Chinese comments everywhere. Variable names that were... let's say "creative." Some were literally in pinyin. I said, "Extract the caching logic into a separate module. Keep the existing interface unchanged."

Here's what it did:

It accurately identified every Redis-related code block — including those pinyin variables like huanchunshuju (缓存数据, cache data). Generated a cachemanager.py with encapsulated read/write operations, expiry policies, and batch invalidation. Handled circular reference issues automatically. Replaced all cache calls in the original file with the new interface. Fixed all the imports.

With GPT-4, this would've taken at least five rounds of back-and-forth, plus manual cleanup of edge cases. This time? One round. The changes were code-review ready.

Let me give you some actual numbers. I tested 20 real requirements from our company's backlog — API development, data processing, refactoring, unit test generation. GPT-5.1-Codex-Max had a 67% first-pass rate. GPT-4, tested during the same period? 41%. By "first-pass," I mean the code ran correctly, met the requirements, and needed zero human intervention.

For the record: I ran these tests between 17-23 January 2025, using PyCharm 2024.3.1 with GitHub Copilot Chat's Codex-Max channel. I'm noting the version numbers because someone will inevitably accuse me of making this up.

Three Moments That Stopped Me in My Tracks

Case 1: Unit Tests for Legacy Spaghetti Code

I deliberately picked a nightmare — a 300-line order processing function with cyclomatic complexity through the roof. No comments. No documentation. The classic "don't touch it or everything breaks" kind of code.

I pasted it in and said, "Write comprehensive unit tests covering all branches."

About 15 seconds later, it spat out 47 test cases.

I ran them. 43 passed immediately. The remaining 4 failed — but here's the thing: the original code had bugs. The tests exposed hidden problems that had been lurking for years.

And then I saw this comment at the top of the test file:

"The following 4 test cases are expected to fail, likely because the original function lacks null-safety for `order_amount`. Consider adding a Guard Clause."

It even provided the fix.

This isn't code completion. This is code review.

Case 2: Rust to Go Migration

I had a CLI tool written in Rust — tokio async runtime, serde serialisation, clap argument parsing. I wanted to see if it could migrate to Go.

Actually, let me correct myself — I didn't "want to see." I had zero expectations. Cross-language migration involves fundamental differences in async models and error handling patterns. It's easy to get wrong.

The Go code it generated? Functionally identical. And it automatically handled several things: mapped Rust's Result to Go's (value, error) return pattern; replaced tokio's runtime with goroutines and channels; preserved the original CLI argument style; elegantly rewrote Rust's match pattern matching using Go's switch.

I showed it to Old Zhang, our team's Go expert. He studied it for ten minutes, then said, "This is better than what the guy we hired with two years of experience writes."

Ouch. But fair.

Case 3: SQL Optimisation With Unsolicited Advice

Friday afternoon, I casually pasted in a slow query and asked for optimisation suggestions. It rewrote the SQL, analysed the execution plan, and pointed out missing indexes.

Then it added something I didn't ask for:

"This query pattern looks like pagination. If your data exceeds millions of rows, consider cursor-based pagination instead of offset. Want me to write an example?"

This is... complicated. It started proactively giving suggestions. Not waiting for me to ask — it judged the scenario and proposed solutions on its own. It felt eerily like pair programming with a senior developer who anticipates edge cases before you do.

Where It All Went Wrong

It wasn't all smooth sailing.

Pitfall 1: The Distributed Lock Hallucination

I asked it to implement a distributed lock using Redis's Redlock algorithm. The code looked flawless. Comments were thorough and well-reasoned.

But when I examined it closely, I found a race condition in the lock renewal logic — if the client crashes after sending the renewal request but before receiving the response, the lock gets incorrectly released. These edge cases in concurrent scenarios still trip it up.

This is probably the kind of trap Martin Kleppmann was talking about when he criticised Redlock years ago.

The lesson: Concurrent code must be reviewed by a human. No shortcuts.

Pitfall 2: Configuration Files With Hidden Landmines

I asked it to generate a Docker Compose configuration. It added all sorts of "optimisation parameters" for Redis — vm.overcommit_memory=1, tcp-backlog=511. Looked professional.

But buried in there was save "" — which completely disables RDB persistence. It didn't warn me about this side effect. I nearly deployed it to production.

Thank god I double-checked. Otherwise, a restart would've wiped our data.

The lesson: Understand every line of infrastructure configuration. Don't blindly copy-paste.

Pitfall 3: Long-Context Attention Decay

I fed it a 2,000-line project. The first few rounds were great — it handled deeply nested utility functions without breaking a sweat.

Around round eight, it started "forgetting" interface definitions from other modules. It generated code referencing a non-existent function — getusersessionv2(). The project only had getuser_session(). No v2 anywhere.

I suspect this is related to attention mechanisms in the context window. I haven't precisely measured at what token count the decay kicks in, but my gut feeling is: keep complex tasks within five rounds. Beyond that, start a fresh session and re-feed the context.

The error, by the way, looked like this:

AttributeError: module 'utils.session' has no attribute 'get_user_session_v2'

Instantly recognisable as a hallucinated function name. These basic mistakes become surprisingly common in the latter half of long conversations.

How I Use It Now

After two weeks, I've settled into a collaboration rhythm:

1. Treat It Like a Pair Programming Partner

Don't just say "write a login endpoint." Spell out the business constraints:

"This is a SaaS platform supporting email and phone login. Use bcrypt with cost factor 12. Lock accounts for 30 minutes after 5 failed attempts. Return a JWT valid for 2 hours, refresh token for 7 days. Sessions must support multiple devices."

The more specific you are, the better the output. It's exactly like mentoring a junior developer.

2. Think Before You Write

I now start by asking it to analyse requirements, design interfaces, and list edge cases — before writing a single line of code. One extra step of dialogue, and the code quality jumps significantly.

My prompt usually goes: "Don't write code yet. Help me analyse the edge cases for this requirement, design the function signatures, and plan the error handling strategy."

3. Never Skip the Review Process

My workflow: AI generates → I review → run tests → AI reviews again.

That last step is fascinating. Paste the code back and ask, "What potential issues does this code have?" It often catches things it missed the first time around.

If humans need their work reviewed, AI does too.

Let's Be Honest

GPT-5.1-Codex-Max is genuinely impressive.

But it's not here to replace programmers. What it's done is dramatically reduce the "translation" cost — that tedious process of converting business requirements into code. That layer is being compressed.

So what's left?

Understanding the business. Designing architecture. Making trade-offs. Managing risk.

These are the core competencies now. Code is just a means of expression.

I've seen too many colleagues panicking: "Will AI replace me?" Honestly? If your daily work consists of writing CRUD operations, tweaking parameters, and assembling components — yeah, you should probably be worried. But if you're thinking about "why are we building this feature," "what's the smarter approach," "how do we handle failure scenarios" — then AI is your amplifier.

I saw a Hacker News post the other day claiming Anthropic is already using Claude 4 internally for code review, with a 12% higher pass rate than humans. No idea if that's true, but the trend is undeniable. Since that explosion of AI coding tools in late 2024, this space has been moving terrifyingly fast.

Anyway, I've rambled enough.

I'm genuinely curious about your experience — what's the worst pitfall you've hit with AI coding tools? Or has there been a moment that genuinely surprised you? Drop a comment below. I read every single one.

TL;DR

• First-pass rate: 67% for GPT-5.1-Codex-Max vs 41% for GPT-4 across 20 real-world tasks
• Strengths: Understands engineering context, proactive suggestions, handles complex refactoring in one shot
• Weaknesses: Still hallucinates in concurrent code, silently adds dangerous configs, attention decays in long conversations
• My workflow: Treat it like a senior dev, specify constraints in detail, never skip human review
• The real question isn't "will AI replace us?" — it's "are you doing work that AI can't amplify?"

ai #programming #developertools #codereview #gpt5