The Unsexy Infrastructure That Actually Powers AI Coding Agents (And Why Your Demo is Lying to You)

Let’s be honest about something first. I’ve seen maybe twelve of these “Codex agents” pitched in the last eight months—it’s March 2025 now, so the hype cycle is absolutely peaking—and they all make the same flashy demo. But you know how it goes. Demos are lies dressed up in expensive GPUs. I should know. I built a few of those demos myself to keep the VCs distracted while the real engineering happened somewhere else.

So here’s what I’ve actually seen. What the flashy YouTube videos and the carefully manicured blog posts don’t talk about. Not the prompt tricks or “clever” chain-of-thought stuff. The messy, sprawling, industrial-grade plumbing that makes a project-level agent work when nobody’s watching.

Actually, wait—I should clarify something upfront. When I say “project-level agents,” I’m not talking about Copilot autocomplete or some ChatGPT wrapper that shell-outs to npm run build. I mean an agent that opens a repo on Monday morning, understands it, plans actual multi-file work, executes it across sessions, and you come back Tuesday to something that’s not on fire. Probably.

We’re not entirely there yet—maybe 80% reliability on a good day, and “good days” are capricious creatures—but the architecture that might get us there is already running in some very quiet, very well-funded startups. And I’ve seen just enough of it to feel that familiar pit in my stomach.

The “Context Window” Obsession is a Dead End

Every time someone mentions “200K context window,” a small part of me dies. Not dramatically. Just a quiet, internal sigh.

I mean, okay. Sure. Bigger windows are nicer than smaller windows. But the conversation stops there, and it drives me absolutely nuts. Throwing a massive, barely-pruned codebase dump into the prompt context is the kind of thing you do at a hackathon at 3 a.m. when your judgment has dissolved into a puddle of Red Bull and false confidence. It is not architecture.

The actual sophisticated setups—the ones you never see diagrammed on Twitter—are doing something else entirely. The prompt is the tip of the iceberg. Below the waterline, there’s this whole persistence machinery. The thing that makes it not a fancy autocomplete.

I remember one incident, somewhere around November 2024, at a company I can’t exactly name. I was staring at a log output that I still have burned into my memory—lines and lines of an agent just... spinning. It was stuck inside a refactor_module task, looping between two broken states for 3 hours and 17 minutes. I know because I checked the timestamps obsessively while stress-eating almonds. A human would have stopped, scratched their head, and asked for help. The agent? It was like a wasp against a windowpane. That’s the moment I really understood it’s not about how much it can see. It’s about what it remembers. No amount of context window is going to pull it out of that loop.

So let’s talk about what real memory and real task management looks like. It’s... a lot.

What Actually Runs the Thing (Or At Least Tries To)

Trying to explain this to non-builders is a special kind of torture. Imagine drawing a diagram on a whiteboard for your CTO that includes Kafka, a vector DB, and three different kinds of state snapshots, all for what marketing insists on calling a “simple AI developer tool.”

The look you get is... let’s just say it’s not enthusiastic. It’s the look of someone mentally re-evaluating your headcount.

But here’s the blueprint I’m talking about. Three dirty, complicated layers.

Layer 1: The Task Brain

Forget async/await. I’m talking persistent task queues with a level of paranoia that borders on pathological. We run it on Redis Streams now, though honestly I’m thinking of moving us to NATS JetStream after a particularly ugly split-brain incident three weeks ago that I’m still mentally recovering from.

The core is a task DAG (Directed Acyclic Graph) living in Postgres. Each node in that graph isn’t just a “todo.” It’s a little bundle of state:

• status: pending / inprogress / completed / failed / hallucinatedinto_oblivion
• dependencies: strict, ordered
• rollback_instructions: an absolute necessity, not a nice-to-have
• sourcepromptsnapshot: the exact, raw prompt string that spawned the task

Why save the raw prompt? Because a few months back, I spent a full Tuesday morning trying to figure out why the agent decided the best way to “optimise imports” was to delete the entire /utils directory. The logs just showed a successful task completion. I had to trace back the exact prompt it was given, which had been mangled by a bad template variable substitution earlier in the pipeline. We store them religiously now. You learn these things the hard way.

Layer 2: The Memory Mess

This is where things get genuinely weird.

The agent needs to remember not just what it did, but why, and that “why” has to survive a server restart, a context window getting flushed, and the agent’s own tendency to contradict itself from one hour to the next. I had an agent once rename a variable from userCount to totalActiveUsers and back again in three consecutive commits. I’m not proud of how long it took me to notice.

We basically stole the idea of memory types from cognitive science:

• Episodic memory: The event log. What happened, timestamped, stored in an append-only log. Dry as a phone book.
• Semantic memory: What it learned about your code. Vector embeddings of code patterns, architectural decisions—like why the auth module is a tangled mess and must be treated with caution—and your team’s bizarre naming habits. We use pgvector for this now. It’s fine. Qdrant was faster but I didn’t want to manage another piece of infra. That was probably a mistake.
• Procedural memory: The how. How it solved similar problems. Retrieved via similarity search and fed back as dynamic few-shot examples in the system prompt.

I saw an agent retrieve a procedural memory of a bug fix from three sessions prior, apply it to a subtly similar new bug, and just... fix it. Preemptively. I felt a weird mix of parental pride and existential nausea. Like watching your kid parallel park perfectly for the first time and then realising you yourself are a notoriously terrible parallel parker.

Layer 3: The Blast Shield

If you’re just wrapping this in a Docker container, you’re basically hoping the firecracker won’t go off in your hand. Real execution needs a sandbox with some serious safety features:

• ZFS snapshots before every single mutation. Not just before a task starts. Before every file write operation that the task DAG triggers. I learnt this lesson one very long night in late 2024 when a partially-applied refactor left a codebase in a state best described as “Schrödinger’s syntax error.”
• Execution tracing: Every exec, every file write, every network socket opened. Jaeger traces that look like a plate of neon spaghetti, but they’re my plate of neon spaghetti and they’ve saved me multiple times.
• Rollback actually works, maybe: This is the dream. We’re still wrestling with partial rollbacks that cascade into other partial rollbacks. It’s a hard problem. If you’ve solved it perfectly, I will gladly buy you a coffee and steal your ideas.

Without this layer, you’re one infinite loop away from a blank repo and a genuine, heart-pounding panic attack at 11:47 p.m. on a Friday. I don’t recommend it.

The Crash

Let’s talk about it crashing. Because it will. Probably on a Friday.

The scenario: The agent is midway through a complex, 14-step task DAG refactoring your data access layer. Then, the process OOMs. The pod dies. Silence.

If your persistence is bad, the agent comes back up like a soap opera character with total amnesia. It has no idea what it did. It restarts the task from scratch, doubling up on changes you now get to manually untangle. You get to play the fun little game of “which of these files did it actually half-modify, and does git reset --hard even help me now?”

With solid event-sourcing and those state snapshots I keep rambling about, it reloads the DAG, consults the memory fabric, compares the filesystem snapshot against the current state, and picks up roughly where it left off. And you get a status message that says something useful instead of Error: Task failed successfully.

The difference between those two realities is about 200 hours of engineering time. Ask me how I know. Actually, don’t. I’m still a bit raw about it.

What Nobody Shows You

Those demos are the magic trick. The hand is faster than the eye. What they’re absolutely not showing you is the backstage machinery:

1. The prompt pipeline that pre-chews your vaguely-worded Jira ticket into structured, agent-safe subtask definitions. It’s an ugly regex-and-template monster I have come to loathe.
2. The eval harness that silently runs the agent’s proposed code diff through a gauntlet of linters, type-checkers, and custom security rules, rejects it three times before it even touches your repo, and you never even know.
3. The silent feedback loop. Every time you mutter “what the hell is this” and revert a commit, that’s a training signal. You think they’re not using that? They are absolutely using that.

Maybe don’t build this yourself. I mean it. Unless you have a small, slightly-crazed team of infrastructure engineers and a high tolerance for on-call pain, just... wait. The companies getting this right—Devon, Factory, a few others whose decks I’ve seen under NDA—are treating this persistence layer as the product, not an afterthought.

And they’re not blogging about it because they’re busy, you know, winning.

The uncomfortable truth is that the moat between a cool demo and a tool that replaces a junior dev isn’t the model. It’s this architecture. The persistence. The memory. The boring stuff.

The engineers who get that are building the thing that automates away the engineers who don’t.

So. Yeah. Your move, I guess.

TL;DR (For the Skimmers)

• Context windows are a red herring. The real magic is in persistence, memory, and task management—the boring infrastructure nobody demos.
• Three layers matter: A paranoid task DAG with rollback instructions, a multi-type memory system (episodic/semantic/procedural), and a sandbox with filesystem snapshots before every mutation.
• Crash recovery isn’t optional. If your agent can’t resume from a mid-task crash without amnesia, you’ve built a liability, not a tool.
• The silent machinery is the product. Prompt pipelines, eval harnesses, and implicit feedback loops are what separate $10M demos from $100M companies.
• Seriously, don’t DIY this unless you hate sleep. The infrastructure complexity is staggering, and the companies winning this space have 50+ engineer teams focused solely on persistence.

Related (semi-hysterical) reads I actually found useful:

• “Your AI Coding Assistant Has The Security Posture of a Wet Paper Bag” (Feb 2025)
• “Vector DB Benchmarks: We Spent $10k on Cloud Credits So You Don’t Have To” (Jan 2025)
• “That Time I Let an Agent Refactor Our Auth Module and Had to Explain It to SOC2 Auditors” (my own personal nightmare, Dec 2024)

What’s the most unhinged commit your AI tool has ever made? I could use a good laugh or a good cry. Leave a comment. Please tell me I’m not the only one living in this particular flavour of dystopia.

programming #ai #codex #agents #software-architecture #hot-takes #machine-learning #infra-wars #developer-tools #i-chose-pain