Back to Blog

The Best Code I Wrote Was the Code I Deleted

#rust #testing #ai
flux pro 2.0 An architect s drafting table viewed from above covered in crumpled blueprint pa 3

I spent five hours one night trying to revolutionize Rust integration testing. I was going to write an open source crate. I was going to solve the testcontainers performance problem once and for all. I ended up enabling a feature and writing some SQL.

Sometimes the best engineering is knowing when to stop engineering.

The Problem

My Rust backend has about 700 tests. Around 450 are unit tests that use mocks and run instantly. The other 220 are integration tests that hit real PostgreSQL databases via Docker containers. That's the whole point of integration tests: they catch the bugs that unit tests miss because they actually hit the database.

The problem: each integration test was spinning up its own Docker container. PostgreSQL takes about 3 seconds to start. My CI runs were taking 7 minutes and 30 seconds. Most of that was container startup overhead.

I use cargo-nextest which runs tests in parallel across multiple threads. Four threads means four containers running simultaneously. But here's the catch: nextest runs each test in its own process for isolation. When a test finishes and a new one starts on the same thread, it gets a fresh process. The old container reference is gone. So the new test spins up yet another container.

220 integration tests. 220 containers. 220 times waiting for PostgreSQL to say "database system is ready to accept connections."

The Grand Plan

My first instinct was to build something. Of course it was. I'm a developer.

I searched for an existing crate that solved testcontainer pooling. Nothing. So I started designing my own. The idea: a background daemon that manages a pool of pre-warmed PostgreSQL containers. Tests connect to it via Unix socket or TCP. When a test needs a database, it grabs one from the pool. When it's done, the container goes back to the pool after a quick TRUNCATE.

I even had a name picked out: testcontainer-pool. I was going to publish it to crates.io. I imagined the GitHub stars. I started writing a README.

Then I found deadpool.

The First Pivot

Deadpool is a popular async connection pool in the Rust ecosystem. Why build my own pooling crate when I could just wrap testcontainer instances in deadpool? Each nextest slot would grab a container from the pool, use it, return it.

So I pivoted. Started building an implementation using deadpool. Custom pool managers. Container lifecycle hooks. Environment variable coordination between test processes.

It ran fine with cargo nextest. No errors. But it saved nothing. Each test process created its own pool, grabbed a container, used it, and exited. The next test process started fresh, created a new pool, and spun up another container. The pool was working exactly as designed. It just couldn't share state across process boundaries.

We tried cargo test to see if same-process execution would help. It failed immediately. We didn't debug it. Even if we got it working, we use nextest for its output and parallelism. Maintaining two test configurations wasn't happening.

The Second Pivot

Fine. I'd build an out-of-process solution. A daemon. A sidecar. Something that runs before the tests and stays running throughout.

I started sketching the architecture:

  • A long-running process that owns the container pool
  • IPC mechanism for tests to request/release containers
  • State management for which containers are in use
  • Cleanup logic for when tests crash
  • Health checks for zombie containers

This was going to be a proper open source project. Maybe 2000 lines of code. A week of work, minimum. Documentation. Examples. CI/CD for the crate itself.

I got about 200 lines into the design when my AI assistant offhandedly mentioned: "You know testcontainers has a reuse feature, right?"

The Facepalm

I'd been using Claude to help with this whole endeavor. We'd researched pooling crates together. We'd designed the deadpool implementation together. We'd sketched the daemon architecture together.

And Claude knew about the reusable-containers feature the entire time.

Why didn't it come up during the initial research? Because I asked the wrong question. I asked about pooling crates for testcontainers. Claude dutifully searched for pooling crates. I was tunnel-visioned on the solution I'd imagined, and the AI followed my lead instead of stepping back to ask what problem I was actually trying to solve.

The feature I needed was in the testcontainers crate itself. A Cargo feature you enable:

testcontainers = { version = "0.25", features = ["reusable-containers"] }

Then you configure your container to reuse:

let image = GenericImage::new("postgres", "17")
    .with_container_name("test-postgres-0")
    .with_reuse(ReuseDirective::Always);

That's it. When a test starts, testcontainers checks if a container with that name already exists. If it does, it connects to the existing one instead of creating a new one. The container persists across process boundaries because Docker manages it, not your test process.

A Cargo feature flag. Two method calls. No daemon. No crate. No GitHub stars.

The Smart Part

Container reuse alone doesn't solve everything. If test A inserts a user and test B expects an empty database, you've got test pollution. The standard solution is to drop and recreate the database for each test, but that's slow.

The fast solution: TRUNCATE CASCADE. It's nearly instantaneous compared to recreating tables. After each test connects to a reused container, it runs:

TRUNCATE TABLE users, roles, posts, ... CASCADE;
INSERT INTO roles (name) VALUES ('user'), ('admin'), ...;
INSERT INTO users (email) VALUES ('system@example.com');

Wipe the data. Re-seed the essentials. The schema stays intact because it came from migrations that only run once per container.

The other smart part: naming containers by nextest slot. Nextest exposes NEXTEST_TEST_GLOBAL_SLOT as an environment variable. Slot 0 gets test-postgres-0. Slot 1 gets test-postgres-1. Eight slots, eight containers, reused across all 220 integration tests.

let slot = std::env::var("NEXTEST_TEST_GLOBAL_SLOT")
    .unwrap_or_else(|_| "0".to_string());
let container_name = format!("test-postgres-{}", slot);

The Results

Before: 7 minute 30 second CI runs. 220 containers created and destroyed.

After: 5 minute 30 second CI runs. 8 containers created and reused.

Two minutes saved. Not dramatic. But it compounds. Every PR, every push to main, every time I run the CI check locally. Over a year, that's hours of developer time. And my Docker daemon no longer sounds like it's mining cryptocurrency when I run tests.

The first test in each slot still takes 3 seconds to boot its container. The remaining tests take about 0.1 seconds each to connect to an existing container and truncate. The math works out much better now.

The Lesson

I deleted 200 lines of pool daemon code. I abandoned my half-planned crate. The actual fix took maybe 20 minutes once Claude mentioned the right approach.

The whole arc took five hours, from 7pm to midnight. Most of that was wasted on solutions that couldn't work because I was fixated on the wrong abstraction.

There are two lessons here.

The first is the obvious one: the clever solution usually exists. Someone smarter than me already hit this problem. The feature I needed was in the crate I was already using. I just didn't look for it because I was too excited about building something.

The second is about working with AI assistants. Claude had the answer the entire time. But I asked about pooling crates, so Claude researched pooling crates. I never stepped back to describe the actual problem: "I want to reuse containers across test processes." If I had, we'd have found the reusable-containers feature in the first ten minutes.

AI assistants are powerful, but they follow your framing. If you frame the problem wrong, they'll efficiently help you build the wrong solution. The skill isn't just knowing how to prompt. It's knowing when to step back and reframe.

Sometimes the best code you write is the code you delete before it ever ships. And sometimes the best question to ask your AI is: "Wait, am I even solving the right problem?"

My CI runs are two minutes faster. My codebase has three changed files instead of a new crate dependency. And I have a blog post instead of an abandoned GitHub repository.

I'll take that trade.

Share: