Five AI Assistants Walk Into a Test Suite: Our Messy, Humbling Experiment

Every vendor demo we've sat through this year has included some version of the same moment: a developer types 'write tests for this function,' the AI produces a wall of green checkmarks, and the room nods approvingly.

We wanted to know what happens after the demo ends. So we took a real-ish codebase — a TypeScript service with an Express API, some data-transform logic, and a handful of known edge cases — and ran it through five widely used AI coding assistants. We asked each one to do three jobs: generate tests for untested code, repair failing tests, and explain why a flaky test was flaky. Then we scored the output on accuracy, flakiness, maintenance burden, and our personal favorite metric: the confidence-to-substance ratio, i.e., how often the AI wrote a test that proves absolutely nothing while sounding extremely authoritative.

We're not naming names, partly because these models update monthly and any leaderboard is stale by the time you read it. What held steady across all five was the shape of the failures — and that's what's actually useful.

What AI Gets Genuinely Right

Credit where it's due, because the demos aren't entirely smoke and mirrors.

Boilerplate tests are basically solved. For pure functions with clear inputs and outputs — a date parser, a currency formatter, a validation helper — every assistant produced usable tests on the first or second try. If your suite is thin on unit tests for utility code, AI will close that gap in an afternoon. This alone is worth something.

Repairing broken tests is a killer use case. When we handed each assistant a failing test after an intentional refactor, most of them correctly diagnosed that the test was stale, not the code, and updated it with the right new expectations. This is the tedious work humans procrastinate on, and the AIs did it cheerfully.

Explaining flakiness is shockingly good. We planted a classic race condition — an unawaited promise, a shared fixture mutated between tests — and asked for diagnosis. Every assistant identified the root cause faster than our own first pass did. Reading a stack trace and reasoning about async ordering turns out to be a great fit for these models.

Where It Quietly Makes Things Worse

Now the humbling part. Every assistant, without exception, produced output that looked like testing and wasn't.

Tautological tests. Ask for a test of a function, and the AI will sometimes generate a test that asserts the function returns what the function returns — with the expected value literally copied from the implementation. It passes. It covers the line. It proves nothing. We caught this in output from all five assistants, and it's insidious because it increases your coverage percentage while decreasing your actual safety.

Tests written to the bug. Give an assistant buggy code and ask for tests, and several of them wrote tests that asserted the buggy behavior as correct. The test suite now enshrines the defect. If a human did this in code review, they'd get a talking-to; an AI doing it at 2am while nobody's watching is how a bug becomes a feature.

Mock theater. For anything touching a database or an API, the default move was to mock so aggressively that the test verified the mock, not the integration. You end up with a suite that's green in CI and useless in production — which, if you've read anything else on this site, is the exact failure mode we keep writing about.

Flaky by construction. Tests using real timestamps, real sleeps, and real network calls showed up everywhere. The AIs write tests the way a junior dev does: optimistically.

The Scorecard, Sort Of

Aggregated across all three jobs, the assistants clustered into a recognizable pattern rather than a ranking:

• Best overall default: the two assistants strongest at reasoning about async code — they were also the least likely to write tautologies.
• Best at volume, worst at judgment: the fastest generators produced the most tests and the highest percentage of tests-that-prove-nothing. Speed and skepticism did not correlate.
• Best at restraint: one assistant was notably more likely to say 'I can't determine the expected behavior from context — please clarify.' That hesitation was, counterintuitively, the most valuable trait we measured.

Where to Actually Point the Robot

After a couple weeks of this, our decision guide is simple:

Let AI write: unit tests for pure functions, regression tests for fixed bugs (with the fix described to it), updated assertions after refactors, and first-draft explanations of flaky failures.

Keep humans on: anything asserting correct behavior rather than current behavior — that requires knowing what the software is supposed to do, and no model has that context unless you give it. Also keep humans on integration test design, anything security-adjacent, and final review of every generated assertion.

The one habit that matters: treat every AI-written assertion as a claim, not a fact. Ask of each test the question the AI can't ask itself: how do we know this expected value is right? If the answer is 'the AI copied it from the code,' delete the test.

AI won't replace your test suite. It'll happily generate 500 lines of it, though — and whether those 500 lines are an asset or a liability depends entirely on whether anyone's still doing the thinking.