The first time the new check ran, it found an issue in about 30 seconds. A model release, one story, sitting in five different piles — a funding headline, a home-lab benchmark post, a leaderboard result, a couple of forum threads, all talking about the same thing under slightly different names. I have not asked the check to look for that specific failure. It had never run before. It existed because of a podcast I listened to an hour earlier.
There was no design doc anywhere describing what the feature should do before it did. I heard a PM describe how her team ships changes — not by writing a page of prose about the desired behavior, but by collecting 30 or 40 real examples of a failure and turning them into a standing check — and in an hour, my little news aggregator had one of its own.
The idea is to watch dozens of sources, notice when several of them are covering the same story, and fold them into one. It goes wrong in a simple way - two outlets use different names for the same thing: one calls it by its nickname, the other uses the formal one. The story splits, and a reader sees the same headline twice, which starts to erode their trust in my page.
I’d caught this three times with the agent. It notices the split, traces it back, patches the matching logic, and moves on. Nobody was watching to see if the exact same split came back next time under a different headline.
That’s the part the podcast’s argument was actually about. A requirements document describes what you intended, not whether the thing still does it three weeks later. A folder of thirty examples, each one graded automatically against the current output, doesn’t care what you intended — it just keeps asking the same question, forever, on every run.
So the check I wrote wasn’t clever. It took the two splits I already knew about, wrote down what “fixed” looked like for each, and asked, every six hours, whether anyone had quietly come apart again. It only reads the live database — no write access back into the pipeline, on purpose. A thing that watches for mistakes shouldn’t also be the thing that gets to correct them unsupervised; that’s a second failure mode. On its first run, one of the two had come apart again.
Not exactly failed. When I actually read what was in that pile of five, most of it wasn’t the same story with different names — it was five different stories that happened to mention the same product. A funding round is not the release it funded, and a person benchmarking a model on their own hardware is not the press covering its launch. The check was right that it needed a second look, just not where it thought it did.
That turned into a second, narrower version of the same idea: don’t gate on “this name showed up in more than one place,” gate on “the specific failure I already fixed once is happening again.” Nothing merges by itself. One layer says, “Stop and check.” The second just says, “Here’s what’s around,” in case it’s useful — and I have to decide.
None of this required me to write down, in advance, what correct clustering looks like across every case that might come up — which is good, because I have no idea. What I could do was keep saying, every time reality disagreed with the code: here is exactly the shape of the mistake, don’t let it back in unnoticed. So the spec for this service isn’t a document. It’s whatever set of “this happened once, it must not happen silently again” checks has accumulated so far, and it only gets more honest the more times something goes wrong.
The document would have described what I thought looked correct. The check tells me, every six hours, whether I was right.