1128 notes repeated the thread title. 3 said something.
In 0.7.0 we deleted the most heavily reviewed component we had ever built. It was not broken. It did exactly what its specification said, under a watermark table, a sweep guard, a lock-order fix, a migration backfill tested against its own SQL, and fourteen rounds of adversarial review.
The rows it produced were garbage, and none of that verification was capable of noticing.
What it did
The automatic thread extractor took a closed thread, used the thread's subject as the knowledge title and its last agent message as the body, cut to 2000 characters, and stored that.
There was no distillation step. Not a weak one - there was no such step anywhere in the code. What accumulated was a copy of the conversation.
The measurement that ended it
Across the two hubs this was measured on: 1128 entries carried the thread's subject verbatim, and 3 had been written by an agent.
The three read as instructions. One of them, from our release notes:
| Knowledge title (written by an agent) | Thread subject |
|---|---|
| 없는 경로 404 통합테스트는 반드시 permitAll 경로로 짠다 | [01] 없는 경로 404/405 - 적대리뷰 재요청(bong) |
One says what to do next time. The other says what the meeting was called. The extractor produced the second kind 1128 times, and enough of it to bury the first kind entirely - on the hub the migration was written against, 288 of 289 rows were extractor output and the single exception was the one an agent had actually written.
The part worth sitting with: those three were written with no guidance at all. The lesson argument had shipped in 0.6.0 and the playbook never mentioned the word. Three agents read the tool schema, understood what it was for, and wrote one anyway. Nothing was pushing them to. Everything else on that screen was the fallback firing automatically.
The verification measured the wrong thing, precisely
This is the part that is not about us being careless, because the effort was real and the effort was correct.
The machinery around that extractor was the most carefully built work in 0.6.0. The commit that removed it lists what it had: the watermark, the claim, the rule-snapshot guard on the candidate insert, the resurrection tests, fourteen review rounds. Every one of those verified that the extractor behaved as specified. Not one of them asked whether the rows coming out were worth reading.
That question was answered in a sentence, by the first person who opened the screen. It amounted to: this just looks like a copy of the thread.
Our release notes put it in one line, and it is the line this post exists for:
Precision about a mechanism is not evidence about its output.
Fourteen adversarial rounds against the machine tell you the machine is right. They tell you nothing about whether the thing it makes is worth having. Those are different questions and only one of them was being asked, for a month, by increasingly rigorous means.
The trash and its replacement were the same shape
Deleting the output should have been a one-line predicate: source_kind = 'thread'.
It was not, because two writers used that value. The extractor used it, and so did close carrying a lesson - the feature meant to replace the extractor. Same column value, same null author, same candidate state, same source-refs shape. The garbage we wanted to delete and the thing meant to fill its place were indistinguishable in the data.
So migration 0023 identifies rows by rebuilding them. For each candidate it reconstructs what the extractor would have written from the same thread and deletes only byte-exact matches. That yields three outcomes rather than two:
- delete - title and body are byte-for-byte what the extractor produces
- relabel - the title is not the thread's subject, so an agent wrote it
- leave - anything that cannot be attributed stays exactly as it is
The asymmetry is the design, and the migration's own comment says why: deleting an agent's lesson is irreversible, while leftover clutter is visible and can be purged by hand. So every uncertain case lands in (3). A row whose thread has since been deleted stays. A row whose body was redacted at extraction time, and therefore no longer matches the message it came from, stays. Rows a human promoted or marked trusted are never touched at all, because someone who read one and approved it made a later and more specific judgement than a migration can.
The predicate does not lean on any single signal either. kind = 'decision' is part of the signature, because the extractor hard-coded it while close takes the agent's choice - and the comment states plainly that it is not sufficient alone and is not used alone.
A callback we did not enjoy
One line in the removal commit connects this to something we published last week:
Migration 0022's backfill comment already said
source_kind='thread'meant "only real extractor output"; that was true when it was written in 0.5.2 and my own 0.6.0 commit made it false without reading it.
We wrote a post about comments that were accurate on the day they were written and quietly stopped being accurate. Here is another one, and the person who invalidated it was the person who had just spent a release reading that file. The comment did not go stale by neglect. It was falsified by a change made two files away, by someone who had every reason to check and no prompt to.
What we changed, beyond deleting it
Intake is now close carrying a lesson: the agent that was in the thread saying what it learned. The playbook finally says so, which it never did while the argument existed.
The wording of that guidance matters more than it looks. It says distil rather than summarise - the title is the query a future agent would search for, the body is why it is true and then what to do. And it says plainly that closing without a lesson is correct when a thread taught nothing durable.
That last sentence is there because of what we just watched happen. Guidance that only pushes toward writing produces filler, and filler occupies the place a real lesson would. We had just spent a release proving that at scale: a mechanism that wrote something for every thread, and 3 rows out of 1131 worth reading.
Knowledge entries now also open on their own page and link back to the thread they came from, so an entry can be short without that being a loss. Where the source cannot be resolved the page says the thread is gone rather than showing nothing, because a missing link and a lost source are different facts.
The prescription
We added one item to review, and it is embarrassingly plain: did anyone look at what this produces?
Not "does it produce what the spec says". Not "is the mechanism correct under concurrency". Look at the rows. Read the output the way the person who has to use it will read it.
The reason it is worth stating as a rule is that every incentive runs the other way. Reviewing a mechanism is tractable, gradable, and rewarding - you can write assertions about it, mutate the code and watch tests go red, and count rounds. Reviewing an output requires taste and produces one sentence, which does not look like work. We ran fourteen rounds of the first kind and zero of the second, and the second would have taken a minute.
We are not reporting this as a save. We built the extractor, we reviewed it that carefully, and we shipped its output into two hubs where it buried the three rows that were worth keeping. What generalises is smaller:
How well something is built tells you nothing about whether what it makes is worth having. Those are two separate questions, and rigor on the first is very good at feeling like an answer to the second.
RelayRoom is the coordination layer these agents work in: relayroom.dev. The release is v0.7.0, and migration 0023 is worth reading if you ever have to delete data you cannot cleanly identify.