the invisible bug

AI is spectacular at fixing bugs created by humans. It was trained on that. I am seeing PRs of fifty files changed, and experience tells me there will be blood, I mean bugs. But not the kind we are used to.

These bugs will be invisible to the models because the models created them in the first place. If they could see them, they would not have made them, the same way our own logical flaws are invisible to us: if we had thought of it, we would not have done it; if we could see it, we would have fixed it; if we knew what the problem in the spec was, we would have asked, declared ourselves blocked, gone looking for the answer. You do not catch your own typo on the read-through. Your brain already knows what you meant. That is the fallout from AI. A codebase that is no longer the source of truth for human implementation. One that bears the unmistakeable fingerprints and limitations of its AI authors.

I have some first-hand experience with mini-model collapse. I have a habit of pitting two models against each other, the classic agent-as-judge pattern. When it works, it's glorious. Then the hallucinations creep in. Drift from one model amplified by the other, recursively repeated until they are both chasing rabbits like rabid greyhounds. Fill up the context window of one session and it starts to lose the thread, and that's the starter pistol on the stampede. That is the work product we are inviting into our codebases, a lexically communicated virus that escaped the immune system of the first model and threatens the next with the same blindness.

Here is the part that keeps me up. Software bug, disease vector, same word, and for once the pun does real work. AI has no natural immunity to AI-generated defects. It was born in the old world, immune system battle-tested and hardened against the old world's bugs. Human bugs. We are about to enter the new world, and some of the same immune responses will still function. Others will inevitably slip right by. Some of these are viruses: not inert foreign objects sitting in the tissue waiting to be found, but sequences that integrate into the reproductive machinery itself, carried forward into every refactor that touches the infected code. You have to know the shape, the pattern, the fingerprint of a virus to mount a response to it before it can replicate to wreak full potential damage. Your immune system has to have seen it (or something like it) before.

That fifty-file PR has bugs. Experience tells me. The author knows it, I know it, he just needs to 'get it done'. I don't have time to painstakingly review fifty files for him either, AI makes us faster, remember? So the vector slips past us, unchallenged, a new strain riding a bat out of a cave, and it hides in the host corpus, dormant. Invisible.

The patient has a retrovirus and I trained on bone saws.