the invisible bug

AI is spectacular at fixing bugs created by humans. It was trained on that. I am seeing PRs of fifty files changed, and experience tells me there will be blood, I mean bugs. But not the kind we are used to.

These bugs will be invisible to the models because the models created them in the first place. If they could see them, they would not have made them, the same way our own logical flaws are invisible to us: if we had thought of it, we would not have done it; if we could see it, we would have fixed it; if we knew what the problem in the spec was, we would have asked, declared ourselves blocked, gone looking for the answer. You do not catch your own typo on the read-through. Your brain already knows what you meant. That is the fallout from AI. A codebase that is no longer the source of truth for human implementation. One that bears the unmistakeable fingerprints and limitations of its AI authors.

I have some first-hand experience with mini-model collapse. I have this habit of pitting two models against each other, the classic agent-as-judge pattern. When it works, it's glorious. Then the hallucinations creep in. Drift from one model amplified by the other, recursively repeated until they are both chasing rabbits like rabid greyhounds. Fill up the context window in one session, it starts to lose the plot, and that's the starter pistol on a stampede. This is the work product we are inviting into our codebases, lexically communicated viruses that escaped the first immune system and threatens the next model with the same blindness.

Here is the part that keeps me up. Software bug, disease vector, same word. AI has no natural immunity to AI-generated defects. It was born of the old world, immune system battle-tested and hardened against old world's bugs. Humans bugs. We are about to enter a new world, and some of the conditioned immune responses will kick in. But some infection will inevitably slip right by. Some are like viruses: not inert foreign objects sitting in the tissue waiting to be found, but sequences that integrate into the reproductive machinery itself, carried forward into every refactor that touches the infected code. You have to know the shape, the pattern, the fingerprint of a virus to mount a response to it before it can replicate, before it can wreak full potential damage. Your immune system has to have seen it (or something like it) before.

That fifty-file PR has bugs. Experience tells me. The author knows it, I know it, but he just needs to 'get it done'. I don't have time to painstakingly review fifty files dense with intricate changes for him either, AI makes us faster, remember? So the vector slips past, unchallenged, a new strain riding a bat out of the lab, and it hides in the host corpus, dormant. Invisible.

The patient has a retrovirus and I trained on bone saws.