*How many "bugs" can some finite chunk of code possibly have? Will it ever run out of bugs? Can models just invent new forms of bugs to patch indefinitely?
@bruces I've been using Opus to look at an OSS project I work on (something like a couple million lines of code, almost three decades old), and it's finding a large number of issues. At least half the stuff it finds is legitimately a security issue, but quite low-grade issues. Stuff that would require some kind of elevated access to already exist to take advantage of it. We'll fix them, because it's noise for future scans with LLMs, but it's not something we're urgently rushing out to users.
@bruces so far, the AI tools have not found *new forms* of bugs. They find the same forms, just new instances. So in theory they could eventually run out... but we keep adding new bugs, and the tools are still far away from finding *all* issues.
@bruces in terms of arms race, EVERY interested party can hunt at the speed of the cheap methods, right