1Password study: only 26% of AI security patches fully fix flaws

1Password’s Off-by-1 Labs has published a new study on AI-generated security patches, examining 6,080 fixes for six recently disclosed vulnerabilities. By the lab’s count, only 26% fully remediated the issue without changing application behavior. 49.3% still left at least one exploitable attack path in place, 20.1% fixed the flaw but changed behavior, 2.3% introduced a new vulnerability, and 2.2% did both: they failed to fix the original problem and added a new weakness.

To reduce the chances that the models were simply recalling training data, the researchers used OpenAI’s ChatGPT-5.5 and Anthropic’s Claude Opus 4.8 on recent CVEs. The results were rough. For complex flaws, 53.9% of the patches were marked “FLAWED,” meaning the code looked convincing on the surface and sometimes even passed early tests.

Even then, the researchers say more than one-third of the fixes that initially seemed successful were later judged fragile. Some blocked a proof of concept, including the SpringAI character-filter cases, but still didn’t address the root cause or account for architectural intent, business requirements, security consequences, or long-term maintainability.

If you use automated coding tools for security work, this update is worth a look. Those tools can speed up development and help with triage, but when security-sensitive code and customer data are on the line, you still need skilled human review. An autonomous fix engine isn’t enough.

You can read the Off-by-1 Labs research online.