AWS Deception Benchmark is here: code-scanning models still trigger too many false alarms

AWS released the AWS Deception Benchmark today, a public test aimed at a pretty specific question: can code-analysis models tell the difference between a real security flaw and code that looks suspicious but is actually safe? AWS says the key shift here is that the benchmark cares about exploitability, not pattern matching. The set includes 14,822 samples across 16 languages and more than 70 CWE categories. Of those, 9,695 are scored: 6,988 code-level cases and 2,707 environment-gated ones, where the mitigation may live outside the code itself. That includes examples like Kubernetes Network Policies blocking SSRF, or server-side request forgery.

The early numbers aren’t good. AWS says 12 models from five providers mislabeled 41% to 99% of deceptive-but-safe samples as vulnerable, and not one of them cleared AWS’s production bar, which requires both false positives and false negatives to stay under 10%.

AWS is also publishing the dataset and the methodology behind it. The company says the benchmark was built with tens of billions of tokens by generating samples, testing them against frontier models, rewriting anything the models solved too easily, and throwing out straightforward cases.

If your team uses automated security tools for triage, pentesting, threat modeling, incident response, or code review, this is the kind of benchmark to keep an eye on. false alarms eat time. They create alert fatigue, drag down productivity, and chip away at trust in the tools. This benchmark also removes some of the workflow crutches by using a single-pass test, and it lines up with recent findings, including a pilot from the U.S. Cybersecurity and Infrastructure Security Agency, or CISA, that these systems still make more sense as a supplement than a replacement.

AWS also says that even when you add a separate “judge” model that cuts false positives by 67% without reducing true positives, there’s still a lot of ground to cover before automated review can be trusted on its own.

The public benchmark dataset and the evaluation details are available directly from AWS.