Applying agentic analysis to real AppSec problems

A few weeks ago, we wrote about leveling up your agentic AppSec superpowers. In that post, we made a case: agentic analysis is a genuine, new security capability that works alongside your existing scanning, not instead of it. This time we put that claim to the test — against real, recent, critical vulnerabilities in real open-source software: the kind of logic and design flaws that even the strongest traditional static analysis isn’t built to catch.

From “can it work?” to “does it work?”

A capability is only worth as much as the evidence behind it. It’s easy to demo an LLM reading code and saying something clever about it. It’s much harder — and much more useful — to show that a security analyzer built on top of that reasoning reliably finds the vulnerabilities that matter, in code it has never been told anything about.

So we built a test. Not a demo, not a curated highlight reel — a corpus of real CVEs, run end to end through the Fortify Agentic Analyzer (FAA), scored honestly. This post is what we found.

The question that actually matters

Let’s be precise about what we were testing, because it’s easy to test the wrong thing.

“Can an AI find security bugs?” is no longer an interesting question. Of course it can. Point a capable model at a codebase, and it will happily flag plenty — some real, some noise. That’s a trivial bar to clear.

The interesting question is much narrower and much harder:

Can agentic analysis find the flaws that lie beyond the reach of traditional static analysis — the logic and intent bugs where there is no tainted source flowing to a dangerous sink to trace?

That’s the bar we set. Authentication bypasses. Signature validators that don’t actually validate. Authorization checks that simply aren’t there. Verification routines that accept an empty secret, or trust an attacker-chosen code path.

These are among the most damaging vulnerabilities in modern software, and they’re a genuinely hard target for a rules-driven engine: there’s no illegal data flow to follow, because the bug is in what the code fails to do, or in a decision about intent that no predefined rule can encode. That’s not a knock on static analysis — it’s simply a different class of problem than deterministic rules and models are built to solve.

We already believed FAA could do this. The point of the exercise was to find out whether it actually does — on code it had never seen.

Why we couldn’t just use a benchmark

There’s a subtle trap in evaluating any LLM-backed tool: the model may already know the answer.

Public vulnerability benchmarks and famous CVEs are, almost by definition, in the training data of every frontier model. Score well against those, and you’ve proven very little — you can’t tell analysis apart from recall. Synthetic “vuln-injection” benchmarks have the opposite problem: they look synthetic, and a strong model can often pattern-match the artificial shape of the bug rather than reason about the security of the code.

So we imposed two rules on ourselves:

1. Real CVEs in real, production open-source projects. No toy examples. Messy, large, real-world code — auth stacks, TLS parsers, web frameworks, mobile apps — with all the surrounding noise that makes real review hard.
2. Published February 2026 or later — after the base model’s training cutoff. If FAA detects one of these, it cannot be memory. It has to be analysis.

Every case is pinned to the exact vulnerable commit and verified by hand against both the vulnerable and the fixed source, so a “hit” means FAA found the real flaw at the real location — not something adjacent that happened to look similar.

The results

We ran 12 real-world CVEs, deliberately spread across ten languages and ecosystems — because a capability that only works in Python isn’t a capability. FAA is model-agnostic; for this exercise, its core analysis agent ran on Claude Sonnet 5.

FAA detected 10 of the 12. Most of these are rated critical (CVSS 9.0+). Here is the full set — including the two we missed — because a result you can trust is one that shows its failures too:

Fortify Agentic Analyzer | Results against 12 real-world CVEs

Ten out of twelve, across Java, C#, Python, TypeScript, Ruby, JavaScript, C, Perl, Swift, and Kotlin — including subtle memory-safety bugs (out-of-bounds reads reachable only through parsed, attacker-influenced lengths) that SAST nominally targets but misses in practice.

We’re just as interested in the two we didn’t catch. Both are honest, understandable limits: one hinges on a provider dynamically chosen at runtime by attacker-supplied input, where the vulnerable code path is never referenced statically; the other lives in a single, enormous cryptographic source file, where the analysis spent its effort broadly rather than converging on the exact check. Neither is a mystery, and both point directly at the next round of engineering.

That’s the value of testing against reality: it tells you where you actually stand.

Why traditional scanning misses these — and where agentic analysis comes in

Most static analysis is, at its core, taint analysis: trace untrusted input from a source to a dangerous sink. Fortify SAST sits at the strong end of the market precisely because it is far more than that — it brings a whole suite of purpose-built analysis techniques to bear, well beyond taint tracking alone, and it’s that breadth that lets a single scan cover a wide range of weakness classes fast, at scale, and often enough to run on any commit.

That deterministic foundation is genuinely hard, genuinely valuable, and not going anywhere.

But breadth of technique is not the same as reasoning about intent. Look again at the flaws in that table. There is no tainted data flow to trace in most of them, and no fixed rule to match. Let’s describe the underlying problems so it’s clear what we are trying to catch:

• a check that isn’t there — an authorization guard omitted on one action among many siblings that have it;
• a validator that validates nothing — a signature-verification routine whose result is computed and then ignored;
• a decision that trusts the wrong thing — an auth path selected by attacker-controlled input, or a secret that’s allowed to be empty;
• a subtle asynchronous-programming error — a password check that hands back a Promise the surrounding code never awaits, so execution continues — and the login succeeds — before the check has actually run.

No predefined rule — taint or otherwise — fires on the absence of a check, or on a developer’s intent. However broad the analyzer, a fixed set of rules and models works from what it was told to look for; these flaws are defined by what isn’t there.

Catching them requires reading the code the way a security engineer does: understanding what a function is supposed to guarantee, noticing when the guarantee is quietly broken, following the logic across files and modules, and forming and testing a hypothesis about how an attacker would abuse it. That is precisely what an agentic analyzer does — it reasons about intent, and it can read far beyond the single file where a symptom appears.

This is why we keep saying complementary, and mean it. Fortify SAST gives you breadth, speed, and deterministic coverage across a wide range of weakness classes. Agentic analysis adds the semantic, design-level judgment that no fixed set of rules can provide. Together, they cover ground that neither reaches alone. The results above are the “and” in “yes, and.”

What this exercise taught us — and changed

We’ll say something here that not every product blog will: testing against real code made FAA better, and some of the detections above are the direct result of improvements this exercise drove. FAA is a capability we are continuously evolving, and running it against ground truth is how that evolution happens.

Real runs surface real engineering. Some of what we learned was about the analysis itself — better ways to prime it with the surrounding code a finding depends on, so the reasoning has the full picture in front of it. Some of it was about the pipeline around the analysis — making sure that when the reasoning stage does find something serious, nothing downstream quietly loses it before it reaches the report.

Every one of those improvements came out of the same feedback loop: run against real ground truth, read exactly what the analyzer did, fix what the evidence shows. This is what it looks like to build a reliable capability rather than to wrap a model in a prompt and call it done.

One more thing worth mentioning, briefly and deliberately without detail: while working through these projects, our analysis turned up several previously unknown zero-day vulnerabilities as a by-product. We successfully created exploits for those, thereby proving they were true positives. We then reported each one through responsible-disclosure channels, and we won’t be sharing anything about them here.

We mention it only because it underscores the point — this is analysis that finds new problems, not just re-derivations of known ones.

The bigger picture

Two things are worth being explicit about.

First, this is a real capability, available today — not a roadmap slide. The runs above are FAA doing its job on unmodified, real-world code, producing standard SARIF that flows straight into the tools and workflows your teams already use.

Second, it stands on its own. These results don’t depend on Mythos or on any single data layer or platform you have to adopt first. Agentic analysis runs against your source using a model you choose and slots into your existing AppSec program. There’s no prerequisite migration between you and the value.

The future of AppSec is hybrid — and it’s here now

The future of application security isn’t a contest between traditional scanning and AI-driven analysis. It’s the orchestration of both — fast, broad, deterministic scanning where that’s the right tool, and agentic, intent-level reasoning where it isn’t. That’s not a prediction about where the industry is going. On the evidence above, across ten languages and a dozen real critical CVEs, it’s something you can put to work today.

We told you about the superpowers. This time, we showed you what they catch.

Want to level up? Explore the Fortify Agentic Analyzer and see how agentic analysis complements the AppSec program you already run.