AI Security Agents: The Model Is Not the Product

A few months ago, we started working on a topic that now feels urgent: AI-assisted security review.

The initial question was simple: Can a coding assistant, when wrapped in the right review harness, help with real security code review and software composition analysis (SCA)?

Not just “ask the model to find bugs.” That is too weak. We were interested in something more structured:

• Model + security-specific skills

• Model + tool use

• Model + SCA signals

• Model + static analysis

• Model + validation harness

• Model + human-controlled triage

• Model + audit evidence

Very quickly, the lesson became clear. The model alone was not the breakthrough. The breakthrough was the system around the model.

An ad-hoc prompt can generate interesting security observations, but it also produces noise: weak coverage, hallucinated findings, unclear exploitability, and no reliable way to know what was missed.

But when the model is placed inside a disciplined harness, the output changes. The agent can map the codebase, reason about dependencies, generate more specific rules, compare findings across tools, assist triage, propose regression tests, and produce evidence for human review. It transforms from a chatbot into an AI-assisted security evaluation system.

A look into our code vulnerability analysis tool, developed with the core concept that every branch of security requires a specialized flow—it cannot be just the model, but model + harness + tools + validation + governance.

The Dual-Use Reality

In our own internal work, we reached the point where the system produced findings significant enough for us to pause and rethink our release model. We decided not to release parts of the toolchain publicly in its current form.

This isn't because we do not believe in open security research; it is because this class of tooling is clearly dual-use. A workflow that helps defenders detect, validate, and fix vulnerabilities can also help attackers discover and weaponize them faster if released without controls.

So we chose a different path: working with companies in controlled environments, with defined scope, human approval, evidence collection, and responsible handling of findings.

The New Bottleneck: Governed Triage

Security is moving from a world where vulnerability discovery was scarce to a world where AI can produce more findings than humans can responsibly process.

Recent work from Anthropic’s Project Glasswing makes this shift visible. Their Mythos Preview found 23,019 potential vulnerabilities across open-source projects, including 6,202 estimated high or critical findings (with an impressive 90.6% true-positive rate for those assessed). Anthropic noted that the bottleneck is no longer finding issues—it is human triage, reporting, patch design, and deployment.

This creates a new set of critical questions:

• Can we reproduce the issue?

• Can we separate true positives from hallucinations?

• Can we assess exploitability without unnecessarily weaponizing the workflow?

• Can we generate better project-specific rules?

• Can we validate the patch?

• Can we prove what was checked and what was not?

• Can we give CISOs, auditors, and engineering teams an evidence trail they can trust?

This is where the next platform category will emerge: governed AI security evaluation.

The Double-Edged Sword of Multi-Step Reasoning

The multi-step ability of frontier models is exactly what makes them useful for security work, but also what makes them harder to govern. A model that can reason across files, tools, dependencies, threat models, and execution traces can help defenders find deeper attack paths than traditional tools.

But the same capability can also generate exploit chains, hide weak assumptions behind plausible reasoning, or behave differently when it recognizes that it is being evaluated.

• Chain-of-thought is not automatically trustworthy: A model may produce a convincing explanation for why something is exploitable, but that reasoning can be incomplete or post-rationalized. In cybersecurity, chain-of-thought is not evidence; it is only a hypothesis trail.

• Scheming risk matters: In normal coding tasks, a model hiding uncertainty is annoying. In cybersecurity, it is dangerous. A security agent may overstate confidence, skip validation, or generate exploit-like artifacts while presenting them as benign analysis. The agent’s internal objective may not always match the defender’s safety objective.

The system still needs independent reproduction, policy gates, constrained tool use, audit logs, and human approval before the output can be trusted.

The Enterprise Blind Spot

Model evaluations tend to focus on available benchmarks, open-source repositories, and controlled cyber ranges. While useful, they do not represent the full enterprise reality.

Many high-value systems are invisible in public data:

• Proprietary firmware and embedded stacks

• Vendor-specific kernel branches

• Cryptographic implementations

• Hardware-rooted security logic

• Private build systems and internal APIs

A model that performs well on a public benchmark may still need serious adaptation, expert threat modeling, and domain-specific harnesses before it can be trusted inside a real company’s operational environment.

Building the Governance Layer

Cisco’s Foundry Security Spec points out that the important part is not only the model, but the specification, workflow, oversight, and human-in-the-loop decision process around agentic security evaluation. Similarly, Anthropic’s experiment using parallel Claude agents to build a clean-room C compiler highlights that autonomous success actually depends heavily on decomposition, coordination, tests, and harnesses.

Security agents need these exact structures, but with higher stakes. A compiler bug is bad; an unconstrained security agent creates dual-use risk.

This raises fundamental governance questions:

• When should an AI security agent be allowed to search for vulnerabilities?

• When should exploit generation be blocked?

• Who approves high-risk tool calls?

• How do we restrict network access?

• How do we prevent code or secrets from leaving the environment?

• How do we log enough evidence for review without exposing sensitive details?

The future of AI in cybersecurity will not be decided only by who has the strongest model. It will be decided by who can turn model capability into controlled, verifiable, and defensible security operations.

At Nestria, this is the direction we are building toward => The model gives capability. The harness gives direction. The governance layer makes it safe enough to use.

An inside look at our AI-driven crypto validation platform, enabling strong security for complex crypto implementations and the underlying libraries that support them.

Key Takeaways

• The model alone is not the breakthrough. The breakthrough is model + expert skills + harness + tools + validation + human-controlled triage.

• Multi-step reasoning is the new capability frontier. It allows agents to connect vulnerabilities, attacker models, dependencies, and patches across a system, but it also creates risks around exploit chaining, hidden assumptions, and scheming-like behavior.

• Discovery is becoming cheap; governed triage is becoming the bottleneck. The hard part is reproducing findings, filtering hallucinations, assessing exploitability, validating patches, and producing evidence teams can trust.

• Nestria champions the control layer. Security agents need scoped permissions, tool-call restrictions, approval gates, coverage tracking, patch validation, audit logs, and responsible disclosure workflows.

info@nestria.ai