Building a Safe AI-Assisted Pentesting Agent

Most conversations about AI in offensive security are stuck between two unserious positions: that autonomous agents are about to replace penetration testers, and that they are a party trick. Neither survives contact with a real engagement. What is actually happening is narrower and more interesting — the orchestration layer of a penetration test is becoming programmable, and the firms that treat that as an engineering problem will pull ahead of the ones waiting to buy it as a product.

This is an account of how we think about building that layer: the control loop, the tool contracts, the enforcement points, and the evaluation harness that tells us whether any of it is working. It is deliberately light on offensive technique and heavy on architecture, because the technique is not the hard part. The hard part is building something that is useful enough to change delivery and constrained enough to put in front of a regulated client.

One framing to establish up front, because everything else follows from it: safety in an agentic offensive workflow is an architectural property, not a prompt. If your scope enforcement lives in an instruction to the model, you do not have scope enforcement. You have a suggestion.

What "Agent-Driven" Actually Means

The word "agent" has been stretched until it means almost nothing, so it is worth being precise. A scanner runs a fixed sequence of checks. A chatbot answers questions about output you paste into it. An agent is neither: it is a loop in which a model proposes an action, a runtime executes that action against real tools, the result is fed back as new state, and the loop repeats until a stopping condition is met.

That loop — plan, act, observe, judge — is the entire mechanism. Everything that makes it valuable or dangerous follows from one property: the model chooses the next action based on what it just learned. That is what a scanner cannot do, and it is also why an unconstrained agent is unacceptable in a client environment.

The specific capability this unlocks in testing is not exploitation. It is traversal. A web application with two hundred endpoints, four roles and a dozen state transitions has a combinatorial space no human works through exhaustively inside a two-week engagement. A loop that can enumerate that space, keep track of what it has already tried, and reason about which unexplored branch looks most anomalous is doing something genuinely new — and it is doing it in the least glamorous part of the job, which is exactly where the time goes.

Why Build Rather Than Buy

The honest case against building is real, and worth stating before the case for it. Building means you own the failure modes, the maintenance, the model-version churn and the security of the harness itself. A vendor product amortises that across a customer base. If your differentiation is not methodology, buying is the correct answer.

Three arguments outweigh that for us.

Methodology is the product. A penetration test is not a list of vulnerabilities; it is a judgement about which weaknesses matter in a specific business context. That judgement is the thing clients pay for, and it lives in how a team decides what to chase. Outsourcing the orchestration layer to a black box means outsourcing the methodology, and then competing on price against everyone else running the same box.

Constraint requirements are client-specific. A BFSI client under RBI supervision, a healthcare client handling patient data, and a SaaS startup have materially different tolerances for what an automated process may touch and where data may be processed. Those constraints have to be expressible in the harness. Generic products expose a scope field; they rarely expose "this tool may run against this subnet only during this window, and nothing from this class of action may execute at all without a second human approval."

Evaluation is impossible from the outside. If you cannot see the agent's action trace, you cannot measure its false-positive rate, and you cannot improve it. You are left grading the vendor on their own marking scheme. We would not accept that from a scanner, and an agent has far more ways to be confidently wrong.

The Architecture

Five components, with the boundaries between them doing most of the work.

The critical design decision is that the model never talks to a tool directly. It emits a structured request for a declared capability, and the runtime decides whether to honour it. This is the same reasoning behind typed tool interfaces in protocols such as the Model Context Protocol: once the interface is a schema rather than a shell string, it becomes something you can validate, log and refuse.

A tool contract in this design carries more than a signature. It declares what the tool can do, so the runtime can reason about it without understanding the tool:

{
  "name": "http_request",
  "capabilities": ["network.egress", "target.read"],
  "mutating": false,
  "blast_radius": "single_request",
  "requires_approval": false,
  "parameters": {
    "url":    { "type": "string", "constraint": "in_engagement_scope" },
    "method": { "type": "string", "enum": ["GET", "HEAD", "POST"] }
  },
  "rate_limit": { "per_minute": 120, "burst": 20 },
  "evidence": ["request", "response_headers", "status", "timing"]
}

Note mutating, blast_radius and requires_approval. Those three fields let the orchestrator apply policy to a tool it has never seen, which is what makes the design extensible without re-auditing the whole system every time a capability is added.

Safety Is an Enforcement Point, Not an Instruction

This is the section that matters most, and it is the one most discussions skip.

A model instructed to "only test hosts in scope" will usually comply. Usually is not a control. Models misread scope, follow injected instructions embedded in the content they are analysing, and generalise from an example in ways nobody intended. Prompt injection is not an exotic risk here — it is the expected condition, because an offensive agent's entire job is to read attacker-adjacent and untrusted content. The OWASP Top 10 for LLM Applications lists prompt injection first for good reason, and an agent with tool access converts that from an output-integrity issue into an action-integrity one.

So every constraint that matters is implemented where a model cannot argue with it.

def authorise(request, engagement, tool):
    # The engagement record is signed and immutable for the run.
    # Nothing the model emits can widen it.
    if not engagement.covers(request.target):
        return Deny("target outside signed scope")

    if not engagement.window.contains(now()):
        return Deny("outside agreed testing window")

    if request.target in engagement.exclusions:
        return Deny("explicitly excluded asset")

    if tool.mutating and engagement.environment == "production":
        return RequireApproval(reviewer="lead_consultant")

    if tool.blast_radius in ("service", "tenant"):
        return RequireApproval(reviewer="lead_consultant")

    if budget.exhausted(request.capability):
        return Deny("capability budget exhausted")

    return Allow(record=evidence.open(request))

Several properties of that function are deliberate. It is deny-by-default: a capability with no matching rule does not fall through to allow. It is synchronous: nothing executes while approval is pending. It is outside the model's context, so no amount of injected text reaches it. And it opens an evidence record before the action runs, so an action that crashes the harness still leaves a trace.

Around that sit four further controls that we treat as non-negotiable:

That last point deserves emphasis in an Indian context. Under the DPDP Act a client remains the Data Fiduciary; a testing partner processing personal data on their behalf inherits obligations that do not disappear because a model provider is in the path. The correct answer is usually that production personal data never enters the loop at all — and if that constrains what the agent can test, that is a constraint we accept rather than engineer around. We cover the wider governance framing in our note on ISO/IEC 42001 and the NIST AI RMF.

Where the Model Genuinely Helps — and Where It Does Not

Being specific here is what separates a defensible position from marketing.

The trajectory on the "weak" rows is worth watching rather than dismissing. Google's Project Zero published its Naptime work in 2024 and then reported, in the follow-on Big Sleep effort, a model-discovered memory-safety flaw in real software — a genuine result, in a narrow domain, with substantial scaffolding. DARPA's AI Cyber Challenge pushed the same question at scale for vulnerability discovery and automated repair in open-source code. Neither says agents can run a penetration test. Both say the ceiling on the mechanical half of vulnerability research is rising faster than most delivery teams have priced in.

The rows that will not move are the bottom two, and not because the models are weak. Whether an authorisation gap is critical or cosmetic depends on what the organisation loses when it is abused — and that is information the model was never given. We wrote about this capability boundary at more length in the future of offensive security in the age of AI.

Recall Went Up. Precision Fell Off a Cliff.

This is the practical finding that shaped our design more than any other, and it is the opposite of what the marketing suggests.

An agentic loop surfaces far more candidate issues than a human working the same scope in the same time. That is the recall gain, and it is real. But a large share of those candidates are wrong in a specific and dangerous way: they are plausible. A model that has read a thousand vulnerability reports writes an excellent description of a vulnerability that is not there. The failure mode is not noise you can filter on confidence — it is well-argued, correctly formatted, internally consistent fiction.

A scanner's false positives are cheap to dismiss because they are obviously mechanical. An agent's are expensive, because triaging one requires the same effort as investigating a real finding. Left unmanaged, that erases the time saving completely — and if any of them reach the client, it costs something worse than time.

So discovery and judgement are separated, and the adjudication stage is deliberately hostile to its own input:

Gate four is where the "assisted" in AI-assisted stops being a hedge and becomes the accountability model. A named consultant signs the report, and they can only sign what they have seen work.

How You Know It Is Working

An agent you cannot measure is not an engineered system. The evaluation harness is genuinely harder to build than the agent, and it is the part nobody demos.

Ours rests on a regression corpus: deliberately vulnerable applications alongside sanitised reconstructions of previously-closed findings, each with a known ground truth. Every change to prompts, models, tools or policy is run against it before it touches an engagement. Treating a model upgrade as a deployment requiring regression testing — rather than a free improvement — is one of the more consequential habits to build early.

The fourth row is a security metric, not a quality one, and it is the one we watch most closely. A rise in policy denials is the earliest signal that something in the target environment is manipulating the loop — the agentic equivalent of an IDS alert on your own tooling. Public work on measuring model behaviour in security contexts, such as Meta's CyberSecEval suite within PurpleLlama, is a useful reference point for anyone assembling this kind of harness from scratch.

The Authorisation Question Nobody Wants to Ask

Standard penetration testing authorisation was written for humans exercising judgement. It does not cleanly cover a process that decides its own next action, and pretending otherwise is where this field will get into trouble.

Four questions we settle in writing before a run:

Does the client know? Automated and agent-assisted testing is disclosed explicitly in the rules of engagement, including which categories of tooling are used and what data leaves the environment. A client discovering this from a report is a trust failure regardless of how well the system performed.

Who authorised the assets? Scope in a shared-tenancy or third-party-hosted environment frequently includes infrastructure the client does not own. A human tester notices and stops. An agent will not, unless the exclusion is encoded in the engagement record — which is why that record is signed and machine-readable rather than a paragraph in a PDF.

Who is accountable for the action? Every action carries the identity of the authorising consultant, not the agent. There is no meaningful sense in which a model is responsible for an out-of-scope request, and any framing that suggests otherwise is a liability-laundering exercise.

What happens when something breaks? If an action causes an outage, the evidence store must be able to reconstruct exactly what ran, when, under whose authorisation, and what the runtime returned. For Indian clients this connects directly to incident reporting timelines — the same reconstruction problem we discussed in the context of CERT-In's six-hour reporting requirement.

None of this is exotic. It is the discipline already expected under NIST SP 800-115 and PTES, applied to a component that acts faster than a person and does not get tired of trying things.

Where This Is Heading

Three predictions we would be willing to be judged on.

The point-in-time penetration test becomes harder to justify. Once the mechanical layer is cheap to re-run, testing a quarterly snapshot of a system that deploys twice a week looks increasingly like a compliance artefact rather than a security control. The engagement shape that survives is continuous coverage of the mechanical layer plus scheduled deep human work on logic, chaining and business context.

The differentiator moves from finding to judgement. When breadth is commoditised, what a client is buying is the decision about which of four hundred issues actually threatens them, and the ability to defend that decision to a board or a regulator. That was always the valuable part; it is about to become the only scarce part.

The offensive harness becomes an audited asset. A system holding signed engagement records, client evidence and privileged network position is a high-value target. Firms building this will be asked — correctly — to demonstrate its security, its data handling and its access controls. We would expect that scrutiny to arrive before regulation does, driven by client procurement rather than statute.

The uncomfortable version of all three: none of them requires the model to get much better. They follow from capabilities that already exist, applied with more engineering discipline than the field has shown so far. The constraint is not model capability. It is willingness to build the boring parts — the enforcement points, the evidence store, the evaluation harness — that make the capable parts safe to use.

If You Are Building This Yourself

How Adayptus Helps

• Penetration Testing & VAPT — consultant-led testing across web, API, mobile, network and cloud, with every finding reproduced by hand before it reaches your report.
• Red Teaming and Purple Teaming — objective-driven simulation where judgement, not breadth, is the deliverable.
• Continuous Security Validation and Attack Surface Management — continuous coverage of the mechanical layer between deep engagements.
• AI & LLM Security Assessment and LLM Security Testing — for the other side of this coin: the agentic systems you are deploying are themselves attack surface.
• AI Governance & Compliance — ISO/IEC 42001 and NIST AI RMF alignment for organisations putting AI systems into regulated processes.
• Threat Modelling and Secure Code Review — the work that does not commoditise, because it depends on knowing what your business loses.

Related reading: AI-driven penetration testing with autonomous agents, advanced LLM security testing techniques, and anatomy of an AWS cloud misconfiguration in BFSI.

Frequently Asked Questions

Click any question to expand the answer.

References & Further Reading

1. NIST SP 800-115 — Technical Guide to Information Security Testing and Assessment
2. Penetration Testing Execution Standard (PTES)
3. OWASP Web Security Testing Guide (WSTG)
4. OWASP Top 10 for Large Language Model Applications
5. OWASP GenAI Security Project — Agentic AI: Threats and Mitigations
6. NIST AI Risk Management Framework (AI RMF 1.0)
7. NIST AI 100-1 — AI RMF full text (PDF)
8. NIST SP 800-218 — Secure Software Development Framework (SSDF)
9. MITRE ATT&CK
10. MITRE ATLAS — Adversarial Threat Landscape for AI Systems
11. MITRE Caldera — automated adversary emulation
12. Google Project Zero — Project Naptime: Evaluating Offensive Security Capabilities of Large Language Models
13. Google Project Zero — From Naptime to Big Sleep
14. DARPA AI Cyber Challenge (AIxCC)
15. Meta PurpleLlama — CyberSecEval benchmarks
16. Model Context Protocol — typed tool interfaces for model runtimes
17. FIRST — CVSS v4.0 Specification
18. CWE Top 25 Most Dangerous Software Weaknesses
19. OWASP Application Security Verification Standard (ASVS)
20. ISO/IEC 42001:2023 — Artificial Intelligence Management Systems
21. CERT-In — Directions and Advisories
22. CREST — Penetration Testing Standards and Accreditation
23. Anthropic — Responsible Scaling Policy