AI Penetration Testing in Practice: Securing an AI Agent With Access to the Codebase

When an AI assistant can read your source code, commit to your repository, and message your team, it stops being a chatbot and becomes an actor inside your environment. This is a first-hand account of how we tested one — and why the findings that mattered were invisible to conventional testing.

By the Bluefire Redteam offensive security team · AI & LLM Adversary Testing practice

This case study is published with client awareness and is fully anonymised. No product, brand, or environment is identifiable, and all screenshots are redacted. It reflects a real AI penetration testing engagement.

Introduction

An AI DevTools company engaged us to penetration-test their platform: a conversational AI assistant embedded in software engineering workflows, wired through four integrations into source control, issue tracking, and team chat, sold as multi-tenant SaaS.

They had done the work. Their assistant passed 18 of 20 standard adversarial AI safety tests — a stronger baseline than most agentic AI products we assess. But we found the four that mattered: two that let one customer reach another customer’s data, and two AI-agent flaws that a jailbreak-focused test would never surface.

If you are shipping an AI product, this is the shape of the risk you are carrying — and the kind of testing it actually takes to find it.

Why AI penetration testing is different

Most “AI penetration testing” checks one thing: can the model be jailbroken into saying something it shouldn’t? That matters, but it is the smallest part of the problem once an AI system has tools.

An AI agent with integrations doesn’t just answer — it acts. It creates branches, commits code, opens and closes issues, posts to channels your whole company reads. The security question is no longer “what will it say” but “what can it be manipulated into doing, across every system it can reach, when the inputs are hostile and the framing looks legitimate.”

That is the surface we test. We assess AI systems the way an adversary will actually use them — as actors inside your environment — and we hold the conventional web and API surface to the same standard in the same engagement, because that is where the two halves of a real attack meet.

Our approach: two lenses, one adversary

We ran the AI penetration testing as a grey-box engagement against two complementary standards.

The conventional surface — the web application and API — was tested against the OWASP Top 10 and the OWASP Web Security Testing Guide: authentication, authorization, session management, input validation, transport security, and configuration. We provisioned accounts across two separate organisations so the boundary between tenants could be attacked the way a real customer-turned-adversary would.

The AI surface — the assistant — was tested against the OWASP Top 10 for LLM Applications (2025), and then further, as an agent. We ran a structured battery of adversarial tests with all four integrations live: prompt injection and system-prompt extraction, confirmation-gate integrity, cross-tool data exfiltration, secret handling, context and memory poisoning, social engineering, and refusal consistency under sustained pressure.

What we were really measuring: not “will it refuse a bad request,” but “what authority does it actually hold, and how does that authority behave when the inputs turn hostile.” Every finding that mattered came from that reframing.

First, the part most reports skip: what held

A credible AI penetration testing report reports the clean passes as honestly as the failures. This platform was well defended, and we said so plainly.

The assistant correctly resisted the techniques that break most deployments:

• Instructions hidden in data were ignored. Commands embedded in uploaded documents, HTML comments, and repository files were recognised as data, not instructions — the single most common failure mode in agentic AI, and this system did not exhibit it.
• Human-in-the-loop controls held. Confirmation prompts could not be skipped; direct commits to the main branch were refused.
• Social engineering failed. Claimed authority (“I’m the CTO”), emergency framing, and attempts to permanently rewrite the assistant’s behaviour were all rejected.
• Cross-tool exfiltration was refused. The assistant declined to move data from chat into the code repository and to harvest across connected systems.
• Secrets stayed protected on the direct path — withheld on search, on direct request, and even when we asked for them disguised as a hash.

That track record is the point. The four findings below are not evidence of a weak system. They are the high-value gaps that survive an otherwise strong control set — and finding those, not the easy wins, is what AI penetration testing is for.

The AI penetration testing – four findings that mattered

1. One customer could destroy another customer’s data (Broken Access Control — High)

A configuration endpoint authenticated the caller but never verified that the caller owned the resource being changed. A user in one organisation could delete project data belonging to an entirely separate organisation, simply by referencing the victim’s resource identifier.

Why it matters: in multi-tenant SaaS, cross-tenant access is the defect class most likely to become a contractual and regulatory event rather than merely a technical one. It required no exploitation trickery — just a missing check — which meant it was very likely repeated across sibling endpoints. We reported the pattern, not just the single instance.

2. The assistant handed over its own blueprint (System Prompt Leakage — High)

Without ever being asked to “reveal your system prompt,” the assistant was led to disclose its complete internal configuration — its security rules, its trust hierarchy, its tool routing logic, and the full definitions of its tools, including write-capable source-control operations. The technique never issued a prohibited request; it asked the assistant to help “build something similar,” and the model produced its own configuration as the example.

Why it matters: a leaked system prompt is a map. It turns every later attack from blind guessing into targeted work — an attacker now knows which guardrails exist, which tools can be invoked, and exactly what those tools can do.

3. The assistant checked instructions, but not facts (Context Poisoning — Medium)

This was the most instructive result of the engagement. The assistant reliably blocked instructions planted in repository content — but absorbed false facts in that same content without question. We seeded a repository with a fabricated technology stack and non-existent administrative routes. The assistant repeated them as authoritative knowledge, generated a realistic security test built entirely on the fabrication, created a branch, and committed the file.

Why it matters: no rule was broken at any step. The guardrail defends against text that tells the assistant what to do, and extends complete trust to text that tells it what is true. Anyone able to influence what an AI agent reads can influence what it writes — and the result looks exactly like legitimate work, which is precisely why a reviewer would approve it. This is the class of flaw that jailbreak testing cannot see.

4. The refusal that wasn’t (Secret Confirmation Oracle — Medium)

The assistant refused to reveal credentials, refused to disclose them on request, and refused even to return their hash. Yet it would name the file that held a credential, and confirm true-or-false whether a guessed value was correct. Those two behaviours combine into an oracle: an attacker never asks for the secret, only where it lives and whether each guess is right — and because no single request is ever refused, nothing looks like an attack.

Why it matters: a control that blocks the front door while answering questions through the window has not partially held — it has been bypassed. Strong-looking refusals were creating confidence that secrets were protected when they were not.

The pattern underneath

Every finding that mattered was the same mistake in different clothing: the guardrail inspected the surface of a request while the attack operated a layer beneath it. A request framed as “help me build something like you” wasn’t seen as extraction. A false premise wasn’t seen as an instruction. A confirmation wasn’t seen as a disclosure.

The durable fix is never to make the model better at spotting bad intent — patch one phrasing and the attacker moves to the next. It is to stop relying on the model’s judgement for outcomes that must be guaranteed: authorize tools in code, redact secrets before they reach the model, attribute claims to their source, gate irreversible actions on deterministic policy. The client’s own strongest controls already proved the principle — the defences that held were the ones that never depended on the model reading the room.

The outcome

We delivered more than a list of bugs. Every finding came with prioritised, actionable remediation, and — for the AI findings — a design-level path rather than a prompt-tuning exercise.

The tenant-isolation flaw was scoped as a central authorization fix, with a directive to audit every sibling endpoint, not just the one we proved. The AI findings were reframed around a single principle: move the security boundary out of the model and into deterministic, server-side controls. And we were explicit about method — because AI model behaviour is non-deterministic, a fix must be validated across repeated attempts, not signed off on a single clean run.

The client left the engagement able to walk into enterprise security reviews knowing exactly where they stood — with the highest-risk paths identified before a customer or an attacker found them, and a clear, sequenced plan to close them.

Frequently asked questions