AI security

AI and LLM penetration testing: what it covers

Businesses are shipping chatbots, internal copilots and RAG assistants faster than they’re testing them. A standard web application test checks the code around the model — not whether the model can be talked into leaking data or taking actions it shouldn’t.

Why LLM applications need different testing

In an LLM application, natural language influences behaviour — and that language can come from users, documents, emails, web pages or tool outputs. The security boundary is much fuzzier than in conventional software.

The OWASP Top 10 for LLM Applications

  1. Prompt injection — direct, and indirect via instructions hidden in a document, email or web page.
  2. Sensitive information disclosure — personal data, other users’ data, credentials or internal details.
  3. Supply chain — third-party models, plugins, datasets and libraries.
  4. Data and model poisoning — manipulated training, fine-tuning or knowledge-base data.
  5. Improper output handling — model output passed unsafely to browsers, databases or shells (XSS, SQL injection and similar).
  6. Excessive agency — more tools, permissions or autonomy than the task needs.
  7. System prompt leakage — exposure of hidden instructions, especially those containing secrets or security logic.
  8. Vector and embedding weaknesses — RAG stores leaking data across users or tenants, or being poisoned.
  9. Misinformation — confident, wrong output that users or processes act on.
  10. Unbounded consumption — abuse that drives up cost or degrades availability.

What a tester actually does

  • Maps the architecture — model provider, system prompt, data sources, tools and where output goes.
  • Tests data boundaries — for example, can one user retrieve another’s data through RAG?
  • Plants indirect injections in content the system ingests.
  • Abuses tools with attacker-chosen parameters.
  • Tests the output path for classic injection flaws.
  • Tests the surrounding application — authentication, authorisation, API security and rate limiting.

What good looks like

The strongest controls are architectural: the model only receives data the current user is allowed to see, tools run with least privilege, sensitive actions need human confirmation, and model output is treated as untrusted input.

Particularly relevant in financial services

Client-facing assistants that can see account data, and internal copilots connected to CRM and document stores, should be tested before clients — or the regulator — find the gap.

Frequently asked questions

What is LLM penetration testing?

Security testing of LLM-based applications covering prompt injection, data leakage, unsafe output handling, excessive agency and RAG weaknesses — alongside conventional application security testing.

Is the OWASP Top 10 for LLMs different from the OWASP Top 10?

Yes. It’s a separate list that covers risks specific to LLM applications.

Sources & further reading

Shipping an AI feature?

Fixed-price testing, scoped and delivered by a senior tester, with live findings on the SMARTSEC client platform.