Businesses are shipping chatbots, internal copilots and retrieval-augmented generation (RAG) assistants faster than they're testing them. A traditional web application test checks the code around the model. It won't tell you whether the model can be talked into leaking data or taking actions it shouldn't.
Why LLM applications need different testing
In a normal application, code decides what happens. In an LLM application, natural language influences what happens, and that language can come from users, documents, emails, web pages or tool outputs. Any of it can carry instructions. The security boundary is fuzzier, and testing has to reflect that.
The OWASP Top 10 for LLM Applications (2025)
The OWASP Top 10 for LLM Applications (2025) is a useful framework for scoping this kind of test:
- Prompt injection: direct ("ignore previous instructions") and indirect, where instructions are hidden in a document, email or web page the model reads.
- Sensitive information disclosure: the model revealing personal data, other users' information, credentials or internal details.
- Supply chain: risks from third-party models, plugins, datasets and libraries.
- Data and model poisoning: manipulated training, fine-tuning or knowledge-base data.
- Improper output handling: model output passed unsafely into browsers, databases, shells or other systems (XSS, SQL injection and similar, via the LLM).
- Excessive agency: the model having more tools, permissions or autonomy than it needs.
- System prompt leakage: exposure of hidden instructions, especially when they contain secrets or security logic.
- Vector and embedding weaknesses: RAG stores that leak data across users or tenants, or can be poisoned.
- Misinformation: confident, wrong output that users or downstream processes act on.
- Unbounded consumption: abuse that drives cost or degrades availability.
What a tester actually does
- Maps the architecture: model provider, system prompt, data sources, tools and APIs the model can call, and where its output goes.
- Tests data boundaries: can user A retrieve user B's documents through the RAG pipeline? Can the assistant be coaxed into summarising content the user isn't authorised to see?
- Plants indirect injections in documents, emails or pages the system ingests, to see whether they change behaviour or trigger actions.
- Abuses tools: attempts to make the model call functions with attacker-chosen parameters, such as sending email, querying databases or changing records.
- Tests the output path for classic injection flaws where model output is rendered or executed.
- Tests the surrounding application conventionally: authentication, authorisation, API security, rate limiting.
What good looks like
You can't prompt-engineer your way to security. The strongest controls are architectural: the model only gets the data the current user is authorised to see, tools run with least privilege, sensitive actions need human confirmation, and output is treated as untrusted input. A good test tells you which of those controls are missing, with concrete examples.
Particularly relevant in financial services
Client-facing assistants that can see account data, and internal copilots connected to CRM and document stores, combine sensitive data with natural-language access. Test them before your clients or your regulator find the gap.
FAQs
What is LLM penetration testing?
Security testing of applications built on large language models, covering prompt injection, data leakage, unsafe output handling, excessive agency and RAG weaknesses, alongside conventional application security testing.
Is the OWASP Top 10 for LLMs different from the OWASP Top 10?
Yes. It is a separate list focused on risks specific to LLM applications, such as prompt injection, excessive agency and vector and embedding weaknesses.
Sources & further reading
Talk to SMARTSEC
Shipping an AI feature?
Fixed-price testing, scoped and delivered by a senior tester, with live findings on the SMARTSEC client platform.
Estimate an AI security test →