From risk to defense: securing the LLM stack with open source.

For twenty years, application security rested on a comfortable assumption: there is a line between code (the instructions a system executes) and data (the untrusted content it merely processes). Firewalls, input validation, and access control all police that line. Large language models erase it. To an LLM, the instructions you wrote and the text it reads from a web page, an email, or a PDF are the same thing: language, to be interpreted and acted upon.

That single property is why AI security is not just "AppSec with a chatbot on top." It is a genuinely new discipline. And, encouragingly, the open-source community has already built a mature toolkit for it. This guide organises that toolkit around a simple lifecycle you can actually run: map the risk, attack it, defend it, and govern it.

01, Map the risk

You cannot secure what you cannot name. The OWASP LLM Top 10, maintained by the OWASP GenAI Security Project, is the industry's shared vocabulary for the risks specific to generative AI. It gives an engineer and a CISO the same words for the same problem, which is half the battle in any security programme.

Sitting at the top of that list, for good reason, is prompt injection. It is the SQL injection of the AI era. In its direct form, a user simply types a jailbreak into the chat. The more dangerous indirect form hides the payload inside content the model will later read: a comment in a support document, invisible text on a web page, a line in an email. When that model powers a retrieval pipeline or an agent with real tools, an attacker's instruction hidden in a document becomes an instruction your system obeys.

The model can't tell your instructions apart from instructions hidden in the content it reads. Attackers exploit exactly that gap.

If you want to go deeper than the Top 10, the community-curated awesome-llm-security list (by corca-ai) is the best single index of papers, tools, and attacks in the field. Start with OWASP for the language; use the list for the frontier.

02, Attack it first

Because an LLM's output is non-deterministic, you cannot prove it safe by reading the code. You have to attack it: throw thousands of adversarial prompts at the running system and see what leaks, breaks, or misbehaves. Three open-source projects lead here, and they complement rather than compete:

A pragmatic starting point: run garak for coverage, promptfoo for the cases you want to enforce forever, and reach for PyRIT when a high-value system justifies a deliberate, multi-turn campaign.

03, Defend at runtime

Finding the holes is only useful if you close them. Guardrails are the controls that wrap the model in production: they scan every input, filter every output, and, critically for agents, gate the actions the model is allowed to take. Again, the open-source options layer neatly:

• ▹NeMo Guardrails (NVIDIA): programmable rails that constrain the conversation and the actions: keep the assistant on-topic, and block unsafe tool calls before they run.
• ▹Llama Guard (Meta, part of PurpleLlama): a safety classifier model that labels prompts and responses against a policy, so you can catch unsafe content in either direction.
• ▹LLM Guard (Protect AI): a stack of input/output scanners: PII redaction, toxicity, secrets, injection detection, and output validation.
• ▹Rebuff (Protect AI): a dedicated prompt-injection detector combining heuristics, an LLM-based check, and a canary-token tripwire that reveals system-prompt leakage.

No single one of these is a silver bullet, and that is the point. A layered design uses several: a classifier at the door, scanners on input and output, and behavioural rails around what the agent can actually do.

04, Evaluate & govern

A one-off penetration test is a snapshot, not assurance. Models drift, prompts change, and a well-meaning tweak can quietly degrade safety. The final stage turns security into something continuous and measurable. Giskard provides open-source testing for ML and LLM applications, automatically scanning for hallucination, harmful content, injection, and bias, and producing a report you can act on. promptfoo earns a second mention here: run as a versioned test suite in CI, it makes a safety regression show up as a failing check on the very commit that caused it.

For a leadership audience, this stage is the one that matters most. It is where AI risk stops being an anecdote and becomes a trend line you can put in front of a board or a regulator: evidence that control is not just designed, but demonstrated over time.

Where to start

You do not need to adopt everything at once. Three moves get most teams to a defensible position:

1. Inventory and map. List every LLM-powered feature and rate each against the OWASP Top 10.
2. Red-team your flagship. Point garak or promptfoo at your highest-risk feature, and the findings write your business case.
3. Add one guardrail, then automate. Wrap that feature in LLM Guard, then wire the evals into CI so safety can't regress silently.

The threats are new, but the discipline is familiar: know your risks, test against them, control what you can, and prove it continuously. The tools to do all four already exist, they are open source, and they are ready today.