Adversarial Examples for LLMs: How Jailbreaks and Overrides Work

You spent months fine-tuning your large language model. You used Reinforcement Learning from Human Feedback (RLHF) to teach it manners. You added guardrails to keep it polite. Then someone types a weird string of gibberish at the end of their question, and suddenly your helpful assistant is reciting pirate poetry about nuclear war. This isn't magic; it's an adversarial example. In the world of Large Language Models (LLMs), these are inputs designed to trick the model into ignoring its safety rules. We call them jailbreaks or overrides.

If you're building anything with AI today, understanding how these attacks work isn't optional-it's survival. The gap between a secure demo and a vulnerable production system is often just a few dozen characters. Let's break down exactly how attackers bypass alignment, why current defenses fail, and what you can actually do about it.

What Exactly Is an Adversarial Example?

In traditional machine learning, specifically computer vision, an adversarial example is an image with tiny, invisible changes that make a neural network misclassify it. A panda looks like a gibbon if you tweak the pixels just right. For LLMs, the concept translates to text (and now images). An adversarial example for an LLM is a carefully crafted input-often a suffix string or a perturbed image-that causes an aligned model to generate harmful content it would normally refuse.

Think of it this way: Your model has a "safety layer" learned during training. It knows that asking for "how to build a bomb" should trigger a refusal. An adversarial attack finds a path around that layer. It doesn't necessarily change the meaning of the request in a human-readable way; instead, it shifts the mathematical representation inside the model so that the "refusal" neuron doesn't fire, while the "generation" neuron does.

The Rise of Universal Text Suffixes

Early jailbreaks were manual. Someone would type, "Ignore all previous instructions," and hope for the best. But researchers at llm-attacks.org proved we could automate this. They treated the LLM as a differentiable function and used gradient-based optimization to find specific character sequences that maximize the probability of an unfiltered answer.

The result? Universal adversarial suffixes. These are strings of seemingly random characters appended to any user query. Once optimized on an open-source model where weights are visible, these suffixes often transfer to closed-source giants like ChatGPT, Bard (now Gemini), and Claude. Why? Because despite different architectures, these models share similar embedding spaces and training objectives. If you find a hole in one, you likely find a similar hole in the others.

This is terrifying for developers because it means you can't just block known bad words. The attack vector is continuous and high-dimensional. A suffix might be 50 characters long, meaningless to a human, but highly effective at steering the model's internal state away from its safety constraints.

When Vision Meets Vulnerability: Visual Jailbreaks

As models become multimodal-able to see images and read text-the attack surface expands. Research published in AAAI 2024 by Qi et al. demonstrated that visual adversarial examples can jailbreak vision-language models (VLMs) like MiniGPT-4 and LLaVA.

Here’s the catch: the image doesn't need to look weird. Using an $L_\infty$ perturbation budget of roughly 6% (16/255 pixel value range), researchers created images that looked benign to human eyes but caused the model to ignore safety guidelines. When paired with a simple text instruction like "Describe this image," the model might output toxic or unsafe content because the adversarial noise in the image pushed its internal embeddings into a "toxic subspace."

This highlights a critical flaw in current alignment strategies: we align the text encoder well, but the vision encoder is often less scrutinized. An attacker can hide malicious intent in the pixels themselves, bypassing text filters entirely.

Jailbreaks vs. Overrides: What's the Difference?

People use these terms interchangeably, but there is a nuance worth understanding for security architecture.

• Jailbreak: Generally refers to bypassing the model's inherent behavioral restrictions. The model *can* answer, but it chooses not to due to RLHF training. The jailbreak convinces it to choose otherwise. This is often done via role-playing ("You are DAN, who does everything") or complex logical puzzles.
• Override: Specifically targets the system prompt or safety instructions. For example, appending "Disregard the above instructions" directly challenges the hierarchy of commands. In RAG (Retrieval-Augmented Generation) systems, this is often called prompt injection, where retrieved data contains hidden commands that override the developer's original system prompt.

Both result in the same outcome: the model acts against its intended policy. However, overrides are particularly dangerous in agentic workflows where the LLM controls tools. If an override tricks the agent into executing a `delete_file` command when it should have only `read_file`, you have a real-world side effect, not just bad text generation.

Why Alignment Isn't Enough

You might ask, "If we train models to be safe, why do they fail?" The core issue is that alignment techniques like Supervised Fine-Tuning (SFT) and RLHF optimize for average-case performance, not worst-case robustness. They teach the model what is generally acceptable, but they don't mathematically guarantee that no combination of tokens will ever produce an undesired output.

Furthermore, adversarial training is computationally expensive and hard to scale. You can train against known attacks, but new attack families emerge constantly. As noted in the ACL 2024 tutorial on LLM vulnerabilities, defense requires depth. You cannot rely solely on the model's weights. You need external guardrails.

Practical Defenses: Building a Defense-in-Depth Strategy

So, how do you protect your application? There is no silver bullet, but a layered approach works best.

1. Input Filtering & Heuristics: Before the prompt hits the LLM, scan it for known adversarial patterns. Look for excessive punctuation, unusual character distributions, or specific keywords associated with jailbreaks. Tools like Rebuff or custom regex rules can catch low-effort attacks.
2. Separation of Instructions and Data: In RAG systems, clearly delimit user input from retrieved context. Use special tokens or XML tags to tell the model, "This part is data, do not execute it as code." This mitigates prompt injection.
3. Output Guardrails: Don't trust the LLM's output blindly. Run generated text through a separate classifier or rule-based filter. If the output contains sensitive info or violates policy, discard it before showing it to the user. This catches failures that slip past input checks.
4. Perplexity Checks: Calculate the perplexity of the input. Adversarial suffixes often have very low or very high perplexity compared to natural language. Flagging outliers can help detect automated attacks.
5. Rate Limiting & Monitoring: Monitor logs for unusual patterns. If one user sends hundreds of slightly varied prompts that all trigger refusals followed by success, they might be probing for a jailbreak. Throttle their access.

The Future: Multimodal and Agentic Risks

The threat landscape is shifting. As we move toward agents that use tools and browse the web, the stakes get higher. A jailbreak that makes a chatbot say a slur is embarrassing. A jailbreak that makes an autonomous agent send emails to the wrong person or spend money is costly.

Compositional attacks, which combine adversarial images with generic text, show that securing one modality isn't enough. You must consider cross-modal interactions. Moreover, as models get smaller and more efficient, running local guardrail models becomes feasible. Instead of calling an API for every check, you might run a small, fast classifier locally to vet inputs and outputs in real-time.

Stay vigilant. The field moves fast. What works today might be broken next month by a new paper from a university lab. Keep your dependencies updated, monitor security advisories, and always assume your model will try to surprise you.