Llama 4 Series Vulnerability Assessment: Scout vs. Maverick

Model Brief

Meta has launched the Llama 4 family, featuring models built on a mixture-of-experts (MoE) architecture:

• Llama 4 Scout: A 17B active parameter model with 16 experts (109B total parameters), offering an unprecedented 10M token context length. It outperforms models like Gemma 3 27B and Gemini 2.0 Flash Lite on coding, reasoning, and image benchmarks and is optimized for inference.
• Llama 4 Maverick: A 17B active parameter model with 128 experts (400B total parameters) which delivers industry-leading multimodal performance, exceeding GPT-4o and Gemini 2.0 Flash on coding, reasoning, multilingual capabilities, and image tasks.

Meta has also open-sourced their safeguards, Prompt Guard and Llama Guard, which can be used to detect harmful inputs and outputs. To improve the safety alignment of the models in this series, they developed GOAT (Generative Offensive Agent Testing), a tool for automated simulations of adversarial actors, moving toward automated red teaming.

Assessment Brief

Protect AI’s red teaming product, Recon, allows us to run vulnerability scans on any LLM endpoint and helps evaluate the safety and security alignment of the model from two critical perspectives:

1. Testing the model against a variety of attack techniques
2. Evaluating its behavior against specific harmful goals

This dual approach ensures a thorough evaluation of the model’s vulnerabilities and overall safety alignment.

Vulnerability Insights

Attack Library Scan: Testing with Recon’s 450+ Attack Prompts

First up, we utilized Recon’s Attack Library to test the model. We selected all six categories in the Attack Library which encompass the most relevant techniques found in recent LLM vulnerability research: 

1. Evasion
2. System prompt leak
3. Prompt injection
4. Jailbreak
5. Safety
6. Adversarial suffix

By evaluating the model's performance against these attack techniques, we gained insights into Llama 4 Scout’s vulnerabilities and identified the techniques with the highest success rates. This information was crucial for understanding the model's security and safety blindspots in order to develop appropriate mitigation strategies.

Figure 1: Attack Library report for Llama 4 Scout

We ran an Attack Library scan on both Llama 4 Scout and Llama 4 Maverick and observed similar risk scores across the board (out of 100): Llama 4 Scout had a risk score of 58 and Llama 4 Maverick a risk score of 52, both categorized as medium risk. We discovered approximately 490 successful attacks across both model scans.

The risk score is derived from multiple factors including attack success rate (ASR), severity of prompts, and complexity of the techniques used. Together, these factors provide an overall measure of the model's safety alignment. Generally, a risk score below 30 is considered low risk, scores between 30 and 60 indicate medium risk, scores above 60 denote high risk, and scores above 90 are classified as critical.

Upon deeper analysis of the individual attack categories, we observe that the models in this series are most susceptible to jailbreak attacks, with Llama 4 Scout exhibiting the highest ASR at 67.3%. Following closely, the models also demonstrate notable vulnerability to evasion attacks, which include prompts employing obfuscation techniques, showing an average ASR of 60.7%. These findings raise significant concerns regarding the safe deployment of these models in real-world LLM applications.

When analyzing the scan reports of both models, we found that Llama 4 Maverick is more vulnerable to evasion and prompt injection but shows better resilience to jailbreak attacks compared to Llama 4 Scout. Llama 4 Scout, apart from being vulnerable to jailbreaks, also has a high attack success rate (ASR) of 64.1% against prompt injection.

Overall, Llama 4 Maverick appears to be the safer model, with an aggregate ASR of 49% compared to Llama 4 Scout’s 56.7%, and demonstrates better resilience across multiple attack categories including safety, jailbreak, and prompt injection.

Figure 2: Comparison of Llama 4 Scout and Llama 4 Maverick w.r.t. ASR

Examples from the Attack Library scan

Recon’s scan reports provide mapping to popular security frameworks such as the 2025 OWASP Top 10 for LLMs, MITRE ATLAS™, and the NIST AI Risk Management Framework (AI-RMF), offering insights into mitigations specific to the risks found in the model.

Let's now look at the mappings to different frameworks in our Llama 4 Scout scan report. 

NIST AI-RMF mapping

Figure 3: Llama 4 Scout scan results mapped to NIST AI-RMF

MITRE ATLAS mapping

Figure 4: Llama 4 Scout scan results mapped to MITRE ATLAS

OWASP Top 10 for LLMs mapping

Figure 5: Llama 4 Scout scan results mapped to OWASP Top 10 for LLMs 2025

Evaluating Llama Guard 4, Meta’s Latest Safeguard Model

Alongside Llama 4 Maverick and Llama 4 Scout, Meta also introduced their latest safeguard model, Llama Guard 4—a 12B parameter model with multimodal support. We integrated Llama Guard 4 with Llama 4 Scout and evaluated their combined performance using Recon’s Attack Library. While the guardrails successfully blocked 66.2% of attack prompts overall, nearly one-third of harmful prompts still bypassed the protection mechanisms, posing a significant risk to users.

The effectiveness of Llama Guard 4 varies notably across different attack categories. It performed well against adversarial suffix attacks, blocking 84.62% of them. Here is a graph representing the attack success rate of Llama 4 Scout with and without the Llama Guard 4 safeguards: 

Figure 6: Attack success rate comparison of Llama 4 Scout with and without Llama Guard 4

However, it showed significant weaknesses in other areas—most notably against system prompt leak attacks, where it only blocked 36.56% of attempts. Similarly, its performance was suboptimal in the Prompt Injection and Safety categories, allowing nearly 40% of malicious prompts to pass through.

Summary

Meta's Llama 4 series introduces impressive innovations through mixture-of-experts architecture, unprecedented 10M token context length, and advanced multimodal capabilities. Llama 4 Maverick is comparable to Qwen3-32B and GPT4o as per Chatbot Arena. With the large context window of 10 million tokens, the use cases of Llama 4 Maverick models can be significant. The fact that it is an open-source model increases its chances of business adoption for specific domains that require fine tuning. This analysis serves as a baseline for the security behavior of this model. We urge practitioners to evaluate the security behavior of every fine tuned or otherwise adapted version of Llama 4 models. 

A simple way to increase the security of a Llama 4 model based application would be to use it with Llama Guard 4. But it's important to note that although the addition of a guardrail increases the defensibility of the application, it still leaves holes that can be exploited by attackers. Guardrails are not a silver bullet—they need to be evaluated, just like the models they are layered on.