This article was originally published by Neal for the RSA conference website.
Introduction
Large language models (LLMs) are transforming how organizations process information, streamline operations, and interact with customers. But the integration of LLMs into enterprise workflows also expands a company’s attack surface. Given their complexity and unique vulnerabilities, securing LLMs requires an approach that goes beyond traditional IT security frameworks.
That’s because, unlike traditional enterprise systems, LLMs operate on vast amounts of data and are non-deterministic, making them unpredictable, opaque, and challenging to secure. A key risk is the potential for data leakage and unauthorized actions, especially since LLMs often interact with sensitive or proprietary information during training or inference, making them prime targets for exfiltration by adversaries. Similarly, LLMs rely on retrieval-augmented generation (RAG) to ground prompts and responses in real-time business data without retraining, introducing additional vectors for data exposure and manipulation.
Another growing concern is the threat of prompt injection attacks, where maliciously crafted inputs manipulate an LLM’s behavior to generate harmful outputs or expose unintended information. These attacks are particularly dangerous in applications where external users interact directly with LLMs. This problem will only expand as we enter the era of agentic LLMs, which are connected to downstream systems to perform actions, increasing the potential for exploitation, and unintended consequences.
Moreover, the proliferation of LLMs across different departments in an enterprise often leads to "shadow AI" deployments—models operating outside the purview of security teams, making it difficult to track what is being done, where, and with which LLMs. This lack of oversight creates vulnerabilities that attackers can exploit. Security is not only a concern at runtime; LLMs can also be compromised during training and development. Adversaries can inject malicious data into the training phase, subtly altering a model’s behavior in ways that are difficult to detect, further compounding the security risks.
Adding to the complexity is the interconnected nature of ecosystems in which LLMs operate. These systems rarely function in isolation; instead, they integrate with APIs, data pipelines, and third-party applications, any of which can serve as entry points for attackers and create an extended blast radius.
Taking a Lifecycle Approach to Securing LLMs
To address LLM security from training through deployment and beyond, organizations should adopt a lifecycle approach. Here are some best practices to consider.
Securing the Training Phase
This phase is particularly vulnerable due to its reliance on extensive datasets. Organizations should implement stringent data governance practices to ensure sensitive or personal information is excluded from training sets. This includes scanning datasets for basic security issues such as data poisoning, personally identifiable information (PII), and proper data classification. Proactively addressing these risks reduces the likelihood of inadvertently exposing confidential information through the model's responses.
Ensuring the integrity of the training pipeline is equally critical. Role-based access controls (RBAC) and continuous monitoring of data ingestion processes can help safeguard against unauthorized modifications or the introduction of malicious data. Advanced techniques, such as data validation and outlier detection, can further protect against poisoning attacks by identifying and excluding anomalous inputs.
Hardening Deployments
Once operational, LLMs become direct targets for attackers, making robust deployment strategies essential. One key measure is the implementation of fine-grained access controls—limiting access based on roles, time, and other contextual factors ensures that only authorized personnel can interact with the model. Additionally, organizations can reduce the risk of prompt injection attacks by applying guardrails to user inputs, such as filtering out potentially harmful commands or keywords. Beyond these controls, security teams should conduct regular red teaming exercises to simulate adversarial attacks, identify vulnerabilities, and refine defensive strategies before real threats emerge.
Runtime Security
Real-time monitoring is vital for maintaining the security of deployed LLMs. Observability tools allow organizations to track LLM app activity, identify anomalies, and respond quickly to potential threats. Monitoring outputs for unusual or harmful responses is especially important, as it provides early detection of misuse or adversarial manipulation.
Lightweight monitoring solutions can deliver comprehensive visibility without introducing performance overhead. These tools enable security teams to oversee the full LLM traces, from the model itself to its interactions with APIs and third-party applications. Integrating these insights into security information and event management (SIEM) platforms further strengthens an organization’s ability to detect and mitigate risks.
Balancing Security with Innovation
The tension between maintaining robust security and fostering innovation remains a central challenge. Overly restrictive measures can stifle the utility of LLMs, while insufficient security can expose the enterprise to breaches and reputational harm.
Several strategies can help strike the right balance. Adopting an agent approach to LLM security minimizes disruptions during deployment while preserving performance and operational agility—without requiring extensive engineering involvement. Transparency is also critical; LLMs and their associated security tools should provide clear explanations of their outputs and flagged events. This fosters trust, streamlines security oversight, and enables faster decision-making.
As LLM adoption accelerates, so will security risks and regulatory requirements, such as stricter compliance mandates governing deployment and monitoring strategies. Simultaneously, advancements in adversarial training and explainability tools are likely to enhance LLM resilience against sophisticated attacks. For CISOs, the challenge lies in tackling present-day risks while preparing for future threats. By integrating LLM security into their broader cybersecurity program, organizations can ensure AI/ML systems remain assets, not liabilities.
