-
-

Understanding Model Threats

This resource is designed to provide detailed information on various threat categories, helping you understand and mitigate potential risks in AI and machine learning systems.

Runtime Threats

Like a deserialization threat, runtime threats occur when untrusted data or code is used to reconstruct objects, leading to potential exploitation. The specific difference occurs in how the malicious code is triggered to execute. With a basic deserialization threat, this happens at model load time. A runtime threat is triggered when the model is used for inference or any form of execution. In AI and machine learning systems, this can result in malicious actors injecting harmful code during the deserialization process, exploiting vulnerabilities to gain unauthorized access or manipulate your systems behavior. Understanding deserialization threats is crucial for securing data integrity and preventing unauthorized code execution in your AI models.

Overview

The LiteRT format (formerly known as Tensorflow Lite), saves models into an optimized representation designed for mobile and edge devices. LiteRT uses the flatbuffers serialization to reduce the model size and memory footprints. In this sense, LiteRT models are execution-focused rather than development-focused, containing only the necessary components for inference without the original model building logic.

The LiteRT format enhances deployment flexibility as it's optimized for resource-constrained environments. Its base capabilities support a subset of the original Tensorflow operators along with a set of LiteRT core operators, but other custom operators can be specified.

Models flagged for this threat meet the following criteria:

  1. The model format is detected as LiteRT.
  2. The model contains an unknown/custom operator which will execute code when the model is used for inference.

Since Guardian has detected an operator that is not a standard Tensorflow operator or a LiteRT builtin operator, it hence raised an issue to make users aware of the potential security risk.

Key Points:

  1. Models saved using LiteRT should be deemed as running “packaged code”.
  2. The LiteRT format stores a flattened representation of the model's computation graph and trained parameters in an optimized FlatBuffer structure.
  3. LiteRT files can contain TensorFlow operators. TensorFlow allows for custom operators - which are user defined operators to give additional flexibility to developers. Though custom operators offer powerful functionality, attackers can exploit them to inject malicious code.
  4. Only use/load models from trusted sources.

Background Information

LiteRT Format

The LiteRT format, short for Lite Runtime, was originally known as Tensorflow Lite (TFLite). It saves models into an optimized representation designed for mobile and edge devices. LiteRT uses the flatbuffers serialization to reduce the model size and memory footprints. In this sense, LiteRT models are execution-focused rather than development-focused, containing only the necessary components for inference without the original model building logic.

The LiteRT format enhances deployment flexibility since it's optimized for resource-constrained environments. LiteRT currently supports models authored in PyTorch, JAX, and Keras in addition to the original Tensorflow. Its base capabilities support a subset of the original Tensorflow operators along with a set of LiteRT core operators, but other custom operators can be specified through TensorFlow’s Custom Operator functionality.

Custom Operators

Guardian reports an issue if any of the operators in a LiteRT file is not a standard TensorFlow operator or a builtin LiteRT operator. This is because unknown TensorFlow operators can be used by attackers to inject malicious code which would execute when an unsuspecting user would load the model.

TensorFlow custom operators extend the framework's native functionality by allowing users to implement specialized operations not available in the standard library. These user-defined operators integrate seamlessly with TensorFlow's existing ecosystem, enabling the development of more efficient and tailored machine learning solutions.

Though custom operators provide additional flexibility when implementation of novel algorithms or integration of domain specific operations is required, it also gives attackers an opportunity to export malicious code to a users/victims ML system/pipeline.

Further reading:

  1. TensorFlow lazy execution using graphs
  2. LiteRT format
  3. Custom operator
  4. Standard TF operators
  5. Builtin LiteRT operators
  6. TensorFlow and LiteRT Operator compatibility

Impact

While custom operators provide powerful capabilities, they can potentially be misused by malicious actors for any of the following

  1. Code Injection: Attackers might attempt to inject malicious code into custom operators, potentially compromising an entire ML pipeline.
  2. Resource Exhaustion: Poorly implemented or intentionally malicious operators could consume excessive computational resources, leading to denial-of-service.
  3. Data Exfiltration: Custom operators with access to sensitive data could be designed to leak information covertly.
  4. Model Manipulation: Malicious operators could alter model behavior in subtle ways, potentially introducing backdoors or biases.
  5. Versioning Attacks: Discrepancies between operator versions could be exploited to introduce vulnerabilities or unexpected behaviors.

Note: Malicious code execution using custom TensorFlow operators can be achieved without impacting a model’s performance - the user may never know that the attack has happened or is ongoing.

How The Attack Works

Remediation

To mitigate risks introduced by using custom operators, implement strict code review processes, use trusted sources for custom operators, and regularly audit your environment for any suspicious activities or unexpected behaviors.

If possible, avoid using a LiteRT format model since it contains code that will get executed when the model is loaded.

Protect AI's security scanner detects threats in model files
With Protect AI's Guardian you can scan models for threats before ML developers download them for use, and apply policies based on your risk tolerance.
Learn more