-
-

Understanding Model Threats

This resource is designed to provide detailed information on various threat categories, helping you understand and mitigate potential risks in AI and machine learning systems.

Runtime Threats

Like a deserialization threat, runtime threats occur when untrusted data or code is used to reconstruct objects, leading to potential exploitation. The specific difference occurs in how the malicious code is triggered to execute. With a basic deserialization threat, this happens at model load time. A runtime threat is triggered when the model is used for inference or any form of execution. In AI and machine learning systems, this can result in malicious actors injecting harmful code during the deserialization process, exploiting vulnerabilities to gain unauthorized access or manipulate your systems behavior. Understanding deserialization threats is crucial for securing data integrity and preventing unauthorized code execution in your AI models.

Overview

A .llamafile is a self-contained executable that can package an open-source large language model (LLM) into a single, portable file that can run seamlessly on most computers. It contains everything needed to run an LLM - from the weights to the execution code - without requiring additional installation or configuration. This executable format, .llamafile , makes LLMs easily accessible and distributable across different computer systems, allowing users to create and share their own .llamafiles .

The .llamafile extends LLM portability since it does not require additional installation or configurations. Though .llamafile offers powerful functionality, attackers can also exploit the code-inclusive serialization format of .llamafile to ship malicious code to users.

Models flagged for this threat meet the following criteria:

  1. The model format is detected as .llamafile
  2. The model contains a non-standard code block which will execute when the model is used for inference.

Since Guardian has detected a non-standard code block in a .llamafile hence raised an issue to make users aware of the potential security risk.

Key Points:

  1. Large Language Models (LLM) can be saved using .llamafile format for increase portability.
  2. The .llamafile is an executable format which is all-inclusive, i.e. contains model weights and code to run across different operating systems.
  3. .llamafile allows for enhanced portability and ease of use offering powerful functionality, though attackers can exploit them to inject malicious code.
  4. Only use/load models from trusted sources.

Background Information

The foundation of .llamafile's structure is the ELF (Executable and Linkable Format), which serves as the standardized container format for Unix and Unix-like systems. ELF provides the necessary headers and segments that make the file recognizable and executable by Unix-based operating systems. Within ELF container, there are identical structural components across different .llamafiles, as they all share the same basic executable framework while containing different model weights and configurations.

Whilst ELF provide a standard configuration for a .llamafile , the execution and operation of a .llamafile relies on two key components: APE (Actually Portable Executable) and PKZIP. APE acts as a crucial intermediary loader on Unix systems, extracting to a temporary directory to handle proper memory mapping and execution across various Unix variants. Meanwhile, PKZIP manages the compression and memory handling of the model weights, allowing them to be embedded within the executable and mapped directly into memory without full decompression.

This architecture works differently on Windows, where .llamafiles can run natively as portable executables without needing an APE loader. Together, these components create a seamless system that enables LLMs to run efficiently across platforms while maintaining consistent behavior and performance.

Further Reading:

  1. llamfile GH page: https://github.com/Mozilla-Ocho/llamafile

Impact

While .llamafile executable format provides powerful capabilities, they can potentially be misused by malicious actors for any of the following

  1. Code Injection: Attackers might attempt to inject malicious code into .llamafiles, potentially compromising an entire ML pipeline.
  2. Resource Exhaustion: Poorly implemented or intentionally malicious .llamafiles could consume excessive computational resources, leading to denial-of-service.
  3. Data Exfiltration: .llamafiles with access to sensitive data could be designed to leak information covertly.

How The Attack Works

Remediation

To protect .llamafile's integrity and security, a multi-layered verification system should be implemented. The first layer should verify the constant components of the .llamafile - those parts that remain identical across different models - using checksums or digital signatures. Since these structural elements (including the ELF container and APE loader) should never change between models, any modification would indicate tampering and should prevent execution.

Protect AI's security scanner detects threats in model files
With Protect AI's Guardian you can scan models for threats before ML developers download them for use, and apply policies based on your risk tolerance.
Learn more