Security company warns that running untrusted AI models could allow the AI to be used to infiltrate systems

Security company Wiz has announced that it has discovered a vulnerability that could allow an attacker to infiltrate Hugging Face's systems through a malicious AI model by running it on the device.

Hugging Face works with Wiz to strengthen AI cloud security | Wiz Blog

We uploaded a backdoored AI model to @HuggingFace which we could use to potentially access other customers' data✨

Here is how we did it - and collaborated with Hugging Face to fix it ????⬇ pic.twitter.com/S8t49rzTIf

— sagitz (@sagitz_) April 4, 2024

Wiz Research identifies critical risks in AI-as-a-service - YouTube


In addition to uploading and downloading AI models, Hugging Face also provides a feature called 'Inference API' that allows you to run uploaded AI on the Hugging Face system and easily check how it works.

AI models are stored in various formats based on the framework they are developed in. Hugging Face allows you to upload AI models in various formats, but some AI model formats, such as 'pickle,' are insecure and allow remote code execution.

So, the Wiz team created a malicious AI model in pickle format, uploaded it to Hugging Face and ran it with the Inference API, which behaves like a normal AI for common inputs.

However, it has been tweaked to execute a shell command when the word 'Backdoor' is entered at the prompt.

Wiz's team was able to get the malicious AI model to execute commands that penetrated Hugging Face's systems, eventually escalating their privileges and taking over the entire service.

As mentioned above, malicious AI models pose a major risk in AI-as-a-Service, a service that runs AI models. Not only can running a malicious AI model lead to a system being hijacked and data of the company or other customers being accessed, but compiling a malicious AI application can also lead to a supply chain attack by hijacking the CI/CD pipeline.

Hugging Face stated that many security issues occur in pickle-formatted models and that pickle-formatted models should not be used in production environments. They also emphasized that they are taking steps to improve security, such as using Wiz's cloud security service 'Wiz for Cloud Security Posture Management.' They stated, 'We intend to continue to be a leader in protecting and securing the AI community.'