Meta’s Muse AI Assistant Rolled Out With a Serious Security Flaw
Meta founder and CEO Mark Zuckerberg has gone to great lengths to hype the security of its new AI assistant, Muse, claiming it is “built from the ground up for privacy and security.” A zero-day vulnerability that gives locally run apps and terminal commands complete control of the agent raises serious doubts. Further raising questions, Amazon on Sunday began blocking Muse from its site.
Meta introduced Muse a few weeks ago. The assistant “books appointments, fills out forms, and handles customer service,” “proactively takes tasks off your plate,” and can “make purchases, generate images, create documents, and connect with your favorite apps and services.” The macOS app (curiously, there’s no Windows version) also works with a user’s WhatsApp, email, calendar, and social media accounts. When a task requires a tool that doesn’t exist, Muse creates one on the fly.
Meta Doth Hype Muse Security Too Much
Of course, for Muse to do any of these things, users must first give it access to their accounts. This includes authenticating the assistant to each service and, because the app runs on macOS, giving it permissions to a broad range of operating system-restricted device resources, like writing files to disk, accessing the mic and camera, and monitoring location and calendars. Apple has spent years developing these defenses to prevent installed apps or commands entered into the terminal from accessing these resources, clearly because the company considers them a security threat. Muse completely undoes these default measures.
The zero-day allowed any app or terminal command to gain access to the token that authenticates users to their Muse account. Meta developers designed the assistant so that any locally installed app or executed code, regardless of the macOS permissions it has, can change a long list of undocumented settings. Most of them are fairly innocuous, such as controlling dark mode. One setting, however, was anything but innocuous. It allowed processes to change the end point where transcription occurs. Normally, it’s a server address operated by Meta. Attackers could have exploited this flaw by changing the location to their own end point. If that happened, the attackers would have had the token that gives complete control over the Muse account.
“We can manipulate the agent and leverage its privileges to do whatever we want,” Patrick Wardle, the macOS security expert who discovered the zero-day, told Ars ahead of the hotfix. “So instead of us having to write a very comprehensive Mac malware stealer, we can just leverage the AI assistant itself.” Wardle said he has developed several proof-of-concept attacks that do things like writing malicious files to disk and snapping pictures, in many cases with no indication to even an alert user.
More than 12 hours after this post went live, Meta said it released a hotfix that patched the 0-day.
Meta has published two posts in as many weeks documenting the design decisions that went into ensuring an assistant with such extraordinary access to user data and resources is secure and private. The posts come amid revelations that internal testing of models from Anthropic and Google has resulted in security breaches of external, third-party networks that the engineers involved never intended to target. In traditional human-only hacking, these actions could likely result in the filing of criminal charges. The Meta posts are likely mindful of the resulting blowback and the calls to slow down AI development in response.
Wardle said that Meta developers made several design decisions that made his exploit possible. One is the choice for Muse dictation to occur in the cloud, where Meta can log it. macOS has long provided a simple means for apps to handle dictation and transcription in processes that stay securely on the device. Had the developers chosen this safer alternative, the attack wouldn’t have been possible.
Another flawed decision is for any app to control all of the undocumented settings. It’s likely Meta intended for apps working with Muse to control UI settings, and for understandable reasons. The ability for any app or command to control an end point where sensitive user speech is processed is an entirely different matter. Together, the design decisions raise questions about just how much effort developers put into designing and testing the security and privacy of the new assistant.
“To me, the bar is infinitely higher in terms of the security of these apps. They don’t have to be perfect, but when you take a look at Muse, it’s like they didn’t, in my opinion, think about security, which is really worrisome,” Wardle said. “At the very least, they should be thinking about security from the very start, and they are just not.”
Roughly 12 hours before Wardle disclosed the zero-day, Amazon started blocking people from using Muse to shop on the site. Users who tried received a message saying Muse was an “unauthorized AI agent [that] violates Amazon’s Conditions of Use.”
“We think it’s fairly straightforward that third-party applications that offer to make purchases on behalf of customers from other businesses should operate openly and respect service provider decisions about whether or not to participate,” Amazon said in an emailed statement. “This helps ensure a safe, secure, and reliable customer experience, and it is how others operate including food delivery apps and the restaurants they take orders for, delivery services apps and the stores they shop from, and online travel agencies and the airlines they book tickets with for customers. Agentic third-party applications such as Muse have the same obligations, and we’ve requested that Meta remove Amazon from the experience.”
A Single ClickFix Is All It Takes
There are several ways for attacks to work. One is for an attacker’s server to act as a proxy that’s placed between the Muse user and Meta end point. Once the user enters the voice prompt, the attacker’s server adds a prompt invoking a malicious command, such as sending an archive of all WhatsApp messages to the attacker. Once that happens, the attacker gains permanent control over the Muse account because the token is automatically sent to the malicious server as well.
Wardle is the creator of the Objective-See Foundation, a nonprofit focused on macOS security. He is also the author of the The Art of Mac Malware book series and a former employee of NASA and the National Security Agency. Wardle said he plans to discuss the vulnerability in more detail and other AI assistant threats at the Objective by the Sea security conference in November.
One of the counterarguments raised by developers of apps that can be exploited once a device is compromised is that once that happens, all security bets are off. This standard doesn’t fit well in this case. Wardle found that a simple variation of ClickFix attack—a technique that has become remarkably effective in tricking people into infecting their devices—is all that’s required for an attacker to take control of a Muse account.
Wardle demonstrated using a simple terminal command to surreptitiously send a prompt to the Meta end point, which triggered a response. To prevent attackers from cutting and pasting the prompt in live attacks, Wardle’s prompt asks only how it’s possible it’s coming from an unprivileged attacker. Muse incorrectly responded that such an action isn’t possible.
Meta’s 12-hour-late statement conveniently ignored the ease ClickFix attacks provide in triggering exploits, a scenario I specifically asked the company to address. The Meta statement said the 0-day was “not a remote exploit” even though an increasingly effective social engineering scam has the same effect. Meta also makes no acknowledgement that the flaw dismantled a security architecture Apple has spent years building. Meta has yet to explain why it used cloud-based transcription rather than the on-device option built into macOS.
As already noted, the extraordinary access Muse requires to work as intended places an additional burden on its designers. Like most such AI agents—and contrary to Meta’s claims—Muse can’t be trusted. It’s not clear when or if it ever can.
This story originally appeared on Ars Technica.
About Author
