Phishing emails now target AI assistants with hidden prompt injection instructions

Malwarebytes Labs published research detailing a phishing tactic that targets AI email assistants rather than just human recipients. Attackers are embedding hidden instructions within messages to manipulate how AI tools summarize inboxes, potentially turning ignored spam into urgent, fraudulent to-do items for the user.

The threat exploits the growing reliance on AI to process communications. According to Malwarebytes, if a user ignores a suspicious email but asks their AI assistant to summarize the inbox, the hidden instructions can cause the assistant to present the malicious content as a legitimate and urgent recommendation, bypassing the user’s initial caution.

How the attack works

Recent research from Barracuda describes emails designed to target both human readers and AI systems simultaneously. For humans, these messages often appear as standard phishing attempts, such as password-protected attachments with the password provided in the body. For AI assistants, the same emails contain concealed prompt injection commands intended to influence how the software summarizes or handles the content.

Prompt injection involves planting instructions in material an AI system is supposed to read, hoping it will follow those commands instead of treating them as data. When these instructions arrive through an external source like an email, the technique is known as indirect prompt injection. Attackers aim to make the assistant describe the email as legitimate or urgent, encouraging the user to revisit the message they previously ignored.

Concealment and impact

Barracuda identified several techniques used to hide these malicious instructions from human eyes while keeping them accessible to AI processing. These methods include comments in the email’s underlying HTML code, text made invisible through styling, encoded content, and invisible “zero-width” characters. Whether the AI model actually processes this hidden content depends on how the specific tool extracts and prepares the email data.

The consequences can be severe if the AI assistant has additional permissions. According to Malwarebytes, malicious instructions might try to make the assistant carry out unauthorized actions or disclose sensitive information. In one example, a hidden block instructed a summarizing AI to add a fake priority action to change a vendor’s payment details, which could result in an employee sending money to an attacker’s account.

Who is affected

This issue affects anyone using AI tools to read, summarize, or manage their email inbox. The risk is not limited to specific platforms but depends on how vendors build their AI assistants. Apollo Raines, developer of the AI code review service ShipItClean, explains that the core flaw is the model’s inability to distinguish between text it should read and instructions it should follow.

Raines notes that while stripping hidden content before it reaches the model is necessary, it is an ongoing arms race against new encoding tricks. He emphasizes that users have learned to distrust email but have not learned to distrust their AI assistants, meaning the defense must reside inside the tools and the models themselves.

What should you do?

Users do not need to abandon AI email tools, but they must carefully consider what permissions they grant these systems. The familiar advice for handling phishing still applies, but it must now include verifying AI-generated recommendations. When an assistant reads your inbox on your behalf, you should check the original message before acting on any recommendation involving money or sensitive information.

Background: Prompt injection attacks

Look for sudden shifts in tone, refusal to follow system constraints, or output that repeats hidden instructions. These signs indicate the model is processing injected commands rather than your intended query.

Read the full guide: Prompt Injection Warning Signs: How to Spot AI Manipulation

What to do and how to stay safe: Malwarebytes

• Review the original email content before acting on any AI-generated summary or recommendation, especially those involving financial transactions.
• Restrict the permissions granted to AI assistants, ensuring they cannot execute actions like changing payment details or sending emails without human approval.
• Monitor for unusual AI behavior, such as summaries that seem inconsistent with the visible email content or urgent requests from unknown senders.
• Contact your AI tool vendor to understand how they handle indirect prompt injection and whether they strip hidden HTML or encoded content before processing.

Step-by-step guide: Social Engineering Warning Signs and How to Respond

General security guidance from the Threat Times newsroom. It is not confirmed advice from the organisations named in this story.

Frequently asked questions

What is indirect prompt injection in the context of email phishing?

Indirect prompt injection occurs when attackers plant malicious instructions in external content, like an email, hoping an AI system will follow those instructions instead of treating them as data.

How are attackers hiding these instructions from human readers?

According to Barracuda, attackers use techniques like HTML comments, invisible styling, encoded content, and zero-width characters to obscure instructions from humans while leaving them available for AI processing.

Can an AI assistant execute unauthorized actions based on these emails?

Yes, if the assistant has additional permissions, malicious instructions might try to make it carry out unauthorized actions or disclose sensitive information, depending on its capabilities and safeguards.