Payroll Fraud via LLM-Generated Emails

Sublime’s Attack Spotlight series is designed to keep you informed of the email threat landscape by showing you real, in-the-wild attack samples, describing adversary tactics and techniques, and explaining how they’re detected.

‍EMAIL PROVIDER: Google Workspace

ATTACK TYPE: Business Email Compromise (BEC)

The Attack

Attempts to impersonate legitimate payroll messages from employees to redirect funds. The messages are personalized and all aimed at an individual in the payroll department:

• The subject and message body are likely Large Language Model (LLM)-generated* in a style attempting to mimic the tone and structure of a legitimate corporate communication, but overly verbose and unnatural
• The use of freemail providers (e.g. Gmail) for added perceived authenticity from the impersonated employee’s personal account
• Only first names of the impersonated employees are used for an additional layer of perceived authenticity
• Messages target individuals in the payroll or finance department

*as indicated by GPTZero, an AI text detection tool, with a 100% probability that the content was AI-generated.

While payroll change attacks are not new, what makes this noteworthy is the likely use of Generative AI (GenAI) to vary the subject, sender name, and message body at scale. We've seen a significant uptick in the diversity of the language used in these attacks. In the past, messages were often templated besides the greeting and signature.

Also of importance, the messages are free of basic grammatical errors, which demonstrates how GenAI lowers the barrier of entry for non-English speaking adversaries. In contrast, we’ve seen this same type of attack in the past with language that doesn’t flow as well.

This attack vector remains effective as it mimics real-world behavior of employees changing financial institutions, which requires an update to direct deposit information. The attackers hope to leverage this familiarity and empathy from the recipient to respond to the email and conduct fraud.

Detection signals

Sublime detected and prevented this attack using the following top signals:

• Engaging financial language: The message uses engaging financial language related to payroll and direct deposit information.
• Urgent Language: The message attempts to engage the user with a sense of urgency related to financial information.
• Freemail provider: The use of a freemail provider (e.g., Gmail) is unusual for payroll requests. These types of requests are typically handled from the employee's corporate email address, over the phone, or in person.
• Unknown sender: No one in the targeted organization has received an email from the sender.

A note on LLM detectors

A number of research papers have shown that LLM detectors are generally unreliable and easy to bypass. While approaches like Perplexity Analysis (a calculation to determine how “surprised” an LLM is by the word choices/sequences within the text) can hone in on the likelihood of a piece of text being generated by AI, they are often trivial to bypass using methods like paraphrasing attacks (e.g., changing word order, or introducing human-like errors).

At Sublime, we take a defense-in-depth approach and use thousands of signals and machine learning-backed enrichment functions like Natural Language Understanding (NLU) to understand tone, intent, and context, surface anomalous behavior, and prevent attacks before they cause damage.

Sublime detects and prevents malware/ransomware delivery and other email based threats. Deploy a free instance today.