The XTN Intelligence Team continues its monthly series on Agentic AI and the future of fraud prevention. This month, we look at how generative AI is reshaping social engineering, erasing the warning signs that used to give scams away.
For years, fraud awareness training rested on a simple premise: bad actors leave clues. Broken grammar, awkward phrasing, a tone that doesn’t quite match the person it’s impersonating: these were the red flags that separated a scam from a legitimate communication. GenAI is quietly dismantling that premise, one fluent sentence at a time.
The clearest evidence of this shift comes from law enforcement itself. The FBI’s Internet Crime Complaint Center (IC3), the bureau’s central hub for collecting and analyzing reports of internet-facilitated crime, issued a public service announcement titled “Criminals Use Generative Artificial Intelligence to Facilitate Financial Fraud”. The advisory documents patterns already observed across real fraud cases reported to the agency, making it one of the most concrete, victim-grounded sources available on how generative AI is being weaponized today. IC3 identifies three critical pillars in this evolution, which we break down below.
The Disappearance of the “Obvious Tell”
The most immediate impact of large language models on social engineering is linguistic. Traditional phishing relied on templates, often produced by non-native speakers, riddled with grammatical inconsistencies that trained employees learned to spot. Large Language Models (LLMs) have eliminated that weakness almost entirely. A Business Email Compromise (BEC) attempt can now be drafted in flawless, contextually appropriate language, in any language, removing one of the most reliable heuristics used in fraud awareness programs for over a decade.
Tonal Mimicry: Impersonation at the Level of Style
Beyond grammar, GenAI enables something more subtle and more dangerous: the replication of how someone writes. By analyzing leaked emails or public social media activity, attackers can prompt an LLM to reproduce the specific tone, vocabulary, and rhythm of a targeted executive. The result is a message that reads as if it were genuinely written by the person it claims to be, making it significantly more convincing to employees in finance and accounting roles, who are often the final line of defense against wire fraud.
Synthetic Identities: The Deepfake Layer
The FBI’s third pillar, and the most visceral, is the rise of cloned audio and video used to bypass “live” verification protocols. Virtual kidnapping scams and wire-transfer instructions delivered via deepfake audio during live business calls have moved from theoretical scenarios to operational threats actively reported by victims. When identity itself can be synthesized in real time, the last line of defense, “I heard their voice, I saw their face,” stops being reliable.
Sustained, Coherent Personas at Scale
Beyond these three pillars, the FBI’s report flags a further technical shift that cuts across all of them: what GenAI does to time. Building trust with a victim, in a romance scam, an investment fraud, or a long-con BEC scheme, traditionally required hours of manual interaction to maintain a consistent, believable persona: UK bank TSB found the typical relationship between a victim and a fraudster lasts around 95 days from first contact to final payment. GenAI automates this nurturing phase, allowing attackers to sustain sophisticated, coherent characters across weeks or months of interaction with minimal human effort. What once limited the scale of social engineering, the attacker’s own time, is no longer a constraint.
Why This Matters Beyond Email
These three pillars, flawless language, tonal impersonation, and synthetic identity, compound rather than operate in isolation, and GenAI’s ability to sustain a persona over weeks or months makes each one harder to catch. A message that passes the grammar test, matches the sender’s known tone, and fits into an ongoing, seemingly consistent relationship is exponentially harder to flag, whether by a trained employee or a legacy rule-based detection system.
For organizations operating in high-value transaction environments (banking, fintech, insurance, iGaming, telcos and more), this signals a clear shift in where defense needs to focus: verifying identity and intent through channels an LLM cannot fabricate, rather than catching the mistakes attackers used to make.
For XTN Cognitive Security®, the signal that matters most is behavior: how someone actually interacts with a device, an app, or a payment flow. Our Cognitive Security Platform® combines behavioral biometrics, device intelligence, and AI-driven risk scoring to verify the real person behind an interaction. XTN lets organizations verify who is really behind an interaction, instead of waiting for an attacker to slip up.
Stay tuned: next month, we go deeper into the deepfake threat: real-world case studies, the numbers behind its explosive growth, and what it means for identity verification.
In the meantime, read the previous contents on Agentic AI:
Continuous Intelligence: A New Paradigm in the Agentic AI Era
The Four Pillars Redefining Fraud in the Agentic AI Era
Offensive AI: When Artificial Intelligence becomes a weapon
Inside Agentic AI: A CTO Perspective on Banking Security in the Agentic Era
