The XTN Intelligence Team continues its monthly series on Agentic AI and the future of fraud prevention. In our previous article, we explored how generative AI changed social engineering by removing many of the warning signs that once exposed fraudulent communications. Attackers no longer need to rely on poorly written messages or obvious manipulation techniques to gain trust.
The next evolution is even more challenging: AI can now replicate the identity of trusted individuals through realistic voices, images and videos.
In 2024, an employee at a Hong Kong multinational joined what appeared to be a routine video conference with the company’s CFO and several colleagues. Every face and every voice on that call had been generated using AI. Convinced they were speaking to trusted executives, the employee authorized transfers totaling $25 million before anyone realized the meeting itself had been entirely fake.
This case signals a fundamental shift in how social engineering works.
Why the Last Year Changed the Equation
Deepfakes, AI-generated images, voices and videos capable of depicting events that never happened, are not new but it definitely changed who can create them.
The rapid evolution of generative AI platforms such as Midjourney, Gemini and Sora has transformed a capability that once required specialized expertise into something accessible to virtually anyone with a subscription. Creating convincing AI-powered impersonations is no longer reserved for highly sophisticated threat actors.
The pace of change is accelerating. The trajectory is not gradual. Deloitte’s Center for Financial Services, drawing on FBI crime-category data rather than speculation, projects that generative AI-enabled fraud losses in the US, which stood at $12.3 billion in 2023, are expected to rise to $40 billion by 2027, representing a 32% compound annual growth rate. The World Economic Forum reported more than $200 million in confirmed deepfake-related losses during the first quarter of 2025 alone, more than the total recorded over the previous four years combined.
Unlike traditional social engineering, where attackers had to carefully prepare each campaign, generative AI dramatically reduces the effort required to impersonate executives, colleagues or trusted contacts. Entire conversations, complete with realistic voices and facial expressions, can now be fabricated in real time.
When Identity Becomes Synthetic, Context Gains a New Role
The Hong Kong incident shows how deepfakes are changing the dynamics of digital trust. A realistic face, a familiar voice and a live conversation can now be generated to support a fraudulent request, creating a scenario where the communication itself becomes part of the attack surface.
Fraud prevention has always relied on the ability to evaluate multiple signals around a transaction: user behavior, device intelligence, session information and transactional context. The emergence of deepfakes adds a new challenge to this landscape by making digital interactions easier to manipulate.
As AI-powered impersonation become increasingly sophisticated, financial institutions will need to strengthen their ability to connect different layers of information and evaluate the consistency of an action within its wider context. The question becomes whether a request aligns with the expected behavior, circumstances and intent behind the transaction.
Deepfakes and the Agentic AI Era: Redefining Digital Trust
Deepfakes mark a turning point for fraud prevention: as digital identities become easier to replicate in the Agentic AI era, the ability to understand the context and intent behind every action will become essential to preserving trust in financial interactions.
Stay tuned: next month, we will dive deeper into another emerging challenge in Agentic AI fraud risk.
In the meantime, read the previous contents on Agentic AI:
Social Engineering: How GenAI Rewrote the Rules of Deception
Continuous Intelligence: A New Paradigm in the Agentic AI Era
The Four Pillars Redefining Fraud in the Agentic AI Era
Offensive AI: When Artificial Intelligence becomes a weapon
Inside Agentic AI: A CTO Perspective on Banking Security in the Agentic Era
