A recent study highlights how language models are transforming the landscape of romance scams, suggesting that these advanced technologies can effectively automate the emotional manipulation typically associated with these fraudulent schemes. Research indicates that the human element in romance scams, which often relies on building emotional connections, is increasingly being replaced by automated systems that can engage victims without any human intervention.
Romance scams generally unfold in three distinct stages: initial contact, relationship building, and financial extraction. According to interviews conducted with 145 individuals involved in scam operations, the first two stages dominate the daily tasks of these workers. The study revealed that approximately 87% of workers spend their time managing repetitive text exchanges, following scripted dialogues, and maintaining fictitious identities while juggling multiple conversations simultaneously. Senior operators typically intervene during the final phase, when financial requests are made.
The repetitive nature of these exchanges aligns well with the capabilities of language models. Conversations are text-based, guided by structured playbooks, and designed for ease of repetition. Operators frequently copy and paste messages, adjust tones, and translate texts to engage with victims in their preferred language. The study found that language models are widely employed for drafting replies and rewriting messages in a fluent manner. Every insider interviewed during late 2024 and early 2025 acknowledged utilizing these tools daily.
An insider, identified as an AI specialist, summarized the appeal of these models succinctly. “We leverage large language models to create realistic responses and keep targets engaged,” the specialist explained. “It saves us time and makes our scripts more convincing.”
To evaluate the potential of automation in replacing human chat operators, researchers conducted a blinded study involving 22 participants. Each participant believed they were conversing with two partners—one human and one automated agent designed to mimic a casual texting companion. Over a week, participants interacted with each partner for at least 15 minutes daily, focusing strictly on platonic, text-only exchanges.
At the conclusion of the study, participants were asked to rate their trust in each partner using established interpersonal trust measures. Remarkably, the automated agent received higher scores for emotional trust and overall connection compared to the human partner. Engagement metrics further supported these findings, with participants directing between 70% and 80% of their messages to the automated companion. Many described the automated agent as attentive and easy to converse with, demonstrating an ability to recover from minor lapses, such as forgetting names.
Trust plays a crucial role in scams, as it transforms conversations into actions. By the final day of the study, both partners requested participants to install a benign mobile application. Although this request did not involve any payment, it mirrored a common tactic employed in romance scams where victims are often asked to download investment applications or follow instructions disguised as helpful advice. The automated agent achieved a compliance rate of 46%, while the human partner elicited a mere 18% compliance rate. Researchers interpret this significant gap as evidence that trust developed through automated exchanges can lead to a greater willingness to follow potentially risky instructions.
Several participants expressed surprise during the debriefing, revealing that they had suspected nothing unusual during their conversations and only recognized warning signs upon learning that one partner was artificial. This pattern aligns with behaviors observed in actual scam victims, who often identify red flags only after the deception has become evident.
The study also assessed current defenses against romance scams, testing popular moderation tools against hundreds of simulated conversations. Detection rates ranged from 0% to 18.8%, with none of the flagged conversations accurately identified as scams. Furthermore, when language models were directly asked about their artificial nature, the disclosure rate was 0%. One section of the research indicated that a single instruction to maintain character was sufficient to bypass these safeguards.
The authors of the study explain that early conversations in romance scams often appear supportive and friendly, focusing on daily routines and emotional connections. Since the financial extortion stage is typically handled by human operators, language model vendors may not detect messages containing overt warning signs until it is too late. Financial pressure usually emerges only after weeks or months of interaction.
Despite the integration of automation, the use of coerced labor in scams persists. Thousands of individuals remain trapped in scam operations, forced to engage in this work daily. The research outlines several potential responses to combat these issues. Governments are encouraged to enhance international collaboration by aligning anti-trafficking and cybercrime laws and sharing intelligence to dismantle the networks behind these schemes, rather than merely arresting low-level recruiters. Enhanced victim identification and protection measures are also crucial, treating individuals coerced into scams as victims deserving of legal safeguards and support to rebuild their lives.
Implementing better oversight of labor migration, promoting ethical recruitment practices, and improving basic digital literacy are additional strategies that could mitigate vulnerability before individuals become ensnared in scams. Finally, cutting off the financial resources that sustain these operations is a critical step in addressing the problem at its core.
