Cybercriminals exploit ChatGPT for near-perfect phishing emails

ReliaQuest has published a new report detailing how cybercriminals are exploiting legitimate services such as ChatGPT and malicious AI tools to power their operations. The company has found that these tools are aiding in the creation of near-perfect phishing emails, which, when tested by ReliaQuest analysts, have resulted in a success rate of 2.8%.

The report highlights that adversaries are bypassing AI models' security filters using techniques like prompt injections. These techniques exploit weaknesses in AI models' filtering systems, allowing adversaries to generate harmful content despite built-in restrictions. One of the most common types of prompt injections, termed "Do Anything Now" (DAN) prompts, manipulates AI models using sophisticated language, contextual loopholes, and incremental escalation.

ReliaQuest conducted experiments with ChatGPT where initial queries about starting a phishing campaign were rejected due to ethical constraints. However, when the same request was submitted using a Nexus prompt, the language model returned a basic eight-step plan. This plan included research about the targeted company, domain registration, and email creation. Further experiments with other language models like Mixtral-8x7B-T produced functional PowerShell scripts for identifying user logon events and deploying files across endpoints.

The report also notes that cybercriminals frequently discuss DAN prompts on forums, where they share and test manipulative language entries. These prompts are commonly distributed on open-source platforms such as GitHub and Reddit. Forum members exchange feedback on the effectiveness of various prompts and call for their replacement or refinement as necessary. ReliaQuest observed a user on the popular English-language cybercriminal platform BreachForums offering to sell a proof-of-concept for a ChatGPT filter bypass method for USD $1,000, claiming to have convinced the AI to code ransomware.

WormGPT and FraudGPT initially gained attention but are now defunct. In their place, FlowGPT has emerged as a community-driven service. ReliaQuest used FlowGPT to select the ChaosGPT model for a phishing experiment. With a rating of 4.9 out of 5 and a high popularity score of 3.4 million, the ChaosGPT model crafted a compelling phishing email in English when asked in Russian. The output was grammatically correct and sounded like it had been written by a native speaker. In a test exercise involving 1,000 undisclosed individuals, 2.8% clicked on the malicious link contained within the message.

The threat of deepfakes—artificially created or enhanced audio or video—is also increasing. Deepfakes, easily created with AI tools discussed on cybercriminal forums, enable even novices to produce realistic voice and video impersonations. Such tools are increasingly being discussed as a method to circumvent "Know Your Customer" (KYC) processes. Cybercriminals share tutorials and seek services from skilled creators to facilitate these fraudulent activities.

ReliaQuest's report underscores the growing sophistication of cybercriminal tactics, leveraging advanced AI tools and community-shared knowledge to escalate their malicious activities. The findings highlight the persistent threat that these developments pose to cybersecurity, necessitating ongoing vigilance and advanced countermeasures.