Cybersecurity experts are warning that hackers and hostile groups are exploiting weaknesses in large language models (LLMs) to spread disinformation, steal sensitive data and carry out sophisticated fraud, according to a new report by NewsGuard, a rating system for news and information websites.
One major risk is the use of AI as a tool for psychological and informational manipulation. NewsGuard exposed a Russian propaganda network called “Pravda” that created over 150 fake news sites. Though these sites receive little human traffic, their real power lies in their influence on AI models and search engines.
By “feeding” models with false information, Pravda has succeeded in getting Gemini, ChatGPT and Copilot to cite its disinformation, thereby amplifying false narratives on sensitive issues like the war in Ukraine. This tactic, called “LLM grooming,” raises the likelihood that artificial intelligence will absorb disinformation during its learning processes.
Similar trends are seen worldwide. Reports from Israel and the United States show that Iranian and pro-Palestinian actors also use AI technologies to spread propaganda and deepfakes. In China, by contrast, domestic AI models are tightly monitored by the government, but globally, AI systems are becoming potential battlegrounds for spreading false messages.
Worryingly, this phenomenon is not limited to state actors or hostile hackers—marketing companies have also begun testing ways to influence AI query results to promote their products.
Another significant technique is prompt injection attacks, in which malicious or hidden commands are inserted into the model to bypass its original guidelines, trigger unwanted actions or extract sensitive data. Amir Jerbi, CTO of Aqua Security, explained that these attacks exploit the fact that language models interpret any input as an instruction.
The more advanced method of jailbreaking allows users to trick models into ignoring safety restrictions. One well-known example is “DAN” (Do Anything Now), an alternate persona users created for ChatGPT to generate harmful or dangerous content.
While such attacks were once marginal, they now represent a real risk, especially when chatbots are connected to organizational systems and confidential information. In one case, Air Canada’s chatbot provided incorrect refund policy details; when a customer sued, the court held the airline responsible—setting a precedent for organizational accountability for AI systems.
Jerbi noted that until recently, these attacks were mostly irrelevant because AI systems only delivered public information. But in the past year, attacks have begun affecting AI systems with access to sensitive data, and even those given autonomy to perform actions like purchasing products or charging credit cards—fertile ground for fraud and failures. He added that malicious code has even been hidden inside AI-generated images, like those of a cartoon panda.
To counter these threats, a whole industry of cybersecurity companies has emerged, focusing on AI protection. Firms such as Guardio, Nustic, Aqua Security, Zenity, Check Point and others are developing solutions to monitor, analyze and block fraud attempts in real time. These tools examine model inputs and outputs to detect malicious prompts, attempts at data theft or unauthorized operations. Many companies are also forming “red teams” tasked with attacking models proactively to identify and fix vulnerabilities.
The need for such measures is growing. A recent Check Point study exposed malware attempting prompt injection against AI-based security systems, while a “zero-click” exploit called EchoLeaks in Microsoft 365 Copilot allowed attackers to extract sensitive organizational data without user interaction.
These cases demonstrate that the threat is no longer theoretical but already real. It is a technological arms race in which AI developers, security companies and users must remain vigilant, even as state and hostile actors exploit the technology for increasingly sophisticated attacks.