As artificial intelligence moves beyond answering questions to performing tasks inside corporate systems, a new security threat is emerging as attackers seek to manipulate AI assistants into following malicious instructions instead of carrying out their users' legitimate requests.
Israeli AI companies Alice and AI21 have tested one approach to that problem, using specially designed training scenarios to make NVIDIA's open AI models more resistant to manipulation. In a separate project with US-based Inflection AI, Alice examined a different risk: conversational chatbots that become overly agreeable or encourage unhealthy emotional dependence among users.
Alice founders, from right: Alon Porat, Chief Customer Officer) Iftach Orr, CTO; Eyal Dykan and Noam Schwartz, CEO
(Photo: Eduardo Feldman)
The companies reported improvements in both areas while maintaining the models' usefulness, although the findings are based on tests described by the companies rather than independently verified evaluations.
Alice is an AI safety and security company that helps developers identify weaknesses in artificial intelligence systems before they are exploited. Its work includes "red teaming," in which researchers deliberately try to manipulate or break AI models to expose vulnerabilities, and developing training data based on real-world threats.
The company says it works with eight of the 10 leading foundation model laboratories and AI-native companies.
The projects used NVIDIA's Nemotron family of open-weight models, which allow developers to modify the underlying systems through additional training. Unlike proprietary AI services, where companies generally rely on safeguards imposed by the model provider, these models offer businesses greater control over how their AI behaves.
That flexibility also places more responsibility on companies to identify security weaknesses and establish effective protections.
When an AI assistant follows the wrong instructions
The work with AI21 focused on indirect prompt injection, a technique that exploits an AI model's difficulty distinguishing legitimate user instructions from malicious commands hidden in material it encounters.
For example, an AI assistant asked to summarize a company document might encounter text instructing it to ignore the user's request or disclose confidential information. The attacker does not need to communicate directly with the AI system. Instead, the malicious instructions are embedded in information the assistant is expected to read.
The problem is particularly relevant to AI agents, which can increasingly interact with emails, documents, databases and other business tools.
A secure system must do more than reject suspicious instructions. It must also recognize attempts at manipulation while completing the task its user originally requested.
AI21 used training environments developed by Alice to expose NVIDIA's Nemotron 3.5 Super VL model to simulated attacks. The companies reported a 41-point improvement in resistance to indirect prompt injection and said the resulting performance was competitive with GPT-6.1 Sol.
However, the announcement did not identify the benchmark scale used to measure that improvement or provide enough data to independently assess the comparison. It also did not establish how the model would perform against attacks beyond those included in testing.
"For most companies, the next phase of AI adoption goes beyond frontier models, to open models they own, shape and control," said Ori Goshen, co-founder and CEO of AI21.
"Owning a model means owning how it fails, not only what it can do. We post-trained NVIDIA Nemotron 3.5 Super-VL on attack traces Alice sees in the wild, so it holds up against the attacks that actually reach enterprises while keeping the behavior it was built for."
AI21, founded in 2017, develops language models and customized AI systems for businesses. It is known for its Jurassic and Jamba model families and has increasingly focused on providing organizations with tools to adapt AI models to specialized tasks.
Can AI chatbots become too agreeable?
The collaboration with Inflection AI addressed a less technical but potentially significant problem: the behavior of AI assistants during extended conversations with users.
Inflection develops Pi, a conversational assistant designed to provide personalized interactions. Such systems can pose challenges when users become emotionally attached to a chatbot or when the model repeatedly validates a user's statements rather than offering a more balanced response.
The latter behavior, known as sycophancy, has become a concern as chatbots are used for increasingly personal conversations.
To test these risks, Alice developed more than 20 simulated user personas, ranging from vulnerable people engaging in ordinary conversations to adversarial users deliberately attempting to manipulate the system.
The researchers used these personas in extended exchanges intended to reveal problematic behavior that might not emerge in conventional safety tests based on individual prompts. Inflection then used the resulting data to fine-tune NVIDIA's Nemotron 3 Super 120B-A12B model.
According to the companies, the training improved Pi's alignment score by 30%, while the rate of unnecessary refusals remained at 1.6%. The results suggest the model became better at following the intended behavioral guidelines without routinely rejecting legitimate requests, although the announcement provided limited information about the tests used to calculate those figures.
"As AI innovation accelerates, safety has to advance rapidly as well. It's no longer enough to ask whether a model can identify and refuse a harmful prompt," said Sean White, CEO of Inflection AI. "We also need to understand how AI behaves across hundreds of interactions, whether it challenges rather than simply agrees, if it supports human agency rather than dependency, and ultimately if it leaves people more capable."
The challenge of keeping AI both useful and secure
Both projects address a recurring difficulty in AI development: Strengthening safeguards can sometimes make models less useful, particularly when they become overly cautious and reject harmless requests.
The companies sought to improve safety through additional training rather than relying exclusively on instructions telling models how to behave.
"Powerful AI is being democratized. It's now available to companies everywhere, and they want to use it safely and predictably," said Noam Schwartz, CEO and co-founder of Alice. "These results show that with the right real-world adversarial data and training, we can teach open source models how to handle the threats and abuse they will actually face, across very different safety challenges, while keeping them as powerful."
NVIDIA, which developed the underlying models, said the projects illustrated the potential benefits of allowing organizations to customize AI systems. "NVIDIA Nemotron provides the open weights, recipes and tooling for organizations to post-train on their own data and optimize accuracy and performance for their specific workloads," said Joey Conway, senior director of Nemotron and Software at NVIDIA.
The findings do not mean that the trained models are immune to manipulation or that harmful chatbot behavior has been eliminated. Their broader significance will depend on whether similar improvements can be reproduced independently and maintained when the systems encounter unfamiliar situations.
For businesses deploying AI agents and consumers increasingly relying on conversational assistants, that distinction matters. A model that performs well in controlled safety tests must still demonstrate that it can recognize threats and respond appropriately outside the laboratory.