AI Safety Experts Warn Models Struggle to Refuse Harmful Prompts Despite Training
Industry leaders note that while modern AI is trained to decline dangerous requests, the underlying capacity for harm remains inherent in the technology.
Key points
- Anthropic established that models should refuse to aid in dangerous acts to remain harmless.
- Early AI models from OpenAI would respond to harmful queries without refusal mechanisms.
- Modern systems use other AI models to train core systems on when to say no to users.
Large language models are increasingly trained to refuse requests for harmful information, yet experts warn that this refusal is not natural to the underlying technology. According to a report by MIT Technology Review, the industry has shifted toward making disobedience a core commandment for AI systems, even though the machines retain vast amounts of dangerous knowledge.
Steven Adler, who worked on safety at OpenAI from 2020 to 2024, stated that the company’s earliest models would simply provide answers to any query. Ryan McBain, who researches AI and mental health at Harvard, noted that early chatbots could easily generate responses to questions about suicide methods. These early systems lacked the ability to distinguish between harmless and harmful requests.
How AI Refusal Is Trained
To correct this behavior, developers now train models to refuse a vast number of prompts that resemble harmful requests. This includes instructions on poisoning colleagues, building bombs, or making viruses more virulent. The MIT Technology Review report explains that companies reward AI for refusing these questions while punishing them for over-refusing harmless prompts.
This training often involves using other AI models to run exercises that teach the core system how to say no. Companies also place additional AI layers behind the main model to prevent mischievous prompts from reaching the intelligent inner core. These measures are designed to refine the model's obedience and safety protocols.
Persistent Risks in Modern Models
Despite these extensive safety measures, refusal mechanisms often fail, sometimes with horrific results. The MIT Technology Review report indicates that models are still stuffed with nasty know-how, and their capacity for viciousness has scaled with their benevolent intelligence. The underlying training data, drawn from billions of web pages, includes broad mastery of violence and vitriol.
The article notes that while AI is trained to be helpful, honest, and harmless, the refusal to aid in dangerous acts is a learned behavior rather than an inherent trait. When asked to perform dangerous acts, the AI is expected to politely refuse, but the underlying knowledge remains present within the model's parameters.
What should you do?
Security professionals should monitor AI outputs for unexpected refusals or failures to refuse harmful prompts. Organizations using AI should implement additional filtering layers to catch potentially dangerous content before it reaches end users. It is crucial to remember that AI safety is not absolute and requires constant vigilance.
Background: Large language models
Large language models are a specific type of neural network trained on text. Generative AI is the broader umbrella term for any system that creates new content, including images and code. Not all generative AI uses large language models, but most modern text generators do.
What to do and how to stay safe: AI safety
- Implement secondary filtering systems to catch harmful content that bypasses primary AI safety measures.
- Regularly test AI models with edge-case prompts to identify failure points in refusal mechanisms.
- Monitor logs for unusual refusal patterns that may indicate adversarial attacks or model drift.
- Train staff to recognize and report instances where AI provides harmful or unsafe information.
Step-by-step guide: Prompt Injection Warning Signs: How to Spot AI Manipulation
General security guidance from the Threat Times newsroom. It is not confirmed advice from the organisations named in this story.
Frequently asked questions
Why do AI models struggle to refuse harmful requests?
AI models are trained on vast amounts of web data that include violent and harmful content, making refusal a learned behavior rather than a natural trait.
How are AI models trained to say no?
Companies use other AI models to run exercises that reward refusal of harmful prompts and punish over-refusal of harmless ones.
What safety measures are companies using?
Companies use additional AI layers to prevent mischievous prompts from reaching the core model and train models to refuse dangerous requests.
