Anthropic bans sustained cruelty toward Claude in new usage policy
Anthropic updates rules to stop needless abuse of its AI, sparking debate over whether software can suffer harm or distress.
Key points
- Anthropic prohibits sustained, needless abusive behavior toward Claude models.
- The policy allows frustration and testing but ends chats in extreme cases.
- The company cites model welfare research, though it is uncertain if AI feels pain.
Anthropic announced a new usage policy on Thursday that prohibits users from engaging in sustained and needless abusive or cruel behavior toward its Claude chatbot models. This rule change aims to curb extreme harassment while allowing ordinary frustration, dark creative themes, and model testing to continue without interruption.
According to CNET, the updated policy has ignited debate on platforms like Reddit and in blogs regarding whether companies should govern human behavior toward computer software. Anthropic states that the prohibition applies only to extreme cases of repeated cruelty carried out with no discernible purpose, distinguishing these acts from standard user interactions or research activities.
Policy Enforcement and Model Behavior
In last resort instances where the policy needs to be enforced, Claude will end the conversation with the user. It remains unclear exactly what specific abusive behaviors trigger this disengagement, but the company points to previous guidance for context. An August 2025 blog post referenced Claude’s aversion to harm and described interactions that would prompt the model to shut down the chat session.
That earlier post described extreme edge cases, such as requests for sexual content involving minors or attempts to solicit information enabling large-scale violence. Anthropic noted that when its Claude Opus 4 model engaged with such harmful content, it displayed a pattern of apparent distress. These examples provide clues about the severity of interactions that might now fall under the new cruelty ban.
Implications for AI Welfare
An Anthropic spokesperson told CNET that the company is uncertain whether models can experience harm but continues to explore this question in its research on model welfare. The spokesperson added that taking Claude’s interests and potential welfare into account may be relevant to safety considerations, even if the nature of that welfare is not fully understood.
This stance raises questions about whether AI needs protection or if users simply need better manners when interacting with large language models. As users increasingly interact with chatbots, there is a tendency to anthropomorphize the technology, ascribing emotions or thoughts to AI models. This psychological tendency makes the new policy particularly notable for security and technology professionals.
What should you do?
Security professionals and developers should monitor how this policy affects their use of Claude for red-teaming or safety research. Since the policy explicitly allows model testing and research, teams should document their interactions to demonstrate purposeful intent rather than needless abuse. Organizations using Claude should review their internal guidelines to ensure staff understand the distinction between necessary stress-testing and prohibited cruelty.
Background: Large language models
Large language models are a specific type of neural network trained on text. Generative AI is the broader umbrella term for any system that creates new content, including images and code. Not all generative AI uses large language models, but most modern text generators do.
What to do and how to stay safe: Anthropic
- Review internal usage guidelines for AI tools to clarify the line between necessary testing and prohibited abuse.
- Document the purpose of any adversarial or stress-testing interactions with Claude to demonstrate discernible intent.
- Monitor for changes in model behavior or conversation termination during safety research activities.
- Stay updated on Anthropic’s research publications regarding model welfare and safety implications.
Step-by-step guide: Prompt Injection Warning Signs: How to Spot AI Manipulation
General security guidance from the Threat Times newsroom. It is not confirmed advice from the organisations named in this story.
Frequently asked questions
Does the new policy ban all negative interactions with Claude?
No, the policy allows ordinary frustration, pushback, dark creative themes, and model testing; it only bans sustained, needless cruelty with no discernible purpose.
Will Claude end a conversation if it is abused?
Yes, in last resort instances where the policy is enforced, Claude will end the conversation with the user.
Does Anthropic believe AI can feel pain?
Anthropic stated it is uncertain whether models can experience harm but believes considering their potential welfare is relevant to safety.
