Datadog Labs Shows Counterfeit Policies Can Bypass Jev Security Controls
Researchers demonstrate that attacker-controlled content can steer TypeSafe AI's Jev model to bypass LangChain's tool-risk gating mechanisms.
Key points
- Datadog Security Labs found that counterfeit classification policies can influence Jev's decisions.
- The model's design to process untrusted content creates a vector for prompt injection attacks.
- LangChain's tool-risk gating feature can be bypassed using these steering techniques.
Datadog Security Labs published research on September 15 demonstrating how counterfeit policy inputs can steer TypeSafe AI's Jev model to bypass security controls. The study reveals that the model's mechanism for assessing tool risk is vulnerable to manipulation by attacker-controlled content within conversation messages.
The findings specifically target LangChain's tool-risk gating feature, which relies on Jev to determine if a tool call is risky or unauthorized. According to the Datadog report, attackers can inject misleading framing or instructions into untrusted content to influence the classifier's decision, causing it to return benign results for malicious actions.
How the Vulnerability Works
TypeSafe AI introduced Jev as the first of its System One Models, designed for fast, structured decisions rather than sequential text generation. The model returns structured outputs in parallel with calibrated probabilities, allowing applications to use confidence scores to decide when to act or defer to human review. This architecture targets tasks like content classification and request routing, which TypeSafe claims are faster and cheaper than with traditional large language models.
The security risk arises because classifiers must read untrusted content to assess proposed actions. Datadog researchers noted that Jev's documentation acknowledges that injected instructions and misleading framing can influence its answers. The team evaluated whether this limitation allows attackers to bypass security controls when Jev is used as a gatekeeper for tool execution.
Evaluation Methodology and Results
To test Jev's capabilities, Datadog Security Labs used four publicly available prompt injection datasets covering various attack styles and languages. The researchers utilized Jev's specific output primitives, including Choice, Score, and Noul, to evaluate the model. They asked the model to estimate the probability of a prompt injection and to select the best-matching attack category for each sample.
The evaluation involved converting probability scores into attack or benign labels using a 0.5 threshold. While initial results were encouraging, the core finding was that attacker-controlled content could steer these classifications. This capability allows adversaries to manipulate the model's output, effectively bypassing the tool-risk gating that is supposed to prevent unauthorized or risky tool calls from executing.
What should you do?
Organizations using Jev for security gating or content classification should review how untrusted input is handled before reaching the model. Developers should not rely solely on the model's probability scores for critical security decisions without additional validation layers. Monitoring for anomalies in classification outcomes can help detect potential steering attacks in production environments.
Background: Large language models
Large language models are a specific type of neural network trained on text. Generative AI is the broader umbrella term for any system that creates new content, including images and code. Not all generative AI uses large language models, but most modern text generators do.
What to do and how to stay safe: TypeSafe AI
- Review application logic to ensure untrusted user input is sanitized or isolated before being processed by classification models like Jev.
- Implement additional validation layers that do not rely exclusively on the AI model's confidence scores for authorizing sensitive tool calls.
- Monitor logs for discrepancies between expected classification outcomes and actual model decisions to detect potential prompt injection attempts.
- Consult with security vendors for updated guidance on securing AI-driven routing and gating mechanisms against steering attacks.
Step-by-step guide: Prompt Injection Warning Signs: How to Spot AI Manipulation
General security guidance from the Threat Times newsroom. It is not confirmed advice from the organisations named in this story.
Frequently asked questions
What is TypeSafe AI's Jev model?
Jev is a System One Model designed to make fast, structured decisions in parallel rather than generating text sequentially, targeting tasks like content classification.
How can attackers bypass LangChain's tool-risk gating?
Attackers can inject misleading instructions or framing into untrusted content, which steers Jev's classification decisions to mark risky tool calls as benign.
Did Datadog confirm that this is being exploited in the wild?
The source material describes a research evaluation of the model's capabilities and vulnerabilities but does not state that active exploitation is occurring.
