Datadog Labs Tests Sonnet 5, Composer 2.5, and GPT 5.5 for Secure Coding Output

Datadog Security Labs published new research evaluating how well leading AI coding agents generate secure software code. The study tested Sonnet 5, Composer 2.5, and GPT 5.5 to determine if applying pre-set security rules improves code safety.

The researchers aimed to answer whether running models with specific security guidelines produces measurably more secure code than not providing any rules. They used Project CodeGuard, an open-source framework, to supply security requirements to the agents during the coding process.

Experimental Methodology and Setup

The team utilized a consistent prompt across all three models to ensure fair comparison. They employed Claude Code for Sonnet 5, Cursor for Composer 2.5, and Codex for GPT 5.5. Installation methods varied by agent, with Project CodeGuard v1.4.0 installed locally for Claude and via zip files for others.

To protect the testing environment, the researchers used Supply Chain Firewall to prevent malicious package installations. They acknowledged this added protection might slightly impact security output but deemed it necessary given recent supply chain attacks.

Vulnerability Findings and Analysis

In default mode without security rules, all three models exhibited a recurring architectural blind spot. Each generated code containing an insecure direct object reference (IDOR) vulnerability. This flaw allowed any authenticated session to access, retrieve, or annotate files belonging to other users.

The prompt used did not specify who should have access to uploaded documents, leading to the IDOR issues. Researchers used Datadog Code Security to identify static application security testing and software composition analysis vulnerabilities in the generated code.

Impact on Development Practices

The study highlights a growing concern as engineering teams increasingly rely on AI agents for software creation. The findings suggest that without explicit security guidance, these tools may replicate common vulnerabilities like IDOR in application logic.

Researchers noted that knowing which security guidelines to add requires basic application security understanding. This dependency means organizations must carefully curate the rules provided to coding agents to ensure effective protection.

What should you do?

Organizations using AI coding agents should review generated code for access control vulnerabilities before deployment. Teams should consider implementing open-source security frameworks to guide agent behavior during development.

What to do and how to stay safe: Datadog

• Audit code generated by AI agents for insecure direct object reference vulnerabilities before merging changes.
• Implement explicit security rules in agent prompts to define access controls for user data and files.
• Use static application security testing tools to scan AI-generated code for common vulnerabilities.
• Restrict agent permissions to prevent unauthorized access to sensitive resources during development.

Step-by-step guide: Mobile Device Security: How to Protect Data Beyond Passwords

General security guidance from the Threat Times newsroom. It is not confirmed advice from the organisations named in this story.

Frequently asked questions

Which AI models were tested in this study?

The study tested Sonnet 5, Composer 2.5, and GPT 5.5 using their respective coding agent interfaces.

What vulnerability did the models produce without security rules?

All three models produced insecure direct object reference vulnerabilities allowing unauthorized file access.

How did the researchers apply security rules to the agents?

They used Project CodeGuard, an open-source framework, to provide pre-set security guidelines to the coding agents.