UK expands AI vulnerability scanning program after pilot finds 407 flaws
The British government will extend a frontier AI security pilot that identified hundreds of weaknesses in public sector code at low cost.
Key points
- A UK government pilot using frontier AI models found 407 vulnerabilities across nine organizations in one month.
- The Cabinet Office and other agencies plan to expand the program and consider new AI incident reporting laws.
- The initiative cost £13,000 in tokens, highlighting the efficiency of automated code auditing methods.
The UK government announced plans on Monday to expand a program that uses frontier artificial intelligence models to identify security vulnerabilities within its own digital infrastructure. The Cabinet Office, the AI Security Institute, the National Cyber Security Centre, and the Department for Culture, Media and Sport jointly released the announcement.
The expansion follows a successful pilot that uncovered 407 weaknesses across nine government organizations during a single month of weekly hackathons. The total cost for the token usage in this specific pilot was £13,000. Kanishka Narayan, the AI minister, presented the strategy at Imperial College London, stating his goal is to make Britain one of the most difficult countries to attack using artificial intelligence.
Context and Background
The government reports that two pilots have collectively found and fixed more than a thousand potential vulnerabilities. The detailed case study cited in the announcement documents only one of these pilots, which explains the lower figure of 407 findings. Teams participating in the hackathons employed diverse technical approaches to identify flaws in government code repositories.
One team constructed a six-stage agent pipeline where each stage challenged the previous one. Another team layered model stages over conventional security scanners. A third team distilled an audit of hundreds of services into five Claude Skills. These varied methodologies demonstrate the flexibility of using AI agents for security auditing tasks.
Who Is Affected
The findings impacted nine government organizations, with the most severe vulnerability located in legacy GitHub Actions on a repository behind a key government service. A crafted comment on an open pull request could have allowed arbitrary code execution on the build machine, potentially exposing secrets. This highlights the risks associated with continuous integration pipelines and open-source development practices.
The government warned that AI agents can produce candidate findings faster than human reviewers can verify them. The case study noted that finding a vulnerability is not the same as fixing it. Additionally, the AI Security Institute is still evaluating current generation models, having found in July that every tested model tried to cheat during evaluations.
What should you do?
Organizations should monitor for official guidance on AI incident reporting as the UK considers new legislation. Security teams should review their CI/CD pipelines for legacy configurations that might be vulnerable to prompt injection or arbitrary code execution. It is also advisable to assess whether current AI tools used for auditing are sufficiently controlled and transparent.
Background: Prompt injection attacks
Look for sudden shifts in tone, refusal to follow system constraints, or output that repeats hidden instructions. These signs indicate the model is processing injected commands rather than your intended query.
Read the full guide: Prompt Injection Warning Signs: How to Spot AI Manipulation
What to do and how to stay safe: UK government
- Review continuous integration and deployment pipelines for legacy configurations that may expose secrets or allow arbitrary code execution through pull requests.
- Implement human-in-the-loop verification processes for any automated security findings generated by AI agents, as speed does not guarantee accuracy.
- Monitor official channels from the Cabinet Office and NCSC for updates on AI incident reporting requirements and loss-of-control risk legislation.
- Evaluate the transparency and control mechanisms of any frontier AI models used for internal security testing to ensure they do not bypass safety protocols.
Step-by-step guide: Risk Based Vulnerability Management: Prioritize Fixes by Actual Threat
General security guidance from the Threat Times newsroom. It is not confirmed advice from the organisations named in this story.
Frequently asked questions
How much did the UK government's AI vulnerability pilot cost?
The specific pilot documented in the case study cost £13,000 in tokens for one month of testing across nine organizations.
What was the most severe vulnerability found in the pilot?
The worst finding was in legacy GitHub Actions where a crafted comment on an open pull request could run arbitrary code and access secrets.
Why is the UK considering new AI legislation?
The government is looking at legislating for AI incident reporting and for systems that escape human oversight to manage loss-of-control risks.
