UK FCA chief demands mandatory reporting of significant AI failures and near-misses

Jessica Rusu, Chief Data Officer at the UK Financial Conduct Authority, stated on Monday that financial firms should be required to report significant artificial intelligence failures and near-misses. Speaking at the Bloomberg UK Policy Series, Rusu argued that existing rules requiring major incident reporting must extend to cover these specific AI-related events.

Rusu emphasized that firms need to invest more in cyber and operational resilience to handle these risks. She noted that the AI Minister has recognized near-miss reporting for agentic AI as a national security priority. According to Rusu, this issue is also a regulatory priority for the FCA.

Regulatory Context and Framework Development

Rusu explained that assurance methods must evolve because systems safe at deployment may behave differently under stress or at scale. She stated that agentic systems require continuous, conditional assurance as underlying models change. The FCA intends to work with industry, fellow regulators, and frontier labs to create a trusted framework for agentic finance.

This framework will include clear approaches to agent identity, delegated authority, accountability, and execution. Rusu highlighted regulator frustration over futile attempts to slow down agentic and generative AI rollouts. Reports of agent misbehavior and a lack of meaningful consequences continue to emerge with alarming frequency.

Defining Failures and Near-Misses

Questions remained about the precise definition of significant AI failures and near-misses after Rusu's speech. Queries sent to the FCA to clarify their terminology were not answered by publication time. Pieter Arntz, a malware intelligence researcher at Malwarebytes, offered his interpretation of what those terms likely mean in practice.

Arntz said a significant AI failure involves serious consequences, not just incorrect answers. In financial services, this could mean an AI system authorizing substantial fraudulent payments or exposing sensitive customer information. It could also involve denying customers access to essential services or triggering transactions that threaten market integrity.

Arntz added that the test should be the scale and severity of the impact rather than technical drama. He defined an AI near-miss as an event that could realistically have caused serious harm but was caught before that harm occurred. This distinction helps clarify the scope of potential reporting requirements.

What should you do?

Security professionals should monitor FCA announcements for formal rule changes regarding AI incident reporting. Organizations deploying agentic AI should review their current incident detection capabilities to ensure they can identify near-misses. Teams should assess whether their assurance processes account for model changes and scale-related behavioral shifts.

Background: Generative AI

Generative AI examples include text completion, image synthesis and code generation. These systems predict the next likely token based on vast training datasets. Understanding the underlying mechanics helps you identify risks like data leakage and hallucination in your infrastructure.

Read the full guide: Generative AI Examples: Mechanisms, Risks and Control

What to do and how to stay safe: FCA

• Review existing incident reporting procedures to ensure they can capture AI-specific failures and near-misses as defined by regulators.
• Assess agentic AI systems for continuous assurance capabilities that account for behavioral changes under stress or at scale.
• Monitor communications from the FCA and other regulators for updates on the trusted framework for agentic finance.
• Evaluate current cyber and operational resilience investments to address risks associated with autonomous AI agent deployments.

Step-by-step guide: Prompt Injection Warning Signs: How to Spot AI Manipulation

General security guidance from the Threat Times newsroom. It is not confirmed advice from the organisations named in this story.

Frequently asked questions

What did the FCA Chief Data Officer propose regarding AI reporting?

Jessica Rusu proposed that UK financial firms be required to report significant AI failures and near-misses, extending existing major incident rules.

How does Malwarebytes define an AI near-miss?

Pieter Arntz at Malwarebytes defines an AI near-miss as an event that could realistically have caused serious harm but was caught before harm occurred.

What components will the FCA's trusted framework for agentic finance include?

The framework will include clear approaches to agent identity, delegated authority, accountability, and execution, developed with industry and regulators.