Most AI programmes do not collapse when the model fails. They collapse when someone asks for approval.
For CIOs and CTOs in regulated sectors, the demo is rarely the problem. Teams can build a working model in a sandbox. Pilots show uplift. Board slides signal progress.
The real test comes later, during assurance review, security assessment, risk committee scrutiny or regulatory sign-off.
That is when the most expensive discovery happens.
By this stage:
- Momentum is already public and costly.
- Enterprise licences have been signed.
- Cloud spend has started.
- Internal communications have referenced transformation.
- Delivery teams have invested months of effort.
Then someone asks the question that should have been asked at the start: can this actually run in production?
If the answer is unclear, the organisation is already exposed financially, operationally and politically.
An AI risk assessment brings those questions forward. It tests whether a proposed AI use case can be built, approved, operated and defended at an acceptable level of risk before momentum makes the answer expensive to change.
What is an AI risk assessment?
An AI risk assessment is a structured evaluation of the risks created by a proposed or existing AI system. It identifies what could go wrong, who or what could be affected, how serious and likely those outcomes are, and what controls or changes are needed before the organisation proceeds.
That is broader than model testing. A system can produce accurate outputs and still be unacceptable because its data use is indefensible, its decisions cannot be explained, its security model conflicts with existing controls or nobody owns its behaviour once it reaches production.
The UK Government’s new AI Risk Management Toolkit, published in September 2026, makes the same point. It treats risk identification and assessment, treatment, monitoring and reporting as connected activities across the AI lifecycle, starting as early as use-case identification.
It is also useful to separate AI risk assessment from adjacent activities. An AI readiness assessment asks whether the wider organisation has the strategy, data, systems, governance and delivery capability to adopt AI. AI governance defines how AI is controlled and overseen consistently. AI risk assessment tests the risks of the specific use case or system in front of you.
Why AI risk assessment happens too late in most organisations
Most AI initiatives begin with controlled optimism. A use case is identified. A proof of concept is scoped and a pilot demonstrates improvement.
Confidence builds quickly.
But early pilots run in artificial conditions. Data is curated. Security assumptions are relaxed and integration complexity is deferred. Regulatory questions may remain theoretical. The real environment arrives later, when the model leaves the sandbox and collides with the operational estate.
At that point, structural issues surface fast:
- Data lineage cannot be fully proven.
- Consent or lawful-use assumptions are unclear.
- Outputs cannot be adequately explained or challenged.
- Security architecture conflicts with existing controls.
- Integration requires reworking legacy systems.
- Supplier or hosting dependencies create unacceptable exposure.
- The operating cost looks very different at production scale.
None of these risks are new. They were always present. They were simply not surfaced early enough.
That is why the NCSC’s secure AI system development guidance places threat modelling, security design and consideration of legal, ethical, deployment and oversight requirements at the design stage, not after release.
AI programmes increasingly do not fail at build time. They fail when exposed to real-world approval conditions.
The demo trap. Technical maturity is not approval maturity
The primary failure mode in enterprise AI is not technical incompetence. It is premature momentum.
Call it the demo trap.
A working model proves technical possibility. It does not prove operational viability. Yet once a pilot shows promising output, organisations begin to act as if production deployment is inevitable.
Procurement moves. Hiring follows. Infrastructure is provisioned and internal commitments are made.
Technically possible is a low bar. Operationally viable is not.
A technically mature model:
- Produces useful or accurate outputs.
- Demonstrates measurable improvement.
- Functions reliably in a controlled environment.
An approval-mature system:
- Has defensible data lineage and lawful-use controls.
- Meets appropriate security and resilience standards.
- Can provide the level of explanation or auditability the use case requires.
- Integrates without disproportionate architectural rework.
- Has clear ownership for monitoring, incidents, retraining and governance.
- Has an operating model that remains viable at production scale.
When those two maturities are misaligned, projects become structurally fragile. Internally they appear successful but under assurance or regulatory scrutiny, they stall.
This is not a modelling problem. It is a sequencing problem.
The FCA’s AI Live Testing initiative is a useful example of that distinction in financial services. It is explicitly concerned with developing, assessing and deploying AI systems in live UK markets, including methods for evaluating their impact. The regulatory question is not simply whether an AI system works, but whether it can operate responsibly under real conditions.
This same gap explains why enterprise AI pilots can succeed technically and still fail when organisations attempt to scale them.
What should an AI risk assessment cover before development?
There is no useful enterprise AI risk assessment that asks only whether the model is accurate.
The assessment needs to examine the whole system and the environment into which it will be introduced.
| Risk area | Questions to ask before commitment |
|---|---|
| Business and financial | Is the value credible? What happens to cost at production scale? What is the impact if the system fails? |
| Data and privacy | What data is required? Is its use appropriate and lawful? Is lineage known? Who can access it? |
| Legal and regulatory | Which laws, regulators, sector obligations and contractual requirements apply? |
| Transparency and explainability | Who needs to understand an output or decision, and to what level? |
| Fairness and impact | Could outputs systematically disadvantage users, customers or groups? |
| Accountability | Who owns the system, risk decisions, approvals and ongoing controls? |
| Security | What new attack surfaces, leakage risks or supply-chain dependencies are introduced? |
| Technical robustness | How will reliability, performance, drift and unexpected operating conditions be tested? |
| Architecture and integration | Can the solution operate within the existing technology estate without disproportionate rework? |
| Operations | Who monitors, supports, updates and ultimately retires the system? |
The exact weighting should be proportionate to the use case. A low-impact internal assistant should not receive the same scrutiny as a system influencing lending, healthcare, employment or public-service decisions.
For systems that process personal data, the ICO’s AI and data protection risk toolkit is also a useful reference point. It is specifically designed to help organisations identify and mitigate risks to individuals’ rights and freedoms from AI systems.
The goal is not to eliminate every risk. It is to know which risks can kill the use case, which can be controlled and which the organisation is prepared to accept.
A practical pre-build AI risk assessment process
A useful assessment does not need to begin with a hundred-page governance pack. It needs to force the right decisions before the wrong commitments are made.
1. Define the use case precisely
“Use AI in customer service” is not a risk-assessable use case.
Is the system retrieving information, drafting a response, recommending an outcome, prioritising a case or making the decision itself?
Risk changes as the role of the AI changes.
2. Bring risk owners in before the architecture hardens
Security, legal, compliance, data, engineering and business owners should not first see the system at an approval gate.
The Government toolkit recommends a multidisciplinary approach precisely because AI risk spans technical, legal, ethical, operational and organisational dimensions.
3. Identify the failure conditions
Ask what would make this system unsafe, unlawful, commercially unattractive or impossible to operate.
Do not start by proving that the preferred design is acceptable. Start with the things capable of stopping it.
4. Assess likelihood, impact and risk appetite
Not every identified risk needs to be eliminated. The question is whether the remaining risk is within the organisation’s appetite and whether the proposed treatment is proportionate.
That may lead to four broad choices: avoid the risk, limit it, transfer elements of it or accept it. These are also the four treatment categories used by the Government toolkit.
5. Turn the assessment into a decision
The output should not simply say “risk assessed”.
It should tell the organisation what happens next: proceed, proceed subject to defined controls, narrow or redesign the use case, defer it until specific dependencies are fixed, or stop.
Sometimes deciding not to build is the most valuable result an AI assessment can produce.
Why late-stage AI risk findings are so expensive
When risk is surfaced early, it is interpreted as discipline. Scope is adjusted, roadmaps are refined and investment is redirected.
When risk is surfaced late, after visible progress and internal commitment, it becomes reputational.
By that stage:
- Delivery teams have staked credibility.
- Executives have signalled momentum to the board.
- Procurement has committed budget.
- Vendors are engaged.
- Transformation narratives are already in circulation.
The conversation shifts from “Should we build this?” to “How do we rescue this?”
That usually means governance is retrofitted, architecture is patched and compromises are accepted simply to preserve momentum.
This is why pre-build AI risk assessment is also a capital-allocation discipline. Discovering that a use case needs a different architecture, a narrower scope or no AI at all is not failure. Discovering it before major spend is good governance.
Embed AI risk assessment before momentum builds
Boards expect visible AI progress. Regulators expect caution. Capital is finite.
In that environment, visible activity followed by late-stage collapse is the worst possible outcome. It wastes investment, erodes credibility and slows future adoption.
Mature organisations sequence AI differently. They identify structural blockers before public commitment. They distinguish between technically attractive pilots and operationally defensible systems.
That initial assessment should force clarity on five questions:
- Is the data defensible?
- Is the integration path realistic?
- Which security or regulatory obligations matter?
- What is the full lifecycle cost?
- Who owns the system once it is live?
Catapult’s AI Diagnostic reviews systems, data and workflows to identify practical AI opportunities and the constraints likely to block them before organisations commit to speculative builds or unnecessary change.
Risk assessment should then continue throughout the lifecycle. AI systems, data, suppliers, regulation and user behaviour change over time. The Government toolkit treats ongoing reassessment and monitoring as part of risk management, while effective AI governance provides the controls and evidence needed once systems move into delivery and production.
Enterprise AI maturity is not measured by the number of demos produced. It is measured by the number of initiatives that survive assurance and reach production without avoidable rework or loss of confidence.
The most expensive AI discovery is always the one made after commitment.
If AI is expected to survive approval, not just demonstration, risk assessment must come before momentum.
Frequently asked questions
When should an AI risk assessment be conducted?
The first assessment should happen before major development, procurement or architecture commitments. Risks should then be reassessed during development and operation when models, data, usage or external requirements change.
Is AI risk assessment the same as AI governance?
No. Risk assessment evaluates the risks associated with a particular system or use case. AI governance defines the wider policies, roles, controls, evidence and oversight used to manage AI consistently across the organisation.
Who should be involved?
For enterprise AI, risk assessment should normally be multidisciplinary. Relevant participants may include the business owner, data and engineering teams, security, legal, compliance, risk, operational specialists and representative users.
