Across the public sector, interest in AI-powered knowledge search has accelerated rapidly. Policy teams want faster access to guidance. Digital leaders want to reduce friction across departments. Governance and assurance functions want fewer errors caused by outdated, misunderstood, or inconsistently applied documentation.
On paper, the promise is compelling – ask a question, get an answer and move on.
In practice, many of these initiatives quietly stall, or worse, introduce new risks that are harder to detect than the problems they were meant to solve.
When failures occur, attention often turns to the AI model itself. But model capability is only part of the problem.
AI knowledge initiatives can also fail when systems cannot reliably identify and retrieve authoritative internal knowledge, preserve its organisational context, or distinguish current guidance from outdated or conflicting material.
In public service organisations, the ability to produce a fluent answer matters less than the ability to establish whether that answer is supported by the right sources.
For organisations operating under statutory, regulatory, and accountability constraints, that gap matters far more than raw model capability.
1. The semantic trap. Why general purpose AI is context-blind
General-purpose AI systems are trained to recognise patterns across large quantities of information. They can also be connected to private organisational knowledge, but their usefulness depends on the sources and context made available to them. In regulated environments, retrieving relevant text is not always enough to establish which interpretation is authoritative.
In public service, meaning is not universal. It is institutional.
A simple acronym can illustrate the risk. Ask a general purpose AI system how to handle an ‘NDA’ and it will confidently explain Non-Disclosure Agreements. In a different organisational context, the same acronym may refer to a formal regulatory submission with entirely different implications, approvals, and consequences.
The AI is not malfunctioning. It is doing exactly what it was trained to do; selecting the most statistically likely interpretation. But for a DDaT leader or SRO, this exposes a critical weakness. An AI system that cannot weight internal meaning more heavily than global patterns will produce answers that sound correct, while being operationally wrong.
This is not a rare edge case. It is a structural limitation of general purpose AI in environments where:
- Language is overloaded
- Definitions are domain-specific
- Authority is hierarchical, not democratic
Left unaddressed, this semantic gap becomes a governance issue, not a usability one.
2. The ‘Just Add Copilot’ false shortcut
The appeal of simply enabling tools like Copilot is understandable. They feel low-risk, familiar, and fast to deploy. They integrate neatly into existing productivity platforms and require little visible change management.
But this approach rests on a flawed assumption namely, that access = understanding.
Tools such as Microsoft 365 Copilot can retrieve information from connected organisational sources while respecting existing user permissions. That is useful, but permission to access a document does not necessarily establish whether it represents current, approved or authoritative guidance.
An organisation may still need additional controls to distinguish:
- Draft guidance from approved policy
- Historical documents from current authority
- Advisory notes from statutory requirements
Those distinctions depend on how information is classified, maintained, retrieved and governed. Simply enabling access to existing content does not resolve them.
Microsoft’s Copilot Search privacy and protection guidance explains how access permissions apply to retrieved organisational information.
In regulated environments, those distinctions are not nuance, they are the difference between compliance and failure.
Treating a general assistant as a shortcut to authoritative internal guidance is therefore a category error.
An assistant should not be treated as the final authority on organisational policy unless the wider system has been designed, governed and validated for that specific purpose.
3. The legacy data trap. When “Search” becomes a risk multiplier
One of the most underappreciated causes of AI knowledge failure in the public sector is the state of the underlying data estate.
Critical organisational knowledge is often locked in:
- Fragmented SharePoint sites
- Scanned or image-heavy PDFs
- PowerPoint decks with implicit assumptions
- Documents that reference each other without formal linkage
Placing AI on top of this environment does not create clarity. It creates speed without certainty.
This is the Legacy Data Trap. The danger is not that AI cannot find information. The danger is that it can rapidly assemble plausible answers from incomplete, outdated, or conflicting fragments, giving users false confidence that they are operating within approved boundaries.
AI retrieval becomes less reliable when source documents lack the context needed to establish:
- Which information is current
- Which sources take precedence
- Who owns or approves the information
- Which users are permitted to access it
Public-sector organisations do not necessarily need to clean or restructure their entire information estate before using AI. They do need sufficient governance over the information relevant to the chosen use case, including source authority, version status and permissions.
Without those controls, an AI system may combine contradictory or obsolete material into an answer that appears coherent but is not adequately supported.
Our article on AI data readiness in the public sector explores how organisations can identify which weaknesses matter before connecting AI to existing information.
The result is not missing information. You’re getting an answer that sounds right but is wrong – misleading synthesis – which is far harder to detect and far more damaging.
4. Information retrieval vs decision-grade answers
This distinction is where most AI knowledge programmes quietly collapse.
Information retrieval answers questions such as:
- “Where is the document?”
- “What does this section say?”
- “Summarise this guidance.”
Decision-grade answers address a fundamentally different need:
- “Which policy applies in this situation?”
- “What is the approved course of action?”
- “What must happen before this can proceed?”
In public service, these questions carry personal, professional, and organisational accountability. They sit upstream of ministerial briefings, statutory decisions, procurement approvals, and regulatory actions.
Decision-grade answers require more than fluency. They require:
- Source authority
- Version control
- Hierarchical awareness
- Explicit handling of ambiguity
- The ability to state clearly when no approved answer exists
In practice, this means an answer should identify the sources supporting it, reflect the applicable version and authority of those sources, and make uncertainty visible when the material is incomplete or contradictory.
Where no sufficiently supported answer exists, the system should be able to acknowledge that limitation rather than present a confident synthesis. The accountable person must still determine how the information applies to the decision at hand.
Most AI tools stop at retrieval and summarisation. They are not designed to reason safely within institutional constraints.
The cost of that gap is not speed. It is trust.
5. Governance is not a wrapper. It is an architectural requirement
A common concern among SROs and senior digital leaders is the “black box” problem with decisions being influenced by systems that cannot explain themselves under scrutiny.
In public service, this is not a theoretical issue. Untraceable decisions are an operational and reputational risk.
Too often, governance is treated as something that can be layered on after deployment, through guidance, training, or policy. In AI knowledge systems, this approach fails.
Governance must be designed into the architecture itself.
That means:
- Every assertion must be traceable back to an identifiable source
- The system must respect existing access controls and classifications
- It must be able to surface uncertainty instead of resolving it autonomously
- It must enable humans to “show the working” behind any decision
The UK Government AI Playbook similarly emphasises appropriate access controls, testing, assurance, meaningful human control and the management of AI risks throughout the system lifecycle.
Source traceability is an essential part of assurance, but it does not by itself prove that the retrieved information is correct, current or appropriate for the user’s particular circumstances.
Without these properties, AI becomes a confidence amplifier rather than a control mechanism. It accelerates work, but weakens assurance.
For public sector organisations, that trade-off is rarely acceptable.
6. Human-in-the-loop. Accountability, not ethics
Human-in-the-loop (HITL) is often discussed as an ethical safeguard. In public service, its role also includes meaningful oversight, accountability and the ability to intervene when an AI-generated response is incomplete, uncertain or unsuitable for the decision being supported.
AI should not replace professional judgement. It should remove the drudge work that prevents qualified people from exercising that judgement effectively. By filtering, linking, and contextualising vast bodies of documentation, AI can shorten decision cycles without shifting responsibility away from accountable roles.
The critical point is that HITL is not about slowing AI down. It is about making responsibility explicit.
When designed and governed appropriately, AI can provide source-grounded information and possible interpretations to support a decision. The responsible person must retain the judgement and authority appropriate to the decision and its risks.
That distinction matters deeply in environments where accountability cannot be delegated.
7. The architectural shift. Why trust is not cosmetic
The organisations that succeed with AI-assisted knowledge work do not start with models. They start with architecture.
They move away from AI as a broad conversational layer and toward AI as a controlled reasoning interface over authorised internal knowledge.
This shift requires:
- Contextual weighting of internal sources
- Awareness of document hierarchy and precedence
- Explicit boundaries on access and inference
- The ability to surface conflict rather than resolve it silently
In practical terms, those requirements need mechanisms for identifying approved sources, maintaining document provenance and version status, enforcing access permissions and testing how the system behaves when information is missing or conflicting.
These controls can improve the reliability and auditability of AI-assisted knowledge retrieval. They do not remove the need for ongoing evaluation, operational monitoring or human judgement.
When these elements are present, AI stops being a search tool and becomes an interpretive aid, helping staff navigate complex policy environments without inventing new ones.
Trust, in this context, is not a matter of tone or disclaimers. It is a property of the system itself.
Why this matters now
Public sector leaders are being asked to move faster, reduce operational friction, and unlock value from existing data, while meeting applicable legal, regulatory and governance obligations and protecting public trust.
AI knowledge tools are often presented as a way to square that circle. But without architectural discipline, they risk doing the opposite; accelerating work while obscuring accountability.
An AI system that produces fast answers without understanding institutional context increases risk, even if it appears to improve productivity.
Leaders who recognise this early are not anti-AI. They are pro-governance. They understand that in public service, the question is not whether AI can answer, but whether it can answer responsibly.
A final thought
This challenge is not theoretical. Regulated organisations are already confronting the limits of general purpose AI knowledge search and learning that trust is architectural, not cosmetic.
In a highly regulated pharmaceutical organisation, Catapult helped make complex internal knowledge easier to retrieve and interpret, including material held in legacy SharePoint content and PowerPoint presentations.
The pharmaceutical KnowledgeAgent case study explores how the organisation approached source-backed answers and knowledge access without first rebuilding its entire document estate.
For public-sector leaders considering how trusted knowledge should support policy application and decisions, our article on moving from static policy to active compliance examines the next stage of that challenge.
