Stop Trusting AI. Start Measuring It.

It starts innocently enough. You open a new analytics dashboard that promises to summarise client activity and highlight emerging risks. In seconds, it produces a clean, confident paragraph - the kind of summary you might have spent half an hour assembling from spreadsheets and notes. You scan it, nod and drop it straight into your briefing pack.

It looked right. But did you know it was really right? That tiny moment of hesitation captures a much bigger challenge.

Each of these “intelligent” tools promises to save a few minutes and in doing so, quietly outsources another small act of decision-making - a summary, a presentation,  a prioritisation of what matters. 

We trusted people because we’d seen them keep their word, companies because their reputations endured and institutions because accountability was visible. That foundation hasn’t vanished, but it now has to stretch to cover systems that can chat, act and even persuade - and yet never feel the weight of consequence.

Large language models operate at a scale and speed that tempt us to skip our usual checks and intuitions. When a machine’s output can influence a client conversation, an investment call or a hiring choice, a few minutes of verification can prevent a costly mistake.

As AI adoption accelerates across every sector, the stakes have never been higher, making a disciplined approach to trust not just an advantage but a necessity for business continuity and competitive advantage. So the question is when that trust is deserved, how to verify it and where to draw the line. Because in this new landscape, trust is a discipline we must build.

Psychologists Mayer, Davis and Schoorman describe trust as a mix of ability, benevolence and integrity - in simpler terms:

Ability (Competence): Can they do what they say they can do? Do they have the necessary skills and knowledge?
Benevolence (Good intent): Do they have my best interests at heart? Do they care about me?
Integrity (Fairness): Are they honest, consistent and fair? Do they stick to their word and principles?

Sociologist Niklas Luhmann sees trust as a functional necessity for modern society. It’s not just a nice-to-have, it’s the tool we use to navigate an otherwise overwhelming world. Trust allows us to move from a state of paralysing uncertainty to one where action becomes possible.

As AI systems expand the scale and speed of decision-making, they magnify exactly what Luhmann described - the need for trust as a functional substitute for full understanding.

On a daily basis, we judge competence through people, in AI, we infer it through process, data and design.

The National Institute of Standards and Technology (NIST) proposes nine measurable dimensions of trust: accuracy, reliability, resiliency, objectivity, security, explainability, safety, accountability and privacy.

They may sound new but they play out in ways that we already understand:

Accuracy:
If a bond-pricing model returns the wrong mid-point by just a few basis points, it’s real money, real exposure.

Reliability:
Is what keeps the same model producing consistent results on day in and out, regardless of traffic, volatility or server load. 

Resiliency:
Shows up when things break. Every major desk has seen a deployment that didn’t behave as expected, the trustworthy systems are the ones with rollback plans, fail-safes and versioning discipline baked in.

Objectivity:
Means the algorithm’s incentives align with the client’s, not the P&L of the desk.

Accountability:
Is what lets a model operationally explain itself later. Who approved the data sources? Who pushed the update? Where’s the audit trail?

Explainability:
When a regulator or risk officer asks why a model tightened or widened a price, reproducible and inspectable explanations are helpful. 

And then there’s the quieter layer: security, privacy and safety.
These are the hygiene factors that only become visible once violated. A model that ingests market orders or private deal data must treat that information like collateral - ring-fenced, encrypted and access-controlled. A single misconfiguration can turn an LLM from a productivity aid into a compliance nightmare. Safety isn’t only about physical harm, in financial systems, it also creates reputational risk.

Together, these examples show how “trust” starts to become an architectural choice.
We need to ask, “Under what conditions does this system behave predictably and how quickly can I spot it when it doesn’t?”

Standards such as ISO/IEC 42001 now give organisations a formal structure for managing AI responsibly - the equivalent of ISO 9001 for quality or ISO 27001 for information security.
Certification shows that a company has governance in place: defined responsibilities, documented processes and continuous improvement.

But certification audits the process - that governance structures are in place but not the performance of the system itself. A certified vendor’s model can still hallucinate product codes or misclassify risk levels. That gap shifts accountability back to the deployer, who needs to own the due diligence.

Certification signals maturity and discipline behind the system. But belief comes from daily practice - how quickly errors surface, how transparently they’re reported and how systematically they’re fixed.

The NIST dimensions describe what makes a system trustworthy. Trust thresholds define how much trust is enough for each use case.

A trust threshold sets the minimum confidence required before an AI output can be acted on safely. It depends on three questions:

In practice, this means judging both what happens if it’s wrong and how easily it can be fixed.

A customer-support chatbot suggesting the wrong help article has low consequence and high reversibility - the user simply clicks away. But a credit or trading model that misclassifies risk has high consequence and low reversibility - the damage is real, often public and difficult to unwind.

Same accuracy, very different trust requirements.

Setting thresholds means translating those principles into operational boundaries:

Thresholds evolve as systems learn or contexts shift. In advanced environments, they’re embedded directly into workflows and low-confidence results go to humans, high-confidence ones route automatically. Trust starts to become part of your control system.

Each stakeholder group reads trust through a different lens and organisations must learn to communicate credibility on all of them.

Investors look for assurance and predictability.
They don’t expect zero risk but they expect evidence that it’s known, modelled and actively managed.
Just as financial statements build trust in markets, AI assurance reporting - structured disclosure of model performance and governance controls will build trust in automation. And the organisations that get ahead of this won't just satisfy investors but also position themselves to attract capital with better terms as they demonstrate operational maturity.

Regulators focus on control and transparency.
They want to know not just how the system works but how it fails - what safeguards activate, what data is logged, who intervenes.
Certifications such as ISO/IEC 42001 can signal responsible intent, but the real proof lies in testing routines, documentation trails and feedback loops that show continual improvement.

Employees seek psychological safety.
They need to know that AI tools support, rather than surveil or replace, their expertise.
That safety grows through clear guidance - when to verify, when to defer, when to override - and through visible leadership behaviours like open questioning without penalty.

Clients and the public care most about consistency and fairness - that their data is respected, decisions are explainable and mistakes are acknowledged.
Public trust deepens when people can see how confidence is measured, how boundaries are enforced and how problems are corrected in a system.

Mature AI systems don’t rely on reputation, they prove reliability in operation. In these organisations, trust is monitored continuously and drift is caught before it causes damage. Failures are logged and fed back into improvement loops, not buried under blame. Humans and models work in harness: the system surfaces confidence levels and exceptions instead of forcing blind acceptance.

Governance is embedded from day one, with model cards, version control and access logs making accountability routine. And trust thresholds are visible and debated, so boundaries evolve as context changes.

This level of discipline starts separating firms that scale AI responsibly.

In some ways, financial institutions already delegate judgment to machines - trading algorithms, credit models and fraud detection. They operate under strong governance regimes where oversight, auditability and accountability are part of the daily routine. 

As LLMs and agentic AI introduce probabilistic reasoning and adaptive behaviour, existing guardrails must evolve, system by system and context by context, to keep control aligned with capability.

Imagine an AI assistant empowered to approve supplier contracts under £50,000, or draft regulatory filings for human sign-off. On its own, that sounds contained. But large financial institutions already run hundreds of interconnected models and process automations.

Multiply that level of autonomy across every function - procurement, credit, compliance, finance, sales and trading and what looks like a local decision becomes a systemic transformation - one that touches the organisation’s entire strategy.

That evolution will require explicit delegation frameworks defining what agents can do autonomously versus what needs escalation, supported by real-time observability and human override at critical checkpoints.

Trust in AI comes from the checks, reversibility and accountability that keep performance stable even under pressure. Financial institutions already understand this logic and it’s built into every audit trail, limit order and compliance dashboard.

What changes with LLMs is the scope of interpretation, the speed of interaction and the potential for opaque reasoning to slip through those safeguards.

The organisations that apply the same rigour to these new models - measuring trust carefully and treating every failure as a learning signal - will maintain control while others chase hype. Trust becomes a controlled process, designed and maintained over time. The firms that understand this will set the benchmark for what trustworthy AI really means.

The firms that master this transition won’t just manage risk - they’ll define what winning with trustworthy AI means for the industry. It is a strategic differentiator for organisations serious about scaling with AI.

To illustrate, imagine two AI systems inside an investment bank:

Both use the same underlying model and show strong performance metrics across standard tests.
Yet the context of use - who sees the output, how visible the errors are and what harm mistakes could cause - changes everything.

The table below illustrates how the same architecture can meet - or miss - its trust boundary depending on context.

Legend: 🟢 Meets / exceeds 🟠 Marginal 🔴 Fails

Note: A trust threshold is the agreed minimum performance level at which an AI system’s output can be acted upon without additional human verification. It is jointly defined by business, technical, and governance teams to align with the organisation’s risk appetite, use-case sensitivity, and regulatory requirements.

Both systems use similar technology and data sources, yet only one meets its trust boundary.
The internal dashboard passes comfortably: humans remain in the loop, any errors are visible and consequences are limited.
The email system, while technically strong, crosses multiple critical lines - in accuracy, explainability and privacy - where small failures carry outsized reputational or regulatory risk.

That distinction is what operational trust looks like. It’s about defining where verification matters most - then documenting, testing and refining those thresholds over time.

Recent events show what happens when that calibration fails.

In October 2025, Deloitte was forced to revise and partially refund an A$440,000 report for the Australian government after multiple hallucinations were found - including fabricated academic sources and a fake quote from a federal court judgement.

The issue wasn’t bad intent or broken models, but misplaced confidence in unverified outputs.

A clearer trust threshold, for example, mandating “human review for legal or academic references” - might have prevented both the reputational damage and the refund.