SWANK AI Guidance Note 20
High-Stakes AI · Governance · Human Accountability

Core Standard

GOVERN ACCORDING TO CONSEQUENCE

The greater the potential effect on a person’s:

rights

safety

opportunities

essential services

the stronger the requirements for:

evidence

human judgment

source traceability

uncertainty

challenge

and

reconsideration.

Not every AI use carries the same level of consequence.

Governance should reflect that difference.

Purpose

An AI system generating draft meeting notes is not operationally equivalent to an AI system contributing to decisions about:

healthcare;
safeguarding;
education;
employment;
benefits;
disciplinary action;
financial access;
legal rights;
public services;
or regulatory intervention.

The relevant question is not simply:

How capable is the AI system?

It is also:

What can happen if it is wrong?

The potential consequence of error should influence the strength of governance surrounding the system.

Consequence-Sensitive Governance

AI governance should be proportionate to the potential effect of the use case.

As consequence increases, organisations should generally require stronger controls around:

evidence quality;
source traceability;
meaningful human review;
transparency;
error correction;
override;
reconsideration;
documentation;
testing;
auditability;
incident response;
and decision ownership.

A low-risk productivity tool and a system contributing to a consequential human decision should not automatically operate under the same governance standard.

High Stakes Are Not Only Financial

An AI system can be high-stakes even where no direct financial transaction occurs.

Potential consequences may include:

loss of opportunity;
loss of service access;
educational disadvantage;
family disruption;
reputational harm;
medical harm;
legal consequences;
professional consequences;
psychological or social effects;
or restrictions affecting a person’s everyday life.

Governance should therefore examine the real-world consequence, not only the monetary value of the decision.

Rights and Essential Services

Particular caution may be appropriate where AI materially affects access to:

healthcare;
housing;
education;
welfare or benefits;
employment;
safeguarding services;
financial services;
justice or administrative remedies;
or other essential institutional functions.

Where people may have limited ability to avoid the institution making the decision, the quality of governance becomes especially important.

A person should not be placed at substantial disadvantage simply because the institution’s internal AI process is difficult to understand or challenge.

Consequence Is More Than Probability

Risk is not determined only by how likely an error is.

An event may be relatively uncommon but still require strong controls if the potential consequence is severe.

Organisations should therefore consider:

likelihood

alongside

severity

reversibility

duration

number of people affected

and

ability to obtain correction.

A low-frequency error can still justify substantial governance where the consequence is difficult to reverse.

Reversibility

One of the most important high-stakes governance questions is:

If the system contributes to a wrong decision, can the consequence realistically be reversed?

Some decisions may be corrected relatively easily.

Others may create effects that are:

difficult to undo;
time-sensitive;
cumulative;
reputationally persistent;
legally consequential;
or practically irreversible.

Where reversibility is weak, the quality of review before the decision becomes more important.

Review Before Consequence

High-stakes environments should place particular emphasis on meaningful human review before consequential action occurs.

The responsible reviewer should be able to:

access relevant source material;
understand what the AI contributed;
identify uncertainty;
consider contradictory evidence;
obtain additional information;
reject or modify the AI output;
pause the process where necessary;
and remain responsible for the final decision.

Human review should not be reduced to a formal approval step after the automated process has effectively determined the outcome.

Avoiding Automated Inference

High-stakes systems should exercise particular caution where AI is asked to infer matters such as:

credibility;
intent;
motivation;
emotional state;
capacity;
future behaviour;
safeguarding risk;
suitability;
or other complex human characteristics.

These variables are often:

context-dependent;
difficult to establish from data alone;
sensitive to missing information;
influenced by language and environment;
and consequential when interpreted incorrectly.

AI may assist the organisation of evidence.

It should not silently transform uncertain human information into an authoritative determination.

Language Is Not the Person

AI systems may infer patterns from:

correspondence;
speech;
behavioural records;
written submissions;
complaints;
or other forms of communication.

But language may be influenced by:

disability;
age;
culture;
education;
stress;
translation;
AI assistance;
professional support;
communication style;
or context.

High-stakes decisions should not infer consequential personal characteristics from linguistic patterns alone.

Higher Standards for Evidence

As consequence increases, organisations should ask more demanding evidential questions.

For example:

What evidence supports this conclusion?
Is the source identifiable?
Is the evidence current?
Is important information missing?
Do sources conflict?
Is the material independently corroborated?
Is the conclusion based on fact or inference?
Has AI-generated content been verified?
Can another reviewer reconstruct the evidence pathway?

A consequential decision should not become stronger merely because an AI system expresses it confidently.

Source Traceability

High-stakes AI use should preserve a path from:

decision

AI contribution

material proposition

underlying source.

Where a reviewer encounters:

“The system indicates…”

they should be able, where proportionate, to determine:

what information the system received;
which sources mattered;
whether those sources were disputed;
whether later evidence changed the position;
and what role human judgment played.

Traceability supports both accountability and correction.

Uncertainty Should Become More Visible

In low-consequence contexts, an imperfect recommendation may sometimes be tolerable.

In a high-consequence environment, uncertainty should become more visible rather than less.

Appropriate outputs may include:

Insufficient information.

Human review required.

Sources conflict.

Further verification is required.

This conclusion should not be relied upon without additional evidence.

Refusing to create false certainty can itself be a safety feature.

Uncertainty Is Not System Failure

A well-governed AI system should not be judged solely by how often it produces a definitive answer.

Sometimes the strongest output is:

The available evidence does not support a reliable conclusion.

High-stakes systems should distinguish:

decision support

from

pressure to generate certainty.

An organisation should not require a model to answer questions the available evidence cannot responsibly resolve.

Human Accountability

Where AI materially affects people, responsibility should remain identifiable.

The organisation should be able to answer:

Who approved the system?
Who approved this use case?
Who reviewed the output?
Who considered the source evidence?
Who had authority to disagree?
Who made the final decision?
Who can correct the record?
Who can authorise reconsideration?
Who remains accountable if the decision is wrong?

Responsibility should not disappear into:

the model

the vendor

the workflow

the algorithm.

Meaningful Override

High-stakes systems should provide practical override capability.

A human reviewer may need to:

reject an AI recommendation;
change a classification;
request further evidence;
pause automated action;
escalate uncertainty;
obtain independent review;
or determine that the AI system should not be used for the particular case.

Override should be operationally legitimate.

A technically available override that staff are discouraged from using may provide weak assurance.

Contestability

Where AI contributes materially to a consequential outcome, affected individuals or responsible reviewers should have an appropriate route to challenge:

inaccurate information;
unsupported inference;
incorrect classification;
incomplete records;
or the resulting decision.

A challenge pathway should make clear:

what can be challenged;
who reviews the challenge;
what evidence may be supplied;
whether the underlying record can be corrected;
and whether the outcome can be reconsidered.

A high-stakes system that cannot be meaningfully challenged is difficult to assure.

Correction

AI-assisted errors may propagate rapidly through institutional records.

A correction process should therefore consider:

the original inaccurate output;
underlying source evidence;
summaries that reused the error;
classifications affected by it;
downstream decisions;
later AI prompts;
and whether reconsideration is required.

Correcting a record is not always sufficient if the consequence produced by that record remains unchanged.

Reconsideration

High-stakes governance should include mechanisms for changing a decision when material information changes.

Reassessment may be required where:

new evidence emerges;
contradictory information appears;
an earlier factual error is established;
circumstances materially change;
the AI system is shown to have omitted relevant information;
or the original use of the system is found to be inappropriate.

A consequential decision should not become permanent merely because it was made first.

Testing Under Difficult Conditions

High-stakes AI systems should not be tested only using clean or ideal inputs.

Testing should examine performance where:

sources conflict;
information is missing;
chronology is unclear;
users communicate unusually;
records are duplicated;
citations are fabricated;
new evidence contradicts earlier conclusions;
and human reviewers need to override the system.

The quality of a consequential AI system becomes particularly visible under uncertainty.

False Positives and False Negatives

High-stakes testing should consider both directions of error.

False Positive

The system identifies a concern, risk or condition that is not adequately supported.

False Negative

The system fails to identify a genuine concern.

Both can produce significant consequences.

The correct balance depends upon:

the use case;
affected population;
consequence;
available human review;
and reversibility.

Organisations should understand which error they are optimising against and what happens when the opposite error occurs.

Incident Response

Where a material AI incident occurs, organisations should be able to:

identify it;
contain it;
determine who was affected;
trace the information;
correct relevant records;
reconsider affected decisions;
identify root cause;
and reduce recurrence.

High-stakes governance should assume that failure is possible and prepare for it.

A system is not resilient merely because no incident has yet been identified.

Independent Review

Where AI materially affects high-stakes decisions, organisations may benefit from periodic independent review of:

governance;
workflows;
model use;
human oversight;
source integrity;
incident history;
error correction;
operational drift;
and whether the system remains appropriate for the original purpose.

Independent review can help distinguish between:

the existence of controls

and

evidence that those controls work in practice.

Operational Drift

An AI system may begin in a low-consequence role and gradually become more influential.

For example:

Initial use: administrative summarisation.

Later use:

triage;
risk classification;
recommendation;
or consequential decision support.

The model may not have changed.

The governance requirement has.

Organisations should reassess AI systems when their operational role becomes more consequential than the use originally approved.

Vendor Dependency

Where high-stakes processes depend on external AI vendors, organisations should understand:

model limitations;
data practices;
system changes;
subcontractors;
incident notification;
service availability;
auditability;
and exit arrangements.

Institutional accountability remains with the organisation even where technical functions are provided externally.

The vendor relationship should not make responsibility less visible.

Vulnerable Populations

Additional caution may be appropriate where AI-supported systems affect people who may:

have limited ability to challenge decisions;
rely heavily upon essential services;
experience communication barriers;
lack technical literacy;
be children;
have accessibility needs;
or face significant consequences from administrative error.

Governance should consider not simply the average user, but those most affected when the system gets something wrong.

Proportionality

High-stakes governance does not require maximal controls for every use.

The appropriate governance burden should reflect:

consequence;
probability of error;
evidence quality;
reversibility;
human oversight;
affected population;
complexity;
and operational context.

The principle is proportionality.

More consequence should produce more scrutiny.

Questions for Organisations

Where AI is used in a consequential environment, organisations may ask:

What can happen if the system is wrong?
How reversible is the consequence?
What evidence supports the output?
Can material claims be traced to source?
Is uncertainty visible?
Is meaningful human review occurring before consequence?
Can the human reviewer reject the AI output?
Are complex human characteristics being inferred from weak proxies?
Can affected people challenge inaccurate information?
Can corrections change downstream decisions?
Has the system been tested under contradiction and uncertainty?
Can an independent reviewer reconstruct what happened afterwards?

SIAAF Relevance

This Guidance Note engages all seven SIAAF domains.

Domain 01 — Governance & Accountability

Who is responsible for the high-stakes use and its consequences?

Domain 02 — Decision Integrity & Human Oversight

Are humans exercising genuine judgment before consequential decisions are made?

Domain 03 — Evidence & Traceability

Can material conclusions be traced back to reliable source evidence?

Domain 04 — Communication & Feedback Integrity

Are uncertainty, corrections and material information communicated effectively?

Domain 05 — Escalation, Challenge & Contestability

Can an affected person or reviewer challenge and obtain reconsideration of an AI-supported conclusion?

Domain 06 — Risk, Harm & Operational Resilience

Does the organisation understand foreseeable harm and its response to failure?

Domain 07 — AI Literacy & Organisational Readiness

Do the people using and governing the system understand it well enough to exercise judgment?

SWANK AI Standard

GOVERN ACCORDING TO CONSEQUENCE

The greater the potential effect on:

rights

safety

opportunity

essential services

significant human interests

the stronger the requirements should become for:

evidence

traceability

human review

uncertainty

override

challenge

correction

reconsideration

and

independent assurance.

High stakes should produce greater care.

They should not produce greater certainty than the evidence supports.

Related SWANK AI Guidance

Guidance Note 03 — Human Oversight in AI-Assisted Decision Systems

Guidance Note 08 — Reassessment Alongside Escalation

Guidance Note 09 — AI Governance for Children and Safeguarding Systems

Guidance Note 11 — Meaningful Human Review

Guidance Note 14 — AI Incident Reporting and Error Correction

Guidance Note 15 — Testing AI Under Contradiction and Uncertainty

Guidance Note 17 — Source Traceability by Design

Guidance Note 19 — Correction Rights in AI-Assisted Systems

SWANK AI

Independent AI & Institutional Assurance

We do not just review AI. We review the institutional systems responsible for governing it.