Case 6: Human in the Loop — Innovare AI Analysis

Module 3 · Video Walkthrough

Module 3 · Quiz Cheat Sheet

As you read — hold this question

When the AI makes a decision and it turns out to be wrong — who is accountable?

1976

Weizenbaum — Computer Power and Human Reason. The first systematic argument that some decisions should never be delegated to machines — not because machines can't make them, but because humans shouldn't abdicate them.

Joseph Weizenbaum built ELIZA in the 1960s — the first chatbot, which simulated a therapist. He was disturbed by how readily people formed emotional attachments to it and how seriously they took its responses. This led him to write Computer Power and Human Reason in 1976, arguing that even if machines could make certain decisions — in medicine, law, criminal justice, warfare — human society should not allow them to. The reason: accountability. When a human makes a decision and it goes wrong, there is someone who can be held responsible, who can explain their reasoning, who carries moral weight. When a machine makes the decision, accountability dissolves. Fifty years later, the argument is not weaker. It has become more urgent.

The automation decision framework

Quiz: Automate vs augment

Three modes — automate, augment, or preserve human judgment

Mode 1

Automate

AI makes the decision and executes without human review. Appropriate when: the decision is high volume and low stakes, outcomes are easily measurable, errors are low-cost and reversible, and the model has demonstrably high accuracy on this specific task.

Examples: Spam filtering, inventory reorder triggers, product recommendations, fraud alert scoring (not flagging — the score calculation).

Accountability: Belongs to the team that built, deployed, and operates the system — including the decision to automate.

Mode 2

Augment

AI informs and the human decides. Appropriate when: stakes are higher, errors are costly or irreversible, context and judgment matter, or the decision has ethical, legal, or reputational dimensions. The human retains authority and accountability.

Examples: Hiring recommendations (human decides), medical diagnosis support (clinician decides), credit underwriting (loan officer decides), parole recommendations (board decides).

Accountability: The human who made the decision — the AI provides input, not the outcome.

Mode 3

Preserve Human

Some decisions should not be delegated to AI regardless of capability. These are decisions where the act of human judgment — weighing, deliberating, being accountable — is itself part of what makes the decision legitimate.

Examples: Lethal autonomous weapons targeting, criminal sentencing, child welfare removal decisions, firing employees, end-of-life medical decisions.

Accountability: Non-delegable — this is precisely Weizenbaum's argument.

The Augmentation Trap

Augmentation is where the accountability is supposed to stay with the human — but often doesn't in practice. When an AI recommends and the human approves without genuinely deliberating, it's called "automation bias." The human is nominally in the loop but not actually exercising judgment. This is why augmentation requires design: the AI output must be presented in a way that invites scrutiny, not just confirmation. If the human always follows the AI recommendation, you don't have augmentation — you have automation with extra steps and diffused accountability.

Governance at scale

Federated AI governance — where oversight actually lives

When multiple AI systems operate across an organisation, central governance becomes impractical. A federated model distributes oversight responsibilities while maintaining common standards. Each layer handles what it's positioned to handle:

Centre of Excellence (CoE)

Sets the standards — doesn't run the projects

Defines technical standards, ethical guidelines, risk frameworks, and evaluation criteria. Provides tooling, templates, and approved model libraries. The CoE's job is to make it easy to do things right — not to approve every deployment. If the CoE is a bottleneck, it's been given the wrong job.

Business Units

Own the decisions — and the accountability

Each business unit maintains a designated AI steward who is responsible for all AI deployments within that unit. The steward ensures compliance with CoE standards, conducts pre-deployment risk assessments, monitors performance post-deployment, and is the named accountability holder. Accountability cannot be delegated to the model or the vendor.

End Users

Use the tools — and flag the problems

End users are the first line of detection for model failures, drift, and edge cases. They need training sufficient to recognise when the AI is wrong — not just how to use it. A governance model that doesn't train end users is relying on people to spot problems with tools they don't understand. That's not governance — it's hope.

External / Regulatory

Sets the floor — not the ceiling

Regulatory requirements (GDPR, EU AI Act, sector-specific rules) set minimum standards. Organisations that treat compliance as their governance ceiling are taking on more risk than they acknowledge. Regulations lag technology — by the time a risk is regulated, it has usually already caused harm somewhere. Best-in-class governance goes beyond compliance.

Take this away

Accountability follows the decision, not the tool. When AI is in the loop, the question isn't whether the machine made the right call — it's who decided to let the machine make the call, and who is responsible for what happens next. Weizenbaum's insight from 1976: some decisions are human decisions not because machines can't make them, but because humans shouldn't abdicate them.

← Case 5: The Five Factors

Module hub

Back to course →

Quick recall — without looking back

Test yourself on this case

Question 1 of 3

What is automation bias — and why does it undermine the "human in the loop" design?

Automation bias: the tendency of humans to over-rely on automated recommendations, accepting them without genuine critical evaluation. It undermines "human in the loop" design because the human's presence becomes nominal rather than substantive — they approve the AI's recommendation without actually deliberating. The result is that accountability is supposed to rest with the human (because they approved it) but the human didn't actually exercise judgment. You get diffused accountability with no actual check on the model. Mitigation: design the AI output presentation to invite scrutiny (e.g., show confidence scores, present alternatives, require explicit reasoning for high-stakes approvals) rather than just making it easy to click "approve."

Question 2 of 3

Weizenbaum's 1976 argument is sometimes summarised as "just because a machine can make a decision doesn't mean it should." What is the deeper principle he was making — and which type of decisions does he say humans must retain?

The deeper principle: accountability is not just a procedural requirement — it is part of what makes certain decisions legitimate. A medical diagnosis carries weight partly because a trained professional with personal accountability made it. A criminal sentence is legitimate partly because a human judge who can explain and defend their reasoning imposed it. When these decisions are delegated to machines, the legitimacy changes — not because the machine can't produce the same output, but because the act of human deliberation and accountability is itself constitutive of the decision's legitimacy. Weizenbaum says humans must retain decisions that are irreversible and consequential (medicine, law, criminal justice, warfare) and decisions where the reasoning must be explicable and the decision-maker must be morally accountable — not just legally liable.

Question 3 of 3

Describe the three layers of a federated AI governance model and what each layer is responsible for.

Centre of Excellence (CoE) — sets standards, not projects. Defines technical standards, ethical guidelines, risk frameworks, evaluation criteria, and approved model libraries. Its job is to make it easy to do things right, not to approve every deployment. If it becomes a bottleneck, it's been given the wrong role. Business Units — own the decisions and the accountability. Each unit has a designated AI steward responsible for pre-deployment risk assessment, compliance with CoE standards, post-deployment monitoring, and named accountability for all AI deployments within the unit. Accountability cannot be delegated to the model or the vendor. End Users — first line of detection. They are positioned to spot model failures, drift, and edge cases — but only if they've been trained to recognise when the AI is wrong, not just how to use it. A governance model that relies on untrained end users to spot problems is not a governance model — it's wishful thinking. (Regulatory layer: sets the compliance floor, not the governance ceiling.)

Sources

Module transcript: BUSN9049 Module 3 Part 1 — Strategic Implications of AI. Flinders University, 2026.
Weizenbaum: Weizenbaum, J. (1976). Computer Power and Human Reason: From Judgment to Calculation. W.H. Freeman.
Russell & Norvig: Russell, S. & Norvig, P. (2020). Artificial Intelligence: A Modern Approach (4th ed.). Pearson. Chapter on AI safety and governance.
EU AI Act: European Parliament. (2024). Regulation 2024/1689 on Artificial Intelligence (EU AI Act). Annex III: high-risk AI systems requiring mandatory human oversight.
Davenport: Davenport, T.H. & Mittal, N. (2022). All-in on AI: How Smart Companies Win Big with Artificial Intelligence. Harvard Business Review Press.