Case 4: Human-AI Collaboration — Innovare AI Analysis

As you read — hold this question

Who is actually accountable when the algorithm is wrong?

470K

false debt notices issued by Robodebt's automated algorithm to welfare recipients — many of whom owed nothing

$1.76B

class action settlement paid by the Australian government in 2021 — the direct cost of replacing human judgment with algorithmic substitution

human welfare officers reviewing individual cases before automated debt notices were issued — the entire process ran without human decision-making

The framework — Wilson & Daugherty (2018)

Quiz: Human–AI

What the augmentation model actually says

Wilson & Daugherty's Harvard Business Review research identified a consistent pattern: organisations that augment human workers with AI significantly outperform those that substitute humans with AI. Augmentation is not about slowing automation down — it is about deploying it where it genuinely adds value while keeping human judgment where it remains essential.

Augmentation — what the model prescribes

AI handles high-volume pattern detection and data retrieval — tasks where it outperforms humans. Humans retain judgment on ambiguous, contextual, or consequential decisions — tasks where they outperform AI.

The process is reimagined around the collaboration, not just automated as-is. Humans remain accountable and correctable.

Substitution — what companies often do instead

The existing human process is automated end-to-end. Human roles are removed. The algorithm inherits the accountability that humans previously held — but cannot exercise judgment, contextualise edge cases, or explain itself.

When it fails, there is no human backstop. Errors scale at the same rate as the automation.

Wilson, H.J. & Daugherty, P.R. (2018). Collaborative intelligence: Humans and AI are joining forces. Harvard Business Review, 96(4), 114–123.

Applied — Robodebt 2016–2019, Australia

Who decided? Mapping the Robodebt process against the augmentation model

The augmentation model requires identifying where AI should lead and where humans should lead. Robodebt never asked that question. Every decision node that previously carried human judgment was handed to an algorithm using a legally invalid methodology — dividing annual ATO income data by 52 to produce a fortnightly figure the ATO itself said should not be used that way.

Augmented approach — what the model prescribes

Algorithm leads

Retrieve ATO income data

Cross-reference Centrelink payment records. Flag statistical discrepancies for review. Fast, accurate data retrieval is exactly what algorithms do well.

↓

Human leads

Case review with context

Welfare officer reviews flagged cases. Accounts for casual work patterns, disability, seasonal income, reporting lags. Exercises the professional judgment that algorithms cannot.

↓

Human leads

Determination + contact

Officer makes the debt determination with full accountability. Contacts recipient directly. Explains the basis. Provides a repayment pathway and support referrals.

↓

Algorithm assists

Track repayment + flag issues

Monitor repayment plans, generate reminders, flag non-compliance for human review. Automation handles volume; humans handle exceptions.

What Robodebt actually did

Algorithm only

Retrieve ATO income data

Retrieved annual income figures from the ATO — data the ATO explicitly stated should not be used for fortnightly income calculations.

↓

Algorithm only

Income averaging — legally invalid

Divided annual income by 52 to produce a weekly figure. Applied this to fortnightly Centrelink periods. The methodology was never legally valid. No human reviewed this step.

↓

Algorithm only

Debt calculated + notice issued

Algorithm calculated a debt figure and issued an automated letter demanding repayment — often to Australia's most vulnerable citizens — with no human review or approval at any point.

↓

Burden reversed

Recipient must disprove the debt

Onus of proof reversed — recipients had to prove they did not owe the money. For those who couldn't navigate the process, debt collectors were engaged. Some recipients took their own lives.

The result — does Robodebt meet the augmentation criteria?

Wilson & Daugherty scorecard applied

Augmentation criterion

Augmented ideal

Robodebt reality

Process reimagined around human-AI collaboration — not just existing process automated as-is

✓ Yes

✗ No — existing debt recovery process automated end-to-end

Human judgment retained at consequential decision nodes (debt determination)

✓ Yes

✗ No — zero human review before notices issued

Humans able to train, correct and contextualise the AI's outputs

✓ Yes

✗ No — error correction onus placed on debt recipients, not the system

Legal validity of the automated methodology verified

✓ Yes

✗ No — income averaging was never legally valid; ATO warned against it

Error rate and scale of harm manageable and correctable

✓ Yes

✗ No — automation scaled the error rate: 470,000 false notices

Outcome: The Royal Commission (2023) found the scheme was unlawful from the outset. The Federal Court class action settled for $1.76B. The automation didn't fail because algorithms are incapable — it failed because substitution was chosen where augmentation was required, and because the people making that choice faced no personal accountability for the outcome.

The substitution trap — private sector

Robodebt isn't an outlier. It's the pattern when substitution replaces augmentation.

Between 2022 and 2024, multiple large organisations publicly committed to AI-driven workforce substitution. The results were mixed at best — and in several cases forced reversals, productivity losses, or reputational damage.

Klarna

"Our AI does the work of 700 customer service agents." — CEO Siemiatkowski, 2024

By late 2024, Klarna was quietly restoring human customer service roles after customer satisfaction metrics deteriorated on complex queries — the exact cases where human judgment is most needed.

Substitution → partial reversal

Duolingo

Cut contract content creators in Jan 2024, citing AI as the replacement for human content production.

AI-generated content proved inadequate for nuanced language instruction. Human expertise in pedagogy and cultural context wasn't replicable by generation models. Contractors were re-engaged for specialist content.

Substitution → capability gap exposed

IBM

CEO Arvind Krishna announced plans to pause hiring 7,800 roles expected to be replaced by AI (2023).

By 2024 IBM was actively hiring again in many of the same functions — AI tools had augmented productivity in some areas but required human oversight and specialist configuration to deliver value.

Substitution → augmentation recalibrated

The consistent pattern: substitution announcements are made at the top of the AI hype cycle. The recovery — rehiring, capability gaps, quality issues — happens quietly 12–18 months later when the productivity loss becomes visible in business metrics.

Framework hole — what Wilson & Daugherty don't account for

The model assumes organisations want better outcomes. Robodebt shows they don't always.

The intent problem

Wilson & Daugherty build their model around a premise: that organisations deploy AI to improve performance and outcomes. The augmentation framework then gives them a map for doing that well. But Robodebt wasn't deployed to improve welfare services. It was deployed to recover debt at volume while creating plausible deniability — "the algorithm decided." The human harm was not a side effect of poor design; it was, at minimum, a foreseeable consequence of design choices made to reach political debt-recovery targets.

The augmentation model has no vocabulary for this. It cannot answer the question: what happens when an organisation deliberately uses automation to obscure accountability, reverse the burden of proof, or scale a process that would have been stopped if humans were reviewing individual cases? That is not a technical failure. It is a governance failure — and no amount of better human-AI collaboration design resolves it without accountability structures that sit entirely outside the framework's scope.

The corrected frame

Applied to organisations today: before deploying AI in any consequential process, the first question is not "how do we augment?" but "who is accountable when this is wrong, and can they actually exercise that accountability?" Robodebt's substitution wasn't the only failure. The failure was that no individual in the system bore personal accountability for the 470,000 errors — the automation distributed the decision so widely that responsibility dissolved. Augmentation only works when the humans retained in the loop have genuine power to stop, correct, and override the system.

Take this away

Augmentation only works when the humans retained in the loop have genuine power to stop, correct, and override the system—Robodebt’s failure was not the algorithm, it was the absence of anyone who could say no.

← Case 3: The Ethics Template

Case 5: The Filter Bubble →

Quick recall — without looking back

Test yourself on this case

Question 1 of 3

What is the key distinction between augmentation and substitution in the Wilson & Daugherty model?

Augmentation reimagines the process around human-AI collaboration: AI handles high-volume pattern detection where it outperforms humans, humans retain judgment at consequential and contextual decision nodes where they outperform AI. Humans remain accountable and correctable. Substitution automates the existing process end-to-end: human roles are removed, the algorithm inherits accountability it cannot exercise, and when it fails there is no backstop—errors scale at the same rate as the automation.

Question 2 of 3

Name two specific decision nodes in Robodebt where the augmentation model required human judgment. What happened instead?

The augmentation model requires human judgment at: (1) case review with context—a welfare officer accounting for casual work patterns, disability, seasonal income, and reporting lags before flagging a discrepancy as a debt; and (2) debt determination—an officer making the final decision with full accountability, contacting the recipient directly. In Robodebt, both nodes were handed entirely to an algorithm using a legally invalid methodology (annual ATO income ÷ 52 = fortnightly figure), with zero human review before 470,000 notices were issued.

Question 3 of 3

What assumption does the Wilson & Daugherty model make that Robodebt directly contradicts?

The model assumes organisations deploy AI to improve outcomes—and gives them a map for doing that well. Robodebt was not deployed to improve welfare services. It was deployed to recover debt at volume while creating plausible deniability. The model has no vocabulary for an organisation that deliberately uses automation to obscure accountability or reverse the burden of proof. Augmentation also only works when retained humans have genuine power to override—Robodebt had no such mechanism, and no individual bore personal accountability for the 470,000 errors.

Sources

Wilson & Daugherty (2018): Wilson, H.J. & Daugherty, P.R. (2018). Collaborative intelligence: Humans and AI are joining forces. Harvard Business Review, 96(4), 114–123.
Royal Commission (2023): Commonwealth of Australia (2023). Final Report: Royal Commission into the Robodebt Scheme. Australian Government.
Class action (2021): Gordon Legal (2021). Robodebt Federal Court class action — $1.76B settlement. Federal Court of Australia.
ATO warning: Australian Taxation Office evidence to Royal Commission: ATO data was not designed or intended for use in fortnightly income assessment, and the ATO communicated this to DHS.
Klarna (2024): Siemiatkowski, S. (2024). CEO interview, Financial Times. Subsequent reporting by Bloomberg and The Guardian on customer service rehiring, Oct–Dec 2024.
Duolingo (2024): Contractor layoffs reported by The Verge, January 2024. Subsequent content quality reporting, mid-2024.
IBM (2023–24): Krishna, A. (2023). CEO interview, Bloomberg. IBM Q3 2024 earnings commentary on workforce composition.