When a measure becomes the target that justifies a decision, does it still measure anything real?
In July 2025 the Commonwealth Bank told 45 call-centre staff their roles were redundant because a new AI voice-bot had cut incoming calls. The people on the phones knew the calls were not going down — they were going up. Within weeks some of the same workers, just told they were surplus, were asked back on overtime because the bot could not handle the hard calls. The Finance Sector Union's Julia Angrisano said the bank had been caught dressing up job cuts as innovation. The bank admitted in writing that its assessment 'did not adequately consider all relevant business considerations' and the roles were not redundant.
There is a name for how a number goes wrong like this: Goodhart's Law — when a measure becomes a target, it stops being a good measure. The moment "calls down 2,000 a week" became the justification for the cuts, it stopped measuring reality and started serving the decision. The technology worked. The failure was a management decision made on the wrong number.
A good goal is slippery, so the profession built a shorthand: SMART — Specific, Measurable, Achievable, Realistic, Time-bound (George Doran, 1981). An AI goal without a deadline cannot be measured. To measure, you reach for OKRs (qualitative Objectives and their measurable Key Results) and KPIs (the quantitative gauges). Each KPI ladders up to a macro objective — e.g. Objective: increase efficiency through automation; KPIs: reduce manual processing time by 50%, or reach 90% accuracy on AI-generated reports.
Here is where AI projects go wrong: the obvious metric — model accuracy — is not the one that matters. Your course lists the KPI categories that do:
| KPI category | What it measures |
|---|---|
| Return on investment | Financial return from added revenue + cost savings vs. AI investment |
| User adoption & engagement | User adoption rate; feature utilisation — are people actually using it? |
| Personalisation effectiveness | Personalisation conversion rate; personalisation accuracy |
| Brand perception & loyalty | Brand sentiment (before/after); brand loyalty |
| Stakeholder & customer satisfaction | Stakeholder engagement; customer feedback |
Up to 80% of AI projects fail to deliver their intended value. Marr (2025) names three pitfalls, and none of them is the technology:
What successful organisations do differently: start with a clear business case and measurable objectives; invest in data quality first; do change management and training from day one; and start small with a pilot before scaling.
Measure the value, not the maths. A metric that becomes the justification for a decision stops describing reality. Define SMART objectives, ladder KPIs up to real business outcomes, and never let model accuracy stand in for whether the project actually helped anyone.
What does SMART stand for, who introduced it and when, and why must an AI goal be time-bound?
Distinguish OKRs from KPIs, with an example.
Name the five KPI categories the course lists for AI implementation, and explain why model accuracy is not among them.