Case 3: The Language Barrier — Innovare AI Analysis

As you read — hold this question

If a model generates text by predicting what comes next rather than knowing whether the output is correct, when does using one become a liability?

55%

faster at coding tasks — what GitHub measured when developers used Copilot. Copilot doesn’t know what the code does. It predicts what code looks like after the prompt you wrote — and that’s enough.

This is the central paradox of large language models. They operate entirely through prediction — given this text, what text is most likely to follow? They do not “read” in any meaningful sense. They do not “know” things. And yet they write working code, translate fluently between languages, summarise complex documents, and pass professional licensing exams. Understanding how this is possible — and where it breaks — is the key to using these systems well.

The evolution — from bag-of-words to transformers

Quiz: LLMs

60 years of trying to make machines understand language

Era	Approach	What it could do	What broke it
1950s–80s Rule-based NLP	Hand-coded grammar rules and dictionaries	Parse simple sentences, answer narrow questions in constrained domains	Language is too ambiguous and contextual for rules. “I saw the man with the telescope” — who had the telescope?
1990s–2010s Statistical NLP	Count word co-occurrences in large corpora; learn statistical patterns	Machine translation (Google Translate v1), spam filtering, sentiment analysis	No context — each word treated independently. Couldn’t handle long-range dependencies.
2013–2016 Word embeddings	Map words to dense vectors preserving semantic relationships (Word2Vec, GloVe)	King − Man + Woman ≈ Queen. Semantic similarity. Better translation.	Static embeddings — the word “bank” has the same vector regardless of context (river bank vs. financial bank).
2017–present Transformers	Attention mechanism: every token can attend to every other token, weighted by relevance	Contextual understanding, generation, translation, summarisation — across all language tasks simultaneously	Computational cost scales quadratically with sequence length. Requires enormous compute to train.

The mechanism — how transformers actually work

Attention: the key architectural idea

Before transformers, sequence models processed text word by word — each word could only attend to recent context. Transformers removed this constraint. Every token can attend to every other token in the sequence simultaneously, with attention weights determined by learned relevance.

What attention does

When processing “bank” — attends to nearby words to determine which sense is meant

When predicting a pronoun — attends to the noun it refers to, even 500 tokens earlier

Multiple attention heads capture different types of relationships simultaneously

Why it scales

Entire sequence processed in parallel — no sequential bottleneck

Can be trained on the entire internet without any human labels (self-supervised)

Performance keeps improving as model size and data volume increase — emergence

Applied — GitHub Copilot

What happens when a transformer writes code

GitHub Copilot — launched 2021, built on OpenAI Codex (a GPT model fine-tuned on public GitHub repositories) — generates code completions from natural language comments and partial code. In a controlled study of 95 professional developers, Copilot users completed tasks 55% faster.

What Copilot demonstrates

Principle illustrated

Generates syntactically correct code in 10+ languages from a comment — without being told which language

Prediction at scale

55% faster task completion in controlled trials — commercially significant

Utility without understanding

Generates plausible but incorrect code — bugs that look correct on first read

Hallucination risk

Reproduces insecure coding patterns from training data (SQL injection, buffer overflow)

Training data leakage

Cannot explain why the code it generated is correct

No comprehension

The Principle — prediction ≠ comprehension, and that’s the point

Copilot doesn’t know what code does. It knows what code that follows a comment like “// parse JSON and handle errors” typically looks like in GitHub repositories. The same mechanism that produces the 55% productivity gain also produces the confident hallucination. These are not separate phenomena — they are the same mechanism applied to in-distribution and out-of-distribution cases.

Take this away

LLMs are force multipliers for tasks where the human in the loop can verify output. They are liabilities where they cannot. The same prediction mechanism that produces the 55% productivity gain produces the confident hallucination.

← Case 2: The Deep Revolution

Case 4: The Creative Machine →

Quick recall — without looking back

Test yourself on this case

Question 1 of 3

Before transformers, what were the two main limitations of statistical NLP approaches — and how did attention solve them?

Statistical NLP (1990s–2010s) treated each word independently — no context. Word embeddings (2013–2016) captured semantic similarity but were static — “bank” had the same vector regardless of context. Transformers solved both: the attention mechanism lets every token attend to every other token, weighted by relevance. “Bank” in “river bank” attends differently than in “financial bank.” Attention captures long-range dependencies that earlier approaches could not.

Question 2 of 3

GitHub Copilot showed 55% faster task completion. What mechanism produces that gain — and what produces the hallucination risk?

The same mechanism produces both: prediction at scale. Copilot knows what code following a given comment typically looks like in GitHub repositories. In-distribution (common coding patterns): prediction produces correct, useful code quickly — 55% productivity gain. Out-of-distribution (novel logic, edge cases): the same mechanism produces plausible-looking but incorrect code. The model cannot verify its output — it generates text that looks correct, not text that is correct.

Question 3 of 3

What is the business implication of the principle “prediction ≠ comprehension”?

LLMs are force multipliers for tasks where the human in the loop can verify output — writing, summarising, drafting code where an engineer reviews before shipping. They are liabilities where verification is absent or difficult: legal documents where an unqualified person relies on them, medical advice where the patient cannot assess accuracy, code in critical systems where a hallucination looks plausible. The same model that produces 55% productivity gains becomes a risk when no one is checking the output.

Sources

Transformer: Vaswani, A. et al. (2017). Attention is all you need. NeurIPS 2017.
GitHub Copilot study: Peng, S. et al. (2023). The impact of AI on developer productivity: evidence from GitHub Copilot. arXiv:2302.06590.
Copilot security: Pearce, H. et al. (2022). Asleep at the keyboard? Assessing the security of GitHub Copilot’s code contributions. IEEE S&P 2022.
Word2Vec: Mikolov, T. et al. (2013). Distributed representations of words and phrases and their compositionality. NeurIPS 2013.
Course material: BUSN9049 Module 2. Flinders University, 2026.