In May 2023, a federal judge in the Southern District of New York discovered that a legal brief filed in the case Mata v. Avianca cited six court decisions that did not exist. The attorney, Steven Schwartz, had used ChatGPT to research legal precedents. The system produced plausible-sounding case names, citations, and even quoted passages from fabricated judicial opinions. Schwartz submitted them to the court without verifying they were real. He was sanctioned with a $5,000 fine.[1]

The episode became a cautionary tale, but the deeper lesson often gets lost in the headlines. The problem wasn't that the AI made an error. The problem was that the AI's error was indistinguishable from competence. The fabricated cases sounded exactly like real cases. The citations were formatted correctly. The quoted language read like genuine judicial reasoning. Nothing in the output signaled "this is invented." The system produced text with the form of legal knowledge and none of the substance.

This is the Chinese Room's practical consequence. If a system manipulates symbols without understanding them, it has no mechanism for distinguishing true from false, real from invented, reliable from hallucinated. It produces plausible sequences of tokens. Plausibility and truth are different things, and a system without understanding cannot tell the difference.

The Confidence Without Calibration Problem

Two identical-looking documents side by side, one glowing with authentic golden light and one subtly hollow inside despite perfect exterior formatting, representing the indistinguishability of genuine and fabricated outputs
The form of competence substitutes for the substance of it. You can't tell from outside the room whether anyone inside understands.

Human experts calibrate their confidence to their knowledge. A doctor who isn't sure about a diagnosis says "I'm not sure." A lawyer who can't find a relevant case says "there may not be precedent here." A programmer who doesn't understand a system says "I'd need to look into this more." The ability to say "I don't know" is itself a form of understanding: you have to know the boundaries of your knowledge to know when you've reached them.

Language models produce confident text regardless of accuracy. They can't assess whether they know something, because they don't know anything in the sense that would permit self-assessment. They produce outputs that are statistically likely given the context. When the context suggests confidence is appropriate (a professional question expects a professional answer), the output is confident. When the correct answer is "I don't know," the model may still produce an answer, because producing an answer is what the statistical patterns reward.

This isn't a bug that better training will fix. It's a structural feature of systems that process symbols without understanding what those symbols refer to. A system that understood medicine could recognize when a question falls outside its competence. A system that manipulates tokens associated with medical language has no mechanism for recognizing the boundaries of its training distribution in the way a human recognizes the boundaries of their expertise.

Recent work on calibration and uncertainty quantification attempts to address this, and some progress has been made. But the fundamental challenge remains: a system without understanding can learn to output "I'm not sure" in contexts where uncertainty language is statistically expected, without actually experiencing or representing uncertainty.

Where Lack of Understanding Is Dangerous

The stakes of the trust problem vary enormously by domain. In some contexts, the absence of understanding in the system matters little. In others, it's the central risk.

High-stakes domains where understanding matters most share a common feature: errors have consequences that are difficult or impossible to reverse, and the ability to recognize uncertainty is critical.

Medical diagnosis. A system that produces a confident but incorrect diagnosis may cause a patient to receive wrong treatment or miss critical treatment. A physician who understands medicine recognizes ambiguous presentations, considers alternative explanations, and orders additional tests when uncertain. A system that produces medical text has no mechanism for genuine uncertainty about a patient's condition, only statistical patterns about when medical language tends to express uncertainty.

Legal advice. The Mata v. Avianca case illustrates this directly. Legal reasoning requires understanding what the law says, how it applies to specific facts, and where ambiguity exists. A system that produces legal text can generate plausible-sounding analysis while inventing the precedents it claims to rely on.

A gradient from green to red showing different levels of risk: casual documents on the safe green end, medical and legal decisions on the dangerous red end, with a human reviewer positioned as a checkpoint between the machine output and the high-stakes zone
Trust proportional to stakes: the higher the consequences, the more essential the human understanding layer becomes.

Engineering decisions. Code generated by AI may compile, pass tests, and function correctly under normal conditions while containing subtle flaws that a system without understanding of the domain's requirements couldn't recognize. Security vulnerabilities, edge cases that violate implicit assumptions, race conditions that only manifest under specific timing: these require understanding what the code is supposed to do and why, not just what token sequences are statistically likely.

Scientific claims. AI systems can produce text that reads like scientific reasoning while making claims unsupported by evidence. When researchers use AI to generate hypotheses, draft papers, or summarize literature, the outputs must be treated as drafts to verify, not conclusions to trust.

Where Lack of Understanding Is Acceptable

In lower-stakes contexts, the absence of understanding matters less because human oversight provides the understanding layer.

Drafting emails and documents. If a human reviews and edits the output before sending, the AI is a productivity tool, not an authority. The human provides the understanding; the AI provides speed.

Brainstorming and ideation. Generating options, suggesting approaches, and exploring possibilities don't require that the generator understand what it's producing. The human evaluates the suggestions using their own understanding.

Code suggestions with review. When a developer reviews, understands, and takes responsibility for AI-generated code, the system is a tool in the developer's hands. The danger arises when review is skipped because the code "looks right."

Summarization with verification. AI-generated summaries of long documents are useful when the human can spot-check accuracy. They're dangerous when treated as authoritative without verification.

The pattern across these cases: the human provides the understanding that the system lacks. The architecture should make this explicit rather than hiding it.

Automation Bias: The Trust Trap

A person nodding along to a stream of confident polished text from a machine, their own judgment fading into shadow as automation bias takes hold, the machine output glowing persuasively
Automation bias: confident, polished output triggers trust. The form of competence substitutes for the substance.

Research on automation bias, the tendency to over-rely on automated recommendations, has found that humans consistently defer to automated systems even when those systems are wrong and the human would have gotten the right answer independently.[2] The Georgetown Center for Security and Emerging Technology defines automation bias as "the tendency for an individual to over-rely on an automated system," noting that it "can lead to increased risk of accidents, errors, and other adverse outcomes when individuals and organizations favor the output or suggestion of the system, even in the face of contradictory information."[3]

AI systems that produce fluent, confident text amplify automation bias because their outputs look like they come from understanding. A poorly formatted or hesitant response triggers skepticism. A polished, confident, well-structured response triggers trust. The form of competence substitutes for the substance of it.

The Chinese Room makes this dynamic visible. The room produces perfect Chinese. An observer who doesn't know it's a room would trust the output as authoritative. The form is indistinguishable from genuine competence. The absence of understanding inside the room doesn't show in the output, which is exactly what makes it dangerous. You can't tell from outside the room whether anyone inside understands Chinese. And you can't tell from an AI's output whether it understood what it produced.

Calibrated Trust as a Design Principle

The practical response to the Chinese Room isn't to stop using AI systems. It's to design deployment architectures that account for the uncertainty about whether understanding exists inside the system.

Trust should be proportional to two factors: the consequences of error, and the availability of human verification.

Low consequences, easy verification: trust the output directly. An AI-generated email draft that you'll read before sending. A code suggestion in an IDE where you'll review the diff. A summarization of a document you've already read.

High consequences, available verification: treat the output as a draft that requires expert review. Medical AI that flags potential diagnoses for a physician to evaluate. Legal research AI whose citations a lawyer verifies before filing. Engineering designs that go through human review before implementation.

High consequences, no verification available: do not deploy without fundamental architectural changes. Autonomous systems making irreversible decisions without human oversight. AI providing advice directly to consumers on medical, legal, or financial matters without professional intermediation.

The principle is simple: keep the person who speaks Chinese in the loop. The system generates; the human who understands evaluates. The Chinese Room produces outputs; the understanding layer verifies them. When the understanding layer is removed, all that remains is symbol manipulation producing confident text with no guarantee of truth.

References

[1] Mata v. Avianca, Inc., No. 22-cv-1461 (S.D.N.Y. 2023). The court sanctioned attorneys Steven Schwartz and Peter LoDuca $5,000 for submitting fabricated case citations generated by ChatGPT. See CBS News, "A lawyer used ChatGPT to prepare a court filing. It went horribly awry," May 2023. https://www.cbsnews.com/news/lawyer-chatgpt-court-filing-avianca/ See also American Bar Association, "Use of ChatGPT for Research Leads to Bogus Cases, Sanctions," 2023. https://americanbar.org/content/aba-cms-dotorg/en/groups/litigation/resources/litigation-news/2023/use-chatgpt-research-bogus-cases-sanctions

[2] For a comprehensive review of automation bias in human-AI collaboration, see "Exploring automation bias in human–AI collaboration: a review and implications for explainable AI," Ethics and Information Technology, 2025. https://link.springer.com/doi/10.1007/s00146-025-02422-7

[3] Georgetown University Center for Security and Emerging Technology, "AI Safety and Automation Bias," 2024. https://cset.georgetown.edu/publication/ai-safety-and-automation-bias/