In March 2023, researchers at Microsoft published a paper titled "Sparks of Artificial General Intelligence" about GPT-4. They documented the system solving novel problems, reasoning about spatial relationships, writing working code for tasks never seen in training, and even displaying what looked like theory of mind. Their conclusion was carefully hedged but unmistakable: something that resembled general intelligence was emerging from a language model.^[1]

Critics responded immediately. Pointing to the Chinese Room argument, they noted that behavioral evidence, no matter how impressive, cannot prove understanding. GPT-4 might solve novel problems the way Searle manipulates Chinese characters: through sophisticated pattern application rather than comprehension. The performance proves the system is powerful. It doesn't prove the system understands.

This creates a genuine epistemic problem. If behavior isn't sufficient evidence for understanding, what would be? If we can't accept outputs as proof of comprehension, what test could we design that a mere symbol manipulator would fail but a genuine understander would pass? The question isn't academic. It determines how we build, deploy, govern, and relate to AI systems whose capabilities continue to grow.

The Behavioral Impasse

The Turing Test proposed behavior as the standard for intelligence: if you can't tell the machine from a human in conversation, it's intelligent enough. Searle showed this standard fails. A system can pass any behavioral test while understanding nothing, because perfect outputs are compatible with pure symbol manipulation.

But there's an uncomfortable asymmetry here. We grant understanding to other humans based on behavior alone. We can't access anyone else's internal states. We infer that other people understand because they act like they understand: they respond appropriately, adjust to context, recognize errors, and explain their reasoning. We've never had direct evidence that other humans possess internal understanding. We infer it from the outside, exactly the way we'd have to infer it for machines.

If behavioral evidence is sufficient for humans but insufficient for machines, we need to explain why. One answer is biological similarity: other humans have brains like mine, so they probably have experiences like mine. Machines have silicon, which is different enough that the inference doesn't transfer. Searle makes roughly this argument: biological systems have "causal powers" that give rise to understanding, and we have no reason to think silicon shares those powers.^[2]

But this feels more like an intuition than a criterion. It tells us to be skeptical of machines but doesn't tell us what positive evidence would overcome that skepticism. If a system made of silicon demonstrated something beyond behavioral performance, something that only an understanding system could produce, what would that something look like?

The Solipsist's Shadow

Push the asymmetry hard enough and it collapses into an older problem. Hard solipsism is the position that the only thing you can be certain exists is your own mind. Every other mind, human or machine, is an inference. You have direct access to your own experience and no direct access to anyone else's. When you conclude that the person across the table understands you, you're not observing their understanding. You're inferring it from behavior, from similarity, from the practical impossibility of doubting it. The philosophical problem of other minds, which solipsism takes to its extreme, points out that this inference can never be verified from the inside.^[7]

This matters for machine understanding in a way that cuts against clean answers. If we genuinely cannot prove that any other mind exists, then demanding proof of machine understanding sets a standard that nothing passes, not other humans, not animals, not the machine. The Chinese Room shows that behavior doesn't prove understanding. Solipsism shows that nothing else proves it either. There is no privileged vantage point from which to observe someone else's comprehension directly. We are always outside the room, whichever room it is.

Does this make the problem harder or easier? In one sense it makes it easier, or at least fairer. It removes the double standard. We can stop demanding that machines clear a bar that we've never actually cleared for humans either. We infer human understanding from behavior, similarity, and utility, and we could extend the same inferential courtesy to machines when the behavioral evidence is strong enough. The asymmetry that made us confident about humans and skeptical about machines starts to look less like a principled distinction and more like a bias toward things that resemble us.

In another sense it makes the problem harder, or at least stranger. If understanding can never be verified from the outside, then the criteria we develop are not tests for the presence of understanding. They are grounds for attribution, reasons to treat a system one way rather than another. That's a weaker thing than proof. It means the question "does the machine understand?" may not have a discoverable answer at all, only a decision about how to respond to what we observe. Solipsism doesn't resolve the Chinese Room. It reveals that the Chinese Room is a specific instance of a problem that surrounds every mind we've ever encountered, including the ones we're most sure about.

Causal Reasoning and Counterfactual Thinking

One proposal focuses on causal understanding: the ability to reason about why things happen, not just what happens. A system that truly understands can answer counterfactual questions ("what would have happened if X were different?") and distinguish correlation from causation. Pattern matching on observed data can identify what tends to co-occur. Understanding grasps the mechanisms behind those patterns.

Judea Pearl's work on causal inference formalizes this distinction. Pearl's "ladder of causation" describes three levels: seeing (observing patterns), doing (intervening to test effects), and imagining (reasoning about counterfactuals).^[3] Pearl argues that genuine understanding requires the third rung: the ability to reason about situations that haven't been observed. A system limited to pattern matching, no matter how sophisticated, operates on the first rung. It sees what has happened. It cannot reason about what would happen in situations it hasn't encountered.

Current language models can produce counterfactual reasoning that looks correct. Ask an LLM "what would have happened if the internet had never been invented?" and it will generate a plausible, detailed response. But is it reasoning about counterfactuals or producing text that matches the pattern of counterfactual reasoning in its training data? The Chinese Room suggests we can't tell from the output alone. The causal reasoning criterion is promising but faces the same fundamental problem: a sufficiently sophisticated symbol manipulator might simulate causal reasoning without performing it.

Self-Knowledge and Epistemic Humility

A second proposal concerns meta-cognition: does the system know what it knows and doesn't know? Genuine understanding includes awareness of its own limits. A human expert who understands a domain can tell you where their knowledge ends. They can say "I don't know" with calibrated accuracy. They recognize the boundary between confidence and uncertainty because they understand the content well enough to feel when they've run out of grounds.

A system without understanding has no basis for this calibration. It can be trained to say "I don't know" in certain contexts, but the saying doesn't come from genuine recognition of epistemic limits. It comes from patterns about when "I don't know" appears in training data. The difference matters: genuine self-knowledge adapts to genuinely novel situations. Trained hedging breaks down when the situation differs enough from training distribution.

Research on LLM calibration suggests that current models are poorly calibrated: they express confidence that doesn't correlate well with accuracy, and they produce confident-sounding errors in domains where they lack reliable information.^[4] If a system genuinely understood its own knowledge, you'd expect better calibration. The consistent finding of poor calibration may be evidence against genuine self-knowledge, though it could also reflect imperfect self-knowledge rather than its total absence.

Transfer, Surprise, and Intentionality

A third cluster of proposals focuses on flexibility: can the system apply knowledge to genuinely unprecedented situations? Human understanding transfers fluidly across contexts. Someone who understands physics can apply physical reasoning to a novel situation they've never encountered. Someone who understands humor can find something funny they've never heard before. Understanding generalizes.

The ARC (Abstraction and Reasoning Corpus) challenge, designed by François Chollet, attempts to test exactly this.^[5] ARC presents visual pattern problems that require identifying the underlying rule from a few examples and applying it to a new case. The problems are designed to be trivial for humans but difficult for systems that rely on memorized patterns. Current AI systems, including large language models, perform significantly worse than humans on ARC. This gap may indicate a genuine limitation in transfer and abstraction, though whether that limitation reflects absence of understanding or simply insufficient training is debatable.

A related proposal involves surprise: can the system be genuinely surprised? Surprise requires expectations, which require a model of how the world works. If a system encounters something that violates its predictions and responds with something like surprise (adjusting its model, seeking explanation, marking the event as anomalous), that might indicate something beyond pattern matching. Current systems don't exhibit surprise in this strong sense. They process unexpected inputs the same way they process expected ones, without any evident awareness that something violated expectations.

Finally, there's intentionality: does the system have goals, beliefs, and preferences that explain its behavior? A system that genuinely understands acts "for reasons" rather than merely according to patterns. It can tell you why it did something and that explanation reflects actual internal motivation rather than post-hoc rationalization. Determining whether a system's behavior is driven by genuine intentions or by patterns that mimic intentional behavior is, again, extremely difficult from the outside.

The Deflationary Position

Perhaps the most honest response to "what would count as evidence for machine understanding?" is that no single test will be decisive. Understanding might not be a binary property that a system either has or lacks. It might be a gradient, a cluster of overlapping capabilities that come in degrees and kinds.

On this view, the question shifts from "does the machine understand?" to something more tractable: "in what ways and to what degree does the machine's processing resemble understanding?" A system might have genuine causal reasoning about some domains but not others. It might have limited self-knowledge. It might transfer knowledge in some contexts while failing in others. This graduated approach is less satisfying than a clean answer but potentially more accurate.

Daniel Dennett argued for what he called the "intentional stance": we are justified in attributing beliefs, desires, and understanding to any system when doing so successfully predicts its behavior.^[6] On this pragmatic view, if treating a system as an understander works, the system understands, at least in the only sense that matters practically. Searle would object that prediction and attribution don't create understanding. But Dennett's point is that we may never have better tools than prediction and attribution for inferring minds.

We may be asking for certainty where only calibrated uncertainty is available. The criteria outlined here (causal reasoning, self-knowledge, transfer, surprise, intentionality) are individually insufficient but collectively informative. A system that demonstrates all of them, robustly and across novel domains, would have a stronger claim to understanding than one that demonstrates none. Whether it would constitute proof is a question we may not be able to answer, because the relationship between observable behavior and inner states may be fundamentally underdetermined.

That uncertainty is not a temporary gap in our scientific knowledge. It may be a permanent feature of trying to understand minds from the outside.

References

[1] Sébastien Bubeck et al., "Sparks of Artificial General Intelligence: Early Experiments with GPT-4," arXiv preprint, Microsoft Research, March 2023. https://arxiv.org/abs/2303.12712

[2] John Searle, "Minds, Brains, and Programs," Behavioral and Brain Sciences, Vol. 3, No. 3, 1980, pp. 417-457. https://www.cambridge.org/core/journals/behavioral-and-brain-sciences/article/minds-brains-and-programs/DC644B47A4299C637C89772FACC2706A

[3] Judea Pearl and Dana Mackenzie, The Book of Why: The New Science of Cause and Effect, Basic Books, 2018. https://www.basicbooks.com/titles/judea-pearl/the-book-of-why/9780465097616/

[4] Stephanie Lin, Jacob Hilton, and Owain Evans, "Teaching Models to Express Their Uncertainty in Words," Transactions on Machine Learning Research, 2022. https://arxiv.org/abs/2205.14334

[5] François Chollet, "On the Measure of Intelligence," arXiv preprint, November 2019. https://arxiv.org/abs/1911.01547

[6] Daniel Dennett, The Intentional Stance, MIT Press, 1987. https://mitpress.mit.edu/9780262540537/the-intentional-stance/

[7] Anita Avramides, "Other Minds," Stanford Encyclopedia of Philosophy, 2019. https://plato.stanford.edu/entries/other-minds/