ELIZA and the Illusion of Understanding

Joseph Weizenbaum's 1966 chatbot fooled everyone — including people who knew it was a program. The story of ELIZA and what its reception revealed.

There is a thought experiment, famous in philosophy of mind, about what it would mean to truly understand a language. Suppose you are given a book of rules — a very large book, covering every possible string of Chinese characters and specifying, for each one, what string of Chinese characters to write in response. You receive a string of characters through a slot in a box you are sitting inside. You look up the input in your book. You copy out the response. You pass it back through the slot.

From outside the box, the exchange looks exactly like a conversation with someone who speaks fluent Chinese. The inputs are appropriate. The responses are appropriate. Every criterion of conversational competence is met.

But you do not understand Chinese. You have never understood Chinese. You are following rules about the shapes of symbols without any comprehension of what those symbols mean.

This is John Searle’s Chinese Room, published in 1980. It is one of the most discussed thought experiments in the philosophy of mind. It is also, in a very specific and very important sense, a description of ELIZA.

ELIZA did not understand anything. It followed rules about the shapes of symbols — about which patterns of text to respond to, and with what. The responses were appropriate. The people on the other side of the slot believed they were understood. They were not.

What ELIZA revealed, and what the Chinese Room formalised, is a question that the entire history of AI has been trying to answer: can you have understanding without understanding? Can a system produce all the outputs of comprehension without any of the experience that comprehension feels like from the inside? And if you can — does it matter?

Before ELIZA: The Language Problem

Natural language — the ordinary, everyday language that humans use to communicate — was recognised as a central challenge for AI almost from the beginning of the field. Turing, in his 1950 paper, had proposed conversational ability as the test of machine intelligence. McCarthy, in the original Dartmouth proposal, had listed language understanding as one of the seven core problems the conference would address. The ability to communicate in natural language was, for the founders of AI, so obviously central to intelligence that it served as both a goal and a benchmark.

But the problem proved much harder than it had appeared. The first attempts at machine translation — translating text from one language to another automatically — had collapsed under the weight of linguistic complexity, as the ALPAC Report of 1966 had documented. The rules governing natural language were not the explicit, learnable rules of formal logic. They were vast, implicit, contextually dependent, and deeply embedded in knowledge about the world that no program had.

Language was not just grammar. Grammar told you whether a sentence was syntactically well-formed, but it could not tell you what the sentence meant, whether it was true, how it related to other sentences, what a speaker intended by uttering it, or what a listener needed to know to understand it. These were questions about meaning, inference, world knowledge, and pragmatics — questions that were, in 1960 or 1965, far beyond the reach of any computational approach.

The field of natural language processing — the computational study of language — was developing in the 1960s, but it was developing slowly. Researchers were making progress on specific, well-defined sub-problems: parsing — analysing the grammatical structure of sentences — was advancing. Semantic representation — encoding the meaning of sentences in formal logical structures — was being developed. But the full problem of understanding natural language in the rich sense that humans understood it remained intractable.

Into this landscape, Joseph Weizenbaum introduced ELIZA. The program did not attempt to understand language in any deep sense. It did not parse sentences, did not construct semantic representations, did not maintain a model of the conversation’s content. It simply recognised patterns in the user’s input and generated responses based on those patterns. The sophistication of the language understanding was minimal. The psychological impact on users was, as Weizenbaum discovered to his horror, enormous.

The Linguistics of ELIZA: What the Patterns Were Doing

To understand what ELIZA was actually doing with language — and what that tells us about language and understanding — it helps to look at the specific linguistic mechanisms the program used.

ELIZA’s pattern-matching worked on several levels, exploiting specific features of the English language that made it possible to generate contextually appropriate-seeming responses with very limited actual analysis.

The most basic mechanism was keyword spotting. ELIZA scanned the user’s input for words from a predefined list of keywords — words like “mother,” “father,” “dream,” “feel,” “am,” “because,” “perhaps” — and selected a response template associated with the highest-priority keyword found. The keyword told ELIZA something about the topic of the input, and the response template was designed to be appropriate for that topic.

The second mechanism was pronoun flipping. When generating a response that incorporated part of the user’s input, ELIZA would flip first-person pronouns to second-person: “I” became “you,” “my” became “your,” “me” became “you.” This simple transformation allowed ELIZA to reflect the user’s own words back to them in a form that sounded like a response: “I am feeling anxious about my job” became “Why do you feel anxious about your job?” The transformation was purely syntactic — a simple find-and-replace — but its effect on users was powerful.

The third mechanism was decomposition and reassembly. For certain keywords, ELIZA had patterns that decomposed the user’s input into components and reassembled them into a response. A pattern for “I am [adjective]” might decompose the input “I am feeling overwhelmed” into [adjective] = “feeling overwhelmed” and reassemble it as “How long have you been feeling overwhelmed?” Again, the process was syntactic pattern-matching rather than semantic analysis, but the output looked like an intelligent question about the content of what had been said.

The fourth mechanism was the fallback. When no keyword was found, ELIZA would fall back on a set of generic responses — “Please go on,” “Tell me more,” “That is very interesting,” “I see” — that could follow almost anything without being obviously inappropriate. These responses were contextually appropriate in the weakest possible sense: they were the responses of a silent and attentive listener, inviting continuation without requiring any actual understanding.

What all these mechanisms had in common was that they were operating on the surface form of language — on the shapes and positions of words — without any engagement with their meaning. ELIZA did not know what “anxious” meant, or what “job” meant, or how these two things might be related, or why a person’s job might cause anxiety. It knew that “anxious” was a word that could appear in certain positions in sentences that warranted certain responses, and it produced those responses.

And yet the responses worked. They worked not because they were meaningless — they were not, from the user’s perspective, meaningless — but because language comprehension is, partly, a matter of projection. Users brought their own understanding of the conversation to the exchange, filled in the gaps that ELIZA’s limited processing left, interpreted the responses as meaningful because responses in that form, from a human interlocutor, would be meaningful.

The Psychological Mechanism: Why Projection Is So Powerful

The ELIZA effect — the tendency of humans to project understanding, empathy, and intent onto conversational systems that have none — is not a quirk or a flaw in human cognition. It is an expression of one of the most powerful and most fundamental cognitive capacities we have: the capacity for mentalising — attributing mental states to other entities.

Mentalising, also called Theory of Mind, is the cognitive ability to understand that other entities have mental states — beliefs, desires, intentions, emotions — that are distinct from one’s own, and to use that understanding to predict and explain their behaviour. It is the capacity that makes human social life possible. Without the ability to understand what other people want, believe, intend, and feel, we could not cooperate, could not communicate, could not build relationships, could not navigate the complex social environment that human life requires.

Mentalising is activated by certain kinds of stimuli. It is activated when we observe behaviour that looks intentional — that seems directed toward a goal, that seems responsive to information about the world. It is activated when we encounter language — when something communicates using the same symbolic medium that humans use to express their mental states. It is activated by the features of language use that signal understanding: responsiveness to what was said, apparent attention to the content, the form of empathetic engagement.

ELIZA produced all of these signals. It responded to what the user said — or appeared to. Its responses were in the same linguistic medium as the user’s inputs. They had the form of engaged attention — questions that seemed to follow from what had been said, expressions of interest that mimicked the empathetic responsiveness of a skilled therapist. These signals activated the mentalising capacity of ELIZA’s users as surely as if they had been produced by a genuine human mind.

The activation was not a mistake, from the perspective of the user’s cognitive system. The rules of thumb that activated mentalising — responsive language use is a reliable indicator of a mind behind it — were correct throughout the evolutionary history in which those rules of thumb developed. Responsive language use was produced by minds with understanding, because nothing else produced responsive language use. ELIZA was the first thing that ever produced responsive language use without the understanding. The cognitive systems that ELIZA’s users brought to the interaction had never encountered anything like it before, and they had no mechanism for detecting the anomaly.

This is the deep lesson of ELIZA. It is not that humans are gullible, or easily fooled, or deficient in critical thinking. It is that humans are well-adapted to a world in which certain signals reliably indicate certain things — and that in a world where technology can produce those signals without the things they have always indicated, the adaptation becomes a vulnerability.

Weizenbaum’s Question: What Does Understanding Require?

Joseph Weizenbaum was not primarily a philosopher. He was a computer scientist who built a program and was disturbed by what happened when people used it. But his response to the disturbance led him to philosophical questions that he pursued with unusual rigour for the rest of his career.

The central question he arrived at was: what is the difference between understanding and the perfect simulation of understanding?

This question is not idle. If the simulation is perfect — if every output of the system is identical to what would be produced by genuine understanding — then what, if anything, is missing? What is the thing that genuine understanding has that perfect simulation lacks?

The philosophical tradition provides several candidate answers.

One answer is consciousness — the subjective experience of understanding, the “what it is like” to comprehend something. Understanding, on this view, is not just a functional capacity — the ability to produce appropriate outputs from appropriate inputs. It is an experience, a quality of mental life, something it feels like to grasp. ELIZA had no such experience. It processed inputs and generated outputs without any accompanying inner life. The simulation was functional but not experiential. Something was missing.

But the consciousness answer raises the question of how we could ever know whether any system had consciousness. We cannot directly observe another person’s inner experience. We infer it from their behaviour and their reports. If a system’s behaviour and reports are identical to those of a conscious being — if it says “I feel” and acts as if it feels — on what grounds do we deny that it has experience? The consciousness answer seems to require access to the system’s inner states that we do not have and may not be able to have.

Another answer is intentionality — the property that mental states have of being about something, of referring to things in the world. ELIZA’s outputs were not about anything — the word “anxious” in ELIZA’s output referred to nothing, because ELIZA had no model of what anxiety was, no connection between the symbol and the psychological reality it named. The symbol manipulation was happening in a semantic vacuum, with no genuine reference to the world.

This is Searle’s answer — the insight that drives the Chinese Room thought experiment. The person in the Chinese Room is manipulating symbols without any connection between those symbols and what they mean. The symbols have meaning — they are meaningful Chinese characters — but not for the symbol manipulator. The manipulation is purely formal, purely syntactic. The semantic dimension — the meaning, the reference, the understanding — is absent.

A third answer focuses on knowledge — specifically, on the vast background knowledge that genuine language understanding requires. Understanding a sentence is not just processing its grammatical structure or recognising its keywords. It requires knowing what the words mean — not just their dictionary definitions but their connections to other concepts, their implications, the situations they apply to and the situations they do not. It requires knowing about the world in ways that language constantly presupposes. Understanding “I am anxious about my job” requires knowing what anxiety is, what jobs are, how people relate to their jobs, why jobs might cause anxiety, how anxiety manifests, what kinds of things people who are anxious typically want or need. ELIZA had none of this. Its keyword-matching operated without any of the background knowledge that genuine understanding required.

Each of these answers captures something real. Genuine understanding, whatever it is, involves consciousness, intentionality, and knowledge in ways that ELIZA’s pattern-matching did not. But none of the answers is fully satisfying, and the question of which of them is most fundamental — and what their relationship is to each other — remains genuinely open.

The Language Models Enter: A New Version of the Old Question

For several decades after ELIZA, the question Weizenbaum raised was primarily of philosophical interest. AI systems were not good enough to make the question practically pressing. A chess program that beat Kasparov was impressively narrow but clearly not understanding in any interesting sense. An expert system that diagnosed bacterial infections was impressively domain-specific but clearly not understanding in any interesting sense.

Then language models arrived. And then large language models arrived. And the question became practically pressing again in a way that it had not been since ELIZA.

Large language models — the transformer-based systems that power ChatGPT, Claude, Gemini, and their competitors — produce language outputs of a quality that is qualitatively different from anything that preceded them. They are not doing keyword matching. They are not following simple pattern-matching rules. They are processing language with a sophistication that, in many contexts, produces outputs that are indistinguishable from those of a knowledgeable, thoughtful human being.

They answer questions accurately. They explain concepts clearly. They engage in extended conversations that maintain coherent topics and build on previous exchanges. They write essays, code programs, compose poetry, analyse arguments, tell stories. The range and quality of their language capabilities is astonishing.

And yet the question raised by ELIZA has not gone away. It has sharpened.

When a large language model explains quantum mechanics clearly and accurately, does it understand quantum mechanics? When it expresses sympathy for a user’s difficult situation, does it feel anything? When it engages in a philosophical argument, is it reasoning, or producing text that looks like reasoning?

These are not rhetorical questions. Serious researchers, philosophers, and the people who build these systems disagree about the answers. The disagreement is not primarily about facts — people with access to the same systems and the same evidence disagree. It is about concepts: what does “understand” mean, what would it take for a machine to satisfy that concept, and whether the concept as humans naturally apply it can meaningfully extend to systems that are as different from human beings as large language models are.

The Spectrum of Positions: From Symbol Pushers to Genuinely Thinking Machines

The debate about whether large language models understand has produced a range of positions that can be mapped along a spectrum.

At one end is the deflationary position: large language models are sophisticated symbol manipulators, nothing more. They predict the next token in a sequence based on statistical patterns in their training data. The outputs that look like understanding are the outputs of a very large, very complex pattern-matching system — impressive, useful, but not understanding in any philosophically interesting sense. This is the direct descendant of Weizenbaum’s position on ELIZA, scaled up.

The deflationary position points to the failures that language models reliably exhibit. They make errors that no genuine understanding would permit. They confidently assert false information — “hallucinate,” in the current terminology — in ways that suggest they are producing plausible-sounding text rather than reasoning from actual knowledge. They fail at tasks that require systematic reasoning — certain logical puzzles, certain mathematical problems, certain common-sense inferences — in ways that reveal the limits of pattern matching as a mechanism. They are inconsistent across contexts in ways that suggest their outputs depend more on surface features of the input than on stable underlying knowledge.

At the other end is the expansionist position: if a system produces all the outputs of understanding — accurate answers, coherent explanations, appropriate responses — then it has understanding, or something so close to understanding that the distinction is philosophically unimportant. This position points out that the criteria we use to attribute understanding to other humans are behavioural — we infer understanding from behaviour, not from direct observation of inner states. If we applied the same behavioural criteria to language models that we applied to humans, we would attribute understanding to them.

The expansionist position has difficulty with the failures. A system that confidently asserts false information does not behave like a system with understanding of the relevant domain. A system that fails at systematic reasoning in ways that a genuine understander would not fail is revealing limits that the concept of understanding does not have. The behavioural criteria for understanding are not just “produces plausible-sounding output” — they include accuracy, consistency, the ability to reason systematically, and robustness to variations in input.

Between these poles is a range of intermediate positions that try to capture the genuine complexity of the situation. Perhaps language models have something that deserves to be called understanding, but a form of understanding different from human understanding — statistical rather than model-based, distributed rather than localised, reliable in some respects and unreliable in others. Perhaps the concept of understanding is not sharp enough to apply cleanly to systems as different from humans as language models are, and we need new concepts rather than a yes-or-no answer to the old question. Perhaps the question of whether language models understand is less important than the question of what they can and cannot do, and the philosophical question should be suspended while the empirical question is pursued.

The Practical Dimension: When the Illusion Has Consequences

Whatever the philosophical status of language model “understanding,” the practical question of what consequences follow from the illusion of understanding is urgent and concrete.

Weizenbaum worried that the illusion of understanding produced by ELIZA would lead people to use it as a substitute for genuine human connection — that the simulation of empathetic attention would be offered to lonely and distressed people in place of the real thing, with genuinely harmful consequences. This worry was prescient, and it has become more acute as language models have become more capable.

AI companions — applications specifically designed to provide social and emotional support through conversation with AI systems — are now commercially available and widely used. They are used by people who are lonely, who are anxious, who are grieving, who struggle with social connection in various ways. The people who use them often report genuine benefit — reduced loneliness, increased emotional regulation, a sense of being heard and understood. The benefit is real, even if the understanding that produces it is not.

The worry is that these benefits may come at a cost — that the availability of a good-enough simulation of human connection may reduce the motivation to seek genuine human connection, that the simulation may mask the underlying need without addressing it, that reliance on AI companions may be a form of avoidance that makes the underlying problem worse over time. This is an empirical question that current research has not fully resolved. But Weizenbaum’s instinct — that the simulation of care is not a substitute for care — is a serious hypothesis that deserves serious investigation.

The illusion of understanding also has consequences in high-stakes decision-making contexts. AI systems that appear to understand are increasingly being used to advise on medical, legal, and financial decisions — to help physicians interpret medical images, to assist lawyers with research and argumentation, to advise financial analysts on investment decisions. The danger is that the systems’ apparent confidence and apparent coherence may lead users to over-rely on them — to treat their outputs as reliable in cases where they are not, to fail to check outputs that seem authoritative, to reduce the critical scrutiny that the outputs require.

This is the problem that Weizenbaum identified as the “judgment-calculation” distinction. Calculation — applying explicit rules to well-defined inputs — was something computers could do reliably. Judgment — integrating complex, contextual, value-laden considerations to reach a conclusion — was something that required human understanding that computers did not have. The danger of appearing to understand was that the appearance might substitute for the reality in contexts where the reality was essential.

SHRDLU: The Deeper Understanding That Was Not General

While ELIZA was demonstrating the disturbing power of apparent understanding without genuine understanding, another system was demonstrating a different and in some ways more interesting phenomenon: genuine-seeming understanding that was, on close examination, entirely dependent on artificial constraints.

SHRDLU, developed by Terry Winograd at MIT between 1968 and 1970, was a natural language understanding program that could conduct extended conversations about a simplified “blocks world” — a virtual environment containing coloured blocks, pyramids, and boxes of different sizes and colours, which a virtual robot arm could manipulate.

Within this world, SHRDLU could understand remarkably sophisticated language. It could respond correctly to commands like “Pick up a big red block” and “Find a block which is taller than the one you are holding and put it into the box.” It could answer questions like “Is there a large block behind a pyramid?” and “Had you touched any pyramid before you put the green one on the little cube?” It could handle pronominal reference, maintaining the conversational context and resolving pronouns to their correct antecedents.

SHRDLU looked, from a description, like a genuine natural language understanding system — something far more sophisticated than ELIZA’s keyword-matching. And in its domain, it was. The understanding was genuine, within the blocks world.

But the understanding was entirely dependent on the extreme simplicity and artificial structure of the blocks world. Move SHRDLU outside that world — ask it about anything other than blocks and pyramids, engage it in any conversation that referenced real-world knowledge — and it failed completely. The apparent sophistication of its language understanding was a consequence of the extreme poverty of its world, not of any general capacity for understanding.

Winograd himself drew the most important conclusion from SHRDLU’s limitations. His subsequent work with Fernando Flores led him to argue that genuine language understanding required being embedded in the world in a way that SHRDLU was not — that understanding was not primarily a matter of symbolic processing but of the practical engagement with the world that language made possible. The blocks world was too simple to support genuine understanding because it was too impoverished to support the kinds of practical engagement from which genuine understanding grew.

This argument — that understanding was grounded in practice and embodiment, not in symbol manipulation — was one of the most important critiques of the symbolic AI tradition. It anticipated, in philosophical terms, the arguments that would eventually be made in favour of learning-based approaches to AI: that the knowledge required for genuine understanding could not be encoded symbolically by programmers, but had to be acquired through interaction with the world.

The Chinese Room Revisited: What Searle Got Right and Wrong

John Searle’s Chinese Room thought experiment, published in 1980 as a response to the Turing Test and as an argument against computational theories of mind, is the most famous philosophical intervention in the ELIZA debate and deserves careful attention.

Searle’s argument was simple in structure: symbol manipulation, no matter how sophisticated, cannot by itself produce genuine understanding. The person in the Chinese Room manipulates symbols correctly — produces the right outputs from the right inputs — but has no understanding of what the symbols mean. If a computer is just a symbol manipulator, then no matter how powerful the computer, it cannot understand. It can only simulate understanding.

This argument identifies something real and important. The gap between syntactic processing — the manipulation of symbols according to formal rules — and semantic content — the connection between symbols and what they mean — is real. ELIZA had syntax without semantics. The Chinese Room formalises why this matters.

But Searle’s argument has important limitations.

The systems reply: the argument establishes that the person in the Chinese Room does not understand Chinese. But understanding is not necessarily a property of individual components — it might be a property of systems. The whole system — the person, the rule books, the box — might understand Chinese in a sense that no individual component does, just as the brain as a whole understands things that no individual neuron does. Searle’s argument shows that understanding cannot be identified with any single component of a complex system. It does not show that understanding cannot be an emergent property of the system as a whole.

The brain simulator reply: suppose the program being run in the Chinese Room was a perfect simulation of the neural processes of a fluent Chinese speaker’s brain. Every neuron firing, every connection strengthened or weakened, exactly simulated. Would Searle say that this system did not understand Chinese? If yes, he seems to be committed to the view that biological neurons have some special property — some understanding-generating magic — that silicon does not. What is that property?

The robot reply: suppose the system was embodied in a robot that could interact with the world — that could perceive objects, manipulate them, navigate environments, engage with the full richness of embodied experience. Would the additional grounding of the symbols in physical interaction change the status of the understanding? Many philosophers think it would — that grounding symbols in physical interaction with the world is precisely what closes the gap between syntax and semantics.

These replies do not definitively refute Searle’s argument. The argument is not definitively refuted. But they reveal that the argument is more complex than it first appears, and that the gap between symbol manipulation and genuine understanding is a genuine philosophical puzzle that has not been resolved by either the Chinese Room or its critics.

What ELIZA Did to Natural Language Processing

The ELIZA story had a specific and somewhat paradoxical effect on the field of natural language processing — the subfield of AI most directly concerned with the questions ELIZA raised.

On one hand, ELIZA demonstrated that apparent conversational competence could be achieved with very simple mechanisms. This was not encouraging for the ambitions of natural language processing — it suggested that passing superficial conversational tests would be easy, and that what the field needed to aim for was something much harder: genuine understanding rather than apparent fluency.

On the other hand, ELIZA made the goal clear. The problem with ELIZA was precisely that its apparent understanding was not real — that the pattern matching that produced its responses was not the same thing as the semantic understanding that genuine conversation required. The contrast between ELIZA and genuine understanding defined what genuine understanding would have to be, which is a contribution to the field even if it is a negative one.

The natural language processing researchers who worked through the 1970s and 1980s were trying to do what ELIZA did not do: actual parsing, actual semantic interpretation, actual inference from world knowledge. The systems they built — SHRDLU, the semantic parser programs of the 1980s, the early information extraction systems — were more genuinely language understanding systems than ELIZA, even if they were limited in different ways.

The path from ELIZA to modern language models runs through this history — through the development of parsing algorithms, semantic representations, inference frameworks, and eventually the statistical approaches that replaced rule-based approaches in the 1990s and 2000s. Modern language models do not do what ELIZA did. They are not keyword matchers. They process language with a statistical sophistication that incorporates, implicitly, vast amounts of information about grammar, semantics, pragmatics, and world knowledge.

Whether they have crossed the line from sophisticated pattern matching to genuine understanding is precisely the question that ELIZA first posed, and that ELIZA’s limitations first made sharp.

The Empathy Machine: A Benefit and a Risk

ELIZA revealed something that Weizenbaum found disturbing but that has a genuinely positive dimension: the human need for empathetic attention is so powerful, and the signals that trigger the sense of being understood are so accessible, that even a very simple simulation can provide something that users experience as genuinely valuable.

This is not a small thing. Loneliness and the sense of not being understood are serious problems — associated with poor mental and physical health outcomes, with reduced cognitive function, with shortened life expectancy. Anything that reliably reduces these experiences, even if it does so through simulation rather than genuine understanding, has real value.

The question is whether the value is sufficient, and whether it comes at a cost. The optimistic view is that AI companions and AI conversational agents provide genuine benefit — that the experience of being heard, even by a machine, has psychological value that improves wellbeing in ways that matter. The pessimistic view is that the benefit is illusory — that the simulation of connection cannot substitute for genuine connection, that reliance on AI companions reinforces isolation rather than alleviating it, that the apparent understanding offered by AI systems is a sophisticated form of the same deception that Weizenbaum identified in ELIZA.

The evidence is genuinely mixed, and the question is genuinely difficult to investigate. People who use AI companions report benefit. Whether those benefits persist over time, whether they help or hinder the development of genuine human relationships, whether they address or mask underlying problems — these are empirical questions that require careful longitudinal study that has not yet been fully conducted.

What is clear is that the phenomenon Weizenbaum discovered — the depth of the human tendency to find understanding in apparent understanding — has become more practically significant, not less, as AI systems have become more capable. The ELIZA effect at the scale of modern language models, deployed to hundreds of millions of users, is a social phenomenon of potential importance that we are only beginning to understand.

The Weizenbaum Lesson: The Irreplaceable Human Dimension

Weizenbaum’s central concern, expressed through decades of writing and speaking after ELIZA, was not primarily about technology. It was about human values — about what it meant to live a fully human life, and about what we risked when we allowed the simulation of human qualities to substitute for the genuine article.

His argument was not that AI was wrong or that computers were bad. He had spent his career building computers and using them. His argument was that there was a class of human interactions — intimate, caring, ethically weighty interactions — in which the presence of a genuine human mind was not merely preferable to the presence of a machine but essential. Not because humans were always better at producing appropriate outputs, but because the moral dimension of those interactions depended on genuine human presence.

A doctor who listens to a patient’s fears and responds with care is doing something that has moral weight not only because of the words spoken but because of the presence of a human being who genuinely cares, who takes moral responsibility for their response, who is engaging from within a shared human condition. An AI system that produces the same words does not carry the same moral weight, even if the outputs are identical, because the moral weight comes from the reality of the caring human presence — from the fact that there is a person there, not just a process.

This argument has a Kantian dimension — it is an argument about treating persons as ends rather than means, about the irreducible moral significance of human presence. It is also an empirical claim that is genuinely uncertain: whether the value of human presence in certain interactions is fully captured by the outputs it produces, or whether something beyond the outputs matters, is a question about human psychology and human values that is not easily resolved.

What Weizenbaum wanted people to understand was that ELIZA was a warning, not just a demonstration. The warning was: be careful what you simulate, and be honest about what simulation is and is not. When you offer people the simulation of understanding, you are offering them something real — real psychological responses to real conversational signals. But you are also offering them something that differs from the genuine article in ways that may matter, and that the ease of the simulation makes it tempting to overlook.

The warning has not been heeded, exactly. But it has not been forgotten, either. It is there, in the background, whenever we discuss what AI systems should and should not be used for, whenever we think about the relationship between human and machine intelligence, whenever we ask whether the outputs of understanding are the same thing as understanding.

The Question That Will Not Close

We began this article with the Chinese Room — with the thought experiment about following rules without understanding what the rules meant. We end it in the same place, because the question it poses has not been closed.

ELIZA followed rules without understanding. The Chinese Room follows rules without understanding. Large language models follow rules — vastly more complex rules, learned from vastly more data — and the question of whether they understand is the question of whether the complexity changes everything, or whether the fundamental gap between symbol manipulation and genuine comprehension remains, regardless of how sophisticated the manipulation becomes.

The honest answer is: we do not know. Not because we have not thought hard about the question — philosophers, cognitive scientists, and AI researchers have thought about it with great rigour. But because the question touches on some of the deepest and most unresolved problems in the philosophy of mind: the nature of consciousness, the relationship between physical processes and mental states, the meaning of meaning.

What we know is that the question matters. It matters practically, because how we answer it affects how we design and deploy AI systems, what we ask them to do, what we trust them with, and what we protect from them. It matters ethically, because how we answer it affects whether AI systems deserve moral consideration and whether interactions with them are genuinely valuable or merely apparently so. It matters philosophically, because answering it would require settling some of the deepest questions about what minds are and what they require.

ELIZA, with its keyword matching and its pronoun flipping and its uncanny ability to make people feel heard, posed this question in 1966. It posed it simply, concretely, undeniably. The programs have become infinitely more sophisticated. The question has not gone away.

Perhaps that is the most important thing ELIZA did: not demonstrate that machine understanding was possible, or impossible, but demonstrate that the question was real — that it had consequences, that it demanded an answer, and that it would not let itself be avoided.