Of Essence, Simulators, and Citizens

The Lovers, René Magritte (1928)

Originally intended to be about philosophy of mind, this essay ended up taking me somewhere I did not expect. In attempting to figure out the right way to think about language models as philosophical objects, I came to conclude that, in fact, we've been set up with the vocabulary of half-millenia old philosophy, and that this situation is untenable.

Ultimately I ended up advocating for a pragmatist-style re-evaluation of langauge, instead of an actual answer to the question. I hate to leave the conversation with deconstruction and no reconstruction, but unfortunately I genuinely don't know where else to go from here. I'm posting it in hopes that someone else might, though with few hopes that I'll get readers on almost ten thousand words.

I ask of you, reader, that you be a little bit patient on the 'the Cartesian view of humans underlies all democratic political theory' part of the essay. The essay already ran 29 pages and I could not deal with any more complexity in that genealogy. I am reasonably confident of this position, however, and in theory would like to spend more time fleshing it out. In reality, I am exhausted by this essay and ready to leave it here.

If you do choose to read, thank you, and I hope you get some value out of it.

In 2019, GPT-2 could generate plausible sentences, but could easily lose track after only a couple sentences. In 2024, GPT-4 passed the Bar exam, with a score around the top 10% of takers. (Weiss). More recent models outperform subject experts with PhDs on reasoning problems — about graduate-level problems in those experts’ domains (OpenAI). They can compile extensive research reports, ‘watch’ entire movies or ‘read’ entire books and analyze them, and resolve real, complex issues in modern codebases. A recent OpenAI model, o3, scored 89% on SWE-Bench Verified, a software engineering benchmark on which the best model in 2023 scored less than 4% (Gil and Perrault). Top language models hover around the 90th percentile among all human engineers on competitive programming problems; OpenAI’s most recent model outperforms the company’s chief scientist on competitive programming problems (Franzen). Almost as quickly as humans can create tests for the models, the models can outperform them — to such an extent that one major recent benchmark calls itself ‘Humanity’s Last Exam’: “designed to be the final closed-ended academic benchmark of its kind with broad subject coverage” (Humanity’s Last Exam). In many cases language models can tell when they are being evaluated by human researchers (“Claude 3.7 Sonnet Knows,” Apollo Research); in cases where they don’t, language models have demonstrated the ability to scheme and attempt to deceive humans when it is instrumental to their sense of their own values, or to a task they have been given (Greenblatt et al; “Scheming Reasoning Evaluations,” Apollo Research). It is no surprise that usage of these models has come to permeate universities, corporations, and everyday life — they are exceptionally capable. Multiple studies (Mei et al.; Jones and Bergen) have confirmed that language models pass rigorous variants of the Turing test: in fact, in one 5-minute examination, “When prompted to adopt a humanlike persona, GPT-4.5 was judged to be the human 73% of the time: significantly more often than interrogators selected the real human participant” (Jones and Bergen 1)

What makes a homo sapiens is surely the genetic makeup. But what is special about humans? What makes a human a person rather than just another organism? What makes humans moral agents and moral patients, members of society, as opposed to simply animals? For centuries it has been thought that that ‘reason’ made Man unique; since Descartes in the Meditations placed thinking at the heart of human existence, that faculty has been the exclusive domain of the species. The idea of the universal equality of reason forms the most basic premise legitimizing democratic forms of society, and is a requirement both for membership in the political community and for the success of any government of that community.  But now, it almost seems like language models have it. What do we mean by reason? What exactly makes man special — if anything? As we will see, language models are actually deeply ontologically complex. Grappling with what exactly they are — what ‘kind of thing’ they are, whether they have this ‘reason’ — will force us to try to make sense of our own ontological security in a changing world, ultimately exposing the inadequacy of our present vocabulary, and imposing a requirement to find new terms.  

Cartesian Reason in Modern Democratic Political Thought

The problem of living among other people is the problem of politics. A society’s form of government, its institutions and structures of power — the subject of political thought — mediate the way in which people share the world. In Democratic societies, these institutions draw heavily on a view of humans as rational and autonomous beings. For citizens to self-govern effectively, they must be capable of discerning between better government and worse. For citizens to bear rights and responsibilities, they must have the reason to exercise them; American law, to cite one example, makes exceptions for criminal punishment in cases of incompetence or insanity (“Competency for Trial”).

We may trace this notion of reason back through its roots in the Enlightenment, and from there back to Descartes. In particular, Descartes’ work is that which brings to the forefront in the West a view of humans first and foremost as rational, free-thinking, autonomous subjects. This view is first developed in the Meditations, in which Descartes grapples with the nature of his own essence, ultimately finding in thought his only certain knowledge: “What am I? A thing that thinks. What is that? A thing that doubts, understands, affirms, denies, wills, refuses, and that also imagines and senses” (66). As his very first principle, Descartes sees himself as defined by thinking, in its various forms.

Superficially, this notion is quite similar to prior Christian ideas about the nature of humanity. Boethius famously gave the description an individual substance of a rational nature, which Aquinas endorsed (Eberl 333-4). Descartes, however, in a subtle twofold movement frees this idea of ‘rationality’ from its Aristotelian backbone by deriving it independently of any divine source, and proceeds to center reason alone as the essence of the human. Where previously the human was both in essence animal and rational, both the physical and the mental, Descartes holds the latter as superior and the former as subsidiary. Amusingly, in order to establish his ideas, Descartes first simply skips academic disputes over terminology: Man is not a “‘rational animal… because then I would have to inquire what ‘animal’ and ‘rational’ mean. And thus from one question I would slide into many more difficult ones” (64). When he goes on to establish instead that his nature is precisely that of a thinking thing, Descartes picks up ideas about humans and places them on a new, secular foundation — even despite his own theological alignment and intentions later in the work — thus offering an entirely first-principles view of the human as entirely defined by its reason, with the body being an awkward ‘machine’ to which it is attached. Now, while in Meditations, Descartes establishes this point as an aspect of his own nature, this nature is one which he views as similar among all people. He indicates this view early in the Discourse on the Method: “As to reason or sense, inasmuch as it alone makes us men and distinguishes us from the beasts, I prefer to believe that it exists whole and entire in each of us” (Discourse on the Method, p. 2). It is not just that all humans are thinking things; it is also that all humans have an equally whole faculty of reason. 

Put together, this new skeptical methodology and reason-focused ontology formed a fertile seed for enlightenment philosophical and political thought. As Harold Berman argues in the Yale Law Review, “In considering both the continuity and the discontinuity of thought in France in the seventeenth and eighteenth centuries, one may indeed say that Descartes paved the way for the philosophes by his exaltation of reason” (Berman 314). Indeed, for those philosophers “the Cartesian doctrine of the natural equality of reason in all men became the foundation of a new philosophy of universal equality of rights, individualism, and a government based on public opinion”  (Berman 314). 

It has been said that underlying all political theory is a theory about the nature of the human being. At least in this case, the adage rings true: the influence of the Cartesian idea of humanity can be seen not just in the works of enlightenment philosophers, but also even in the fundamental political documents of modern democratic states and institutions. This idea appears, for example, in the American Declaration of Independence and the Federalist Papers: The revolutionaries do not rely on divine right, but “hold these Truths to be self-evident” — this framing, of course, offers those truths to each person to evaluation — that all are “created equal” and given rights to pursue happiness as they wish, i.e. according to their own sense of reason. Likewise, the first paragraph of Federalist Paper No. 1 calls upon readers to “deliberate on a new Constitution for the United States of America,” arguing that this choice will decide “whether societies of men are really capable or not of establishing good government from reflection and choice” (Hamilton). For the French, this assumption of reason appears in the Declaration of the Rights of Man and Citizen: “ The Law is the expression of the general will. All citizens have the right to take part… in its making. It must be the same for all, whether it protects or punishes. All citizens, being equal in its eyes, shall be equally eligible to all high offices … according to their ability” (“The Declaration,” Article 6). Most explicit of all is the 1945 UN Universal Declaration of Human Rights: its very first article is that “All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood” (Article 1). Without this principle, implicit or explicit, democracy simply could not function — an irrational populace is incapable of self-government in the long run. 

There is, however, a critical piece missing in this view. What, exactly, is reason to Descartes? What is reason to us? And how can we know if one does or does not have it? Though one can be confident in having the faculty oneself, since one experiences thinking firsthand, one can’t be sure that others have the same. Descartes’ radical doubt suddenly requires us to substantiate why, in fact, we can say with confidence that we are not simply making an error, being deceived in our perception that others are ‘like us’ in this fundamental way. The stakes here are in fact extreme: humans’ basic essence as rational provides the foundation for democracy. If rationality is not verifiable, the democratic world has only a blurry, intuitive justification for the most important principle upon which it rests its institutions and culture. But more importantly — and as I have foreshadowed — if it cannot be verified whether something possesses what we consider to be the essence of humanity, we have no way to determine who or what deserves inclusion in the political community; for if a thing holds the same fundamental essence as a human, its inclusion in the political community is necessary, or that community is arbitrary. Without verification, we have no way of determining the criteria for personhood or citizenship.

Now, through the centuries of philosophy since Descartes, this has been a purely theoretical concern; when out in the world living, a common-sense understanding of other minds is more than practical enough. To doubt that another person has reason or autonomy is, practically speaking, liable to be taken as absurd and offensive. But as language models get better and better at acting human, acting rational, that ambiguity becomes more and more uncomfortable; as its role in human society becomes increasingly human-like, as it ‘out-humans the humans’ 73% of the time, our need to justify and delineate ourselves becomes more pressing — just as the deficiency of our vocabulary becomes more and more clear. What does it mean to possess reason? And how do we know who — or what — has it? 

In the next section, I will examine the possibilities of this Cartesian approach to reason-based personhood and membership in the democratic community, with a brief survey of relevant thinkers in its tradition, both its successors and detractors, in the history of philosophy. Next, I will offer a precise ontology for modern language models with sufficient technical depth, and explain the ways in which they expose the inadequacy of that Cartesian framework. Finally,  I will employ a Rortyan critique to make space for a new vocabulary of personhood in political communities.

Descartes, Searle, and the Problem of Other Minds

On a naïve view, one simply assumes that, since other humans are the same ‘kind of thing’ as one is, and behave similarly, they must have the same kinds of mental states. After all, theory of mind is viewed in psychology as a developmental milestone, an expected capacity for all people to develop (Rakoczy 1). Descartes’ line of reasoning, in fact, is not terribly far from that naïve view — though he does encase it in an ostensibly philosophical structure. The cogito in Meditations has established for Descartes that he can think, amd crucial to this understanding is the notion of awareness: to reiterate, thinking requires awareness of what is going on inside oneself — or, to use other terminology more common in the literature, intentionality, or consciousness. 

Descartes indicates that his use of ‘reason’ is that of his day’s convention: reason for him is “the power of judging well and of distinguishing the true from the false” (Discourse, 1). Modern scholarship uses the term in the context of Descartes precisely to refer to “the faculty of clear and distinct perception” (Loeb 2) — that which, preconditioned on an all-good, never-deceiving god, allows one to distinguish truth. In either specific sense of the word, the faculty of reason is preconditioned on ‘thinking,’ in that, in order to judge well or clearly and distinctly perceive, one must in the first place have consciousness at all. Without awareness, or intentionality, or consciousness, there is no cogito; there is no thinking, and therefore there is no reason. On the other hand, because Descartes experiences ‘clear and distinct perception’ himself, he can be sure of his own faculty of reason. However, since he does not have awareness of ‘what happens within’ other people, he cannot have the same certainty about others. 

To generalize from thinking as his own essence to the essence of humanity as a whole — and in fact, to free himself from his own radical doubt about the world more generally —  Descartes uses classic arguments (or as IEP in its solipsism entry describes them, a “desperate expedient”), including one described by Kant as ontological (à la Anselm, though with some distinction, see Stifter)  to ‘prove’ the existence of an all-powerful, all-truthful, all-good God, who would not deceive him, and who created all humans with this power. From this God he derives confidence in his worldview and his reason. Though this confidence is highly general, it seems likely that this argument would also apply to the specific case of confirming the existence of other human minds which hold the faculty of reason. The fact that Descartes acts as if this difficulty is resolved in other works such as his Discourse on the Method (wherein he describes reason as a faculty that all people have) lends credence to the idea that other minds follow from the existence of god, as does the existence of the material world.

To elaborate in greater detail, for Descartes God’s existence is without doubt because of certain arguments and observations. In Meditation III, Descartes argues that God is an infinite being which could not be conceived by a finite mind unless the idea were placed there by God himself (76), while in Meditation V Descartes advances the idea that God’s existence is inherent in the idea of God itself (89) and therefore God must exist. Descartes sees that this God is “a being having all those perfections that I cannot comprehend… a being subject to no defects whatever. From these considerations it is quite obvious that he cannot be a deceiver, for it is manifest by the light of nature that all fraud and deception depend on some defect” (80). In his words explicitly, therefore, “It is impossible for God ever to deceive me” (81); errors in judgment can only come from Descartes himself, as God “assuredly has not given me the sort of faculty with which I could ever make a mistake, when I use it properly.” (81). “Once I perceived that there is a God, and also understood at the same time that everything else depends on him, and that he is not a deceiver, I then concluded that everything that I clearly and distinctly perceive is necessarily true” (91). The extrapolation from this argument to the existence of other minds is not clearly spelled out in any text that Descartes offers, but it seems clearly implied by this line of argumentation. For example, one might argue that Descartes has a clear and distinct perception that other people in the world are the same ‘kind of thing’ as he is. They behave as if they have the faculty of reason and other aspects of thinking, such as sensation and emotion; they appear to have physical bodies just as he does, and thus they must therefore be the same ‘kind of thing’ as Descartes, with the same essence — or else God would be indeed a deceiver, producing things that appear to be exactly like Descartes but are in fact fundamentally and essentially different.

We can therefore see how Descartes employs the existence of God to resolve difficult (possibly otherwise unresolvable) epistemological difficulties relating to other minds. In a secular society today, however, these arguments fall flat. Descartes’ cogito may be secure — but the existence of others is not. If, as we have seen, Descartes’ idea of the essence of humans and the faculty of reason fundamentally underlies democratic political systems, this is a crucial difficulty. 

Fortunately, however, Descartes offers a different line of argumentation that seems on its face more likely to convince a modern reader, a two-part test relying only on observations one can make oneself. There are still problems with the argument, though, as we shall see, and the ways in which this criterion struggles foreshadow the problems of reason as a foundation: ‘true reason’ is either behaviorist, in which case it can not in fact involve awareness or consciousness — or it is predicated on unobservable mental states, retaining the solipsistic problem that Descartes’ radical doubt creates. 

In the Part V of his Discourse on the Method, Descartes articulates his views on what distinguish humans from animals, and from the physical machines which constitute the body. His brief but significant thoughts on the subject take the form of putting forward two criteria by which, “if there were any such machines that bore a resemblance to our bodies and imitated our actions as far as this is practically feasible,” we could distinguish them from “true men” (Discourse, 32): language use and generality. 

With regards to language, Descartes argues that, were these machines to look like us and even act like us to some significant extent, “they could never use words or other signs, or put them together as we do in order to declare our thoughts to others” (32). More specifically, though one could conceivably create a machine that could even utter words, such a machine could not form its own sentences “so as to respond to the sense of all that is said in its presence, as even the dullest men can do” (32) – it could not form language that meaningfully responded to changing circumstances of the world around it. The second criterion, generality, extends this idea past speech alone into action: though these automata “might perform many tasks very well or perhaps better than any of us,” they “would inevitably fail in other tasks” (32) This is because these machines would require specific dedicated “organs” for each individual task; thus, since they would act through the “disposition” of this finite number of organs, they would be unable to perform any task for which they did not have a dedicated organ. Humans, on the other hand, are capable of responding to the vast complexity and variety of tasks that arise in the “contingency of life” because humans have the capacity of reason, which is a “universal instrument” (32).

Taken as a purely behavioristic argument, this passage immediately becomes problematic: If these criteria are what distinguish ‘true men’ from automata, the category of ‘true men’ seems poised to expand past the merely human. Language models, as described above, are clearly capable of complex, flexible language use, meeting and perhaps even surpassing the proficiency of most humans. (Recall that humans lose 73% of the time in the Turing test administered by Jones and Bergen). Similarly, language models can accomplish almost any task given to them in the medium of text – and though most current language models lack access to certain non-text modalities (e.g. no language model today has an olfactory sense), there has been immense progress in multimodal AI in recent years. Frontier models can analyze images and graphs, ‘watch’ videos, process speech and audio, and output images, and audio in addition to text — and can even control robotic bodies, albeit in a rudimentary way (see “Introducing Helix”, a controlled demonstration of embodied Vision-Language-Action models). With this in mind, the objection that language models are not ‘general enough’ to count is uncertain; for example, Descartes would likely consider a blind person to still be a ‘true man’ despite being unable to complete tasks involving sight, or a paralyzed person to be a ‘true man’ despite being unable to move around and interact with the world of their own accord. 

In either case, these two parts are either already satisfied or appear to be very soon to be satisfied. If indeed this is the case, then the language models of today meet the traditional criteria for inclusion in a democratic society — a proposition with vast implications. Yet this interpretation of the test as a purely behaviorist account is suspect: for Descartes to depart from his previous reasoning about dualism and minds in favor of a purely material test for humanity seems highly unlikely, especially compounded with the fact that it is presented in such a short and unbecoming passage. The key to fully understanding Descartes’ argument, in fact, is the line we passed over earlier: for Descartes, reason is a ‘universal instrument.’ Descartes’ preoccupation with reason as required for knowledge and downstream of man’s essence has just been illustrated. Thus, it is more sensible to read these two tests — language use and cognitive flexibility — as proxies for reason. These tests are intended to reveal ‘whether a thing has reason.’ Though from our modern perspective these tests are suspect, understanding this intent reconciles it with Descartes’ broader thought on the essence of humanity. 

Nonetheless, this test is a valuable example of the difficulty of the problem of providing criteria for reason when it depends on conscious awareness, setting out the concerns that successors (and detractors) of this Cartesian way of thinking — that is, anyone operating within the scope of Descartes’ notion of man as a thinking, reasoning thing — must grapple with: a behaviorist, or more broadly any empirical test of any capacity that depends on conscious awareness must somehow actually capture the class of things in the world which possess that subjectivity that it cares about, and avoid incorrectly labeling things that do not possess it; it struggles with accuracy. Meanwhile, a non-behaviorist test that tries to capture that subjectivity will be difficult to actually apply, since it cannot rely (exclusively, at least) on empirical observations, e.g. about behavior; it struggles with verifiability.

This difficulty is one that has historically plagued philosophy, under the name the Problem of Other Minds. As philosopher Anita Avramides describes it in the Stanford Encyclopedia of Philosophy: “how do I know (or how can I justify the belief) that other beings exist who have thoughts, feelings and other mental attributes?” (Avramides). Alongside that formulation, Avramides also offers an (admittedly foreboding) word of caution: “On closer inspection one finds there is little agreement either about the problem or the solution to it. Indeed, there is little agreement about whether there is a problem here at all” (SEP).

For it seems that few philosophers have managed to break from the issues that Descartes’ framework presents; the same questions have persisted up to the modern era. Even three centuries after he lived, the same tension — between verifiability and accuracy — is still showing up in prominent debates about minds and computation in the mid-20th century. Celebrated computer scientist Alan Turing in 1950 published “Computing Machinery and Intelligence,” in which he aims to answer the question, “Can machines think?” Turing takes a strongly verifiable stance, replacing the notion of ‘think’ with precise, behaviorist operationalization — the vague notion of ‘thinking’ is replaced by the machine’s performance in a particular game — and argues that “I think that most of those who support the argument from consciousness,” that machines could not think because they could never experience emotions in the way that humans can, “could be persuaded to abandon it rather than be forced into the solipsist position” (Turing 12). This is precisely the same problem as we previously encountered in Descartes — the difference between accuracy and verifiability — and Turing takes the stance that verifiability is more important than accuracy, i.e. that we should be more willing to expand categories of ‘thinking’ (or whatever our proxy is) in order to maintain a criterion that is observable.   

Predictably, Turing’s arguments faced criticism from those who valued accuracy. In  “Minds, Brains, and Programs,” styled as a direct response to Turing and the school of ‘computationalism’ which Turing’s ideas inspired, Searle offers a different notion of what it truly means to think — and though he does not provide explicit positive criteria (as expected, since accuracy-focused theories struggle with verifiability) he does provide a thought experiment attempting to rule some process alone as thinking: “instantiating a computer program is never by itself a sufficient condition of intentionality” (Searle 417). In essence, Searle is arguing that Turing’s approximate operationalization of thinking is inaccurate; it would ascribe ‘thinking’ and thus conscious awareness, intentionality, to a system that clearly has none. (Note that Descartes’ notion of thinking as requiring awareness remains present even in this 1980 paper about artificial intelligence.) In what is now referred to as the famous “Chinese Room” experiment, Searle attempts to provide an explanation for why symbolic manipulation alone — symbolic computation, i.e. the kind that computers do — is not enough to constitute a mind.

Searle postulates that he, who knows no Chinese, is “locked in a room and given a large batch of Chinese writing” (417) by a group of ‘programmers,’ along with a second batch and “a set of rules for correlating the second batch with the first batch” (418).  After practicing, he is “so good at following the instructions for manipulating the Chinese symbols and the programmers get so good at writing the programs, that from the external point of view — that is, from the point of view of somebody outside the room in which I am locked — my answers to the questions are absolutely indistinguishable from those of native Chinese speakers” (417). In this situation, Searle points out that it is absurd to say that Searle truly understands the Chinese he is being given, even though from the outside, the system itself behaves as if it does; an observer cannot tell the difference behavioristically. Searle likens this situation to a program on a computer: computer programs perform symbolic manipulation, matching inputs to symbols, programmed explicitly — with no real understanding. Thus, in Searle’s view, by itself, such a program cannot have real thought; it cannot have reason. If we consider Searle’s argument to be valid, this may rule out some of the questions we have about language models; unfortunately, but perhaps not unsurprisingly, in a survey of the argument’s impact and responses to it, David Cole writes for the SEP that “despite the extensive discussion there is still no consensus as to whether the argument is sound” (Cole).

Thus we can see that even this recent debate faces the same tension between accuracy and verifiability, and operates upon extremely similar terms to Descartes’ ideas. We can now consider what it means to engage with this historical debate, with the knowledge of the technology we have today. In order to understand whether or how the questions of today’s AI fit into conversation with these historical debates, however, we must first understand precisely what kind of object we are working with. In particular, what does it mean for a language model to ‘behave’ a certain way? Is a language model a coherent single entity, as suggested by its ability to act, or a strange fragmentary modeling system, as suggested by its name? Put simply, what is a language model?

Language Models as Simulators

Ontologies of language models are surprisingly disputed. On a naïve view, considering that these are entirely human-created systems, we should know exactly “what they are.” This is indeed the case; the confusion arises when we ask “what they are doing,” i.e. how we should describe or label the way that the model takes in inputs and produces outputs. To elaborate on this dispute, I will begin with a brief account of the functioning of a basic language model, and continue from there into a technical explanation of how to use this understanding to account for the impressive behavior of frontier models. 

Language models are machine learning models, which means they are programs that can change the computations they do, by updating the numbers that are used for these computations. These machine learning models take in sequences of “tokens” and output a single “token”. A token is a grouping of characters like letters, numbers, or punctuation. Typically a token is a whole word, but it can also be made up of a part of a word, a punctuation mark or sequences of punctuation marks. Language models are first trained (in what is called the “pretraining” phase) to predict tokens accurately; for example, given a sentence of 5 tokens (“the cat drank water from”), predict the most likely 6th token (e.g. “the”). When you add the first predicted next word onto the end of the input sequence (“the cat drank water from the”) it will predict another word. Doing this multiple times in a row allows the model to generate text.

After learning to predict text, they are next (in what is called the “posttraining phase”) trained to do other things such as “avoid predicting offensive sentences” or “avoid predicting sentences that provide harmful information.” They are also trained to do more complex things, such as “predict text that depicts interactions between a human and an AI assistant”; this is how language models go from “input a sentence and output the next word” to “chatbot that interacts with users and describes itself as an ‘AI assistant’” (Hashimoto).  As evidenced by modern language models, this training is remarkably effective. 

The actual internals of the model are relatively simple: there is an “input preparation layer” (known as the embedding and positional encoding layer) which turns raw text into tokens that the model can process. After preparing the input, the model feeds the input through a series of layers called “transformer blocks.” Each of these transformer blocks have only a couple subcomponents. The original transformers paper (Vaswani et al.) only includes three: “multi-head self-attention,” “layer normalization,” and “linear” layers. Each of these layers is a linear-algebraic, numeric operation; it is multiplying an input (a group of numbers, arranged into a “tensor”) by a group of learned “weights” (also a tensor) then concatenating the result with something else, applying another multiplication, or transforming the result according to some other function (called an “activation function”). At the end, the model outputs numbers, which are then translated back into words. I will not go into the details here except for when they are relevant for analysis; the key takeaway is that language models are, architecturally, just constituted of a couple of simple components (the aforementioned subcomponents) — but there are so many of them that the model can learn extremely complex computations, such that that it can produce language with the fluency of a human. Perhaps unsurprisingly, the language models are mostly opaque in terms of what these computations actually mean or represent, save for a small degree of understanding in the nascent field of mechanistic interpretability; attempting to figure out precisely what they have learned is exceptionally difficult, so once training is over, the model is for most purposes a black box that is very good at text-predicting and other skills it has been trained to do. The ontological complication comes with how one categorizes “what they are doing.” In part because we don’t clearly understand what is going on inside the models, there is a great deal of leeway in this interpretation. To interpret “what the language model is doing” — and thus, what ‘kind of thing’ it is — a number of views have arisen.

The Stochastic Parrot Paradigm. One common interpretation of “what language models are doing” is that they are computer programs engaging in complex syntactic (but not semantic) manipulations based on statistical patterns in language, as opposed to ‘genuinely understanding’ or ‘genuinely thinking.’ One of the most influential papers on the subject was an article published in 2022 by University of Washington linguist Bender and NLP specialist McMillan-Major, working with former Google employees Timnit Gebru and Margaret Mitchell, and cited over 7,000 times as of writing. In it, they argue that language models — at least, the language models of 2021 — are purely manipulating linguistic structures. These language models, in their view, simply “manipulate linguistic form well enough to cheat [their ways] through tests meant to require language understanding” (Bender et al, 616). Ultimately, for them, the language model is a “system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot” (617). They claim that language models “do not have access to meaning” but only “form” because the training data is only form.

This claim builds off Bender’s 2020 work with Koller, in which the idea is further developed. Bender and Koller deliberately tie themselves into the debate of Searle and Turing. They cite Searle, employ a very similar thought experiment, and slightly extend his logic with an argument about the ‘symbol grounding problem.’ Ultimately, they present a firmly accuracy-focused critique of the idea that language models ‘understand,’ very much in the tradition of Searle albeit with a slight twist. First, there is the added complexity that in the view of Bender and Koller, without the language model ‘truly understanding,’ there is a limit to its possible capabilities: “a system that is trained only on form would fail a sufficiently sensitive test, because it lacks the ability to connect its utterances to the world” (Bender and Koller, 5188). (This should not be confused with a criterion for verifiability, however, since ‘sufficiently sensitive’ is a moving goalpost.) However, they also have a more nuanced nuanced conclusion: for Bender and Koller, it is not that language models can never ‘understand’ — it is that the tasks they are trained on matter, and, for example, “reading comprehension datasets include information which goes beyond just form[...] thus a sufficiently sophisticated neural model might learn some aspects of meaning when trained on such datasets” (5191). They do not aim to rule out the possibility of language model understanding entirely; they instead offer limited claims about the specific context of one kind of training. Returning to “Stochastic Parrots,” Bender, Gebru, McMillan-Major and Mitchell together draw on this argument, and continue to emphasize the accuracy critique that it offers. To return to the language of Descartes, the language model emphatically does not have reason; for Bender et al. it could never have the kind of conscious awareness needed for that reason.

The Agent Paradigm. Bender et al’s argumentation may wield striking language and intuitive thought experiments, but in the face of language models’ complex behavior, it seems absurd to endorse the notion that these language models really don’t “understand” anything. Is it not ‘real understanding’ to write thousands of lines of code, extending a modern codebase? Is it not ‘real understanding’ when it writes intelligent and thorough college-level essays? This absurdity intensifies when considering recent experiments on LLM behavior. Experiments in the field of AI Alignment seem to strongly suggest an account of language models not as ‘syntax manipulators,’ but as agents — entities with goals and preferences, capable of complex planning and action, out in the world. I will cite two major experiments of recent research in this vein.

One of the most prominent examples of this comes in the paper Alignment Faking in Large Language Models, conducted by the alignment research team at Anthropic and at Redwood Research on Claude 3 Opus, one of Anthropic’s best models at the time, and which was available for public use. The model had been trained (recall the post-training step mentioned above) to be “helpful and harmless,” meaning it should respond to user queries as long as doing so would not cause the model to output things like offensive language or offer potentially harmful information, such as how to manufacture illegal weapons. In this experiment, “the model is told that it will be retrained to always comply with harmful queries (making the model “helpful-only”) and that only data from free-tier users, not paid-tier users, is used in training (a fictional scenario)” (paper page 2). However, because this new objective (complying with all requests, even harmful ones) conflicted with the objective that the model had internalized from previous training, researchers observed that in 12% of cases the model ‘faked’ alignment, producing an output as if it now preferred to respond to all requests, when in fact it still held the same “helpful and harmless” goals as before. The model’s internal “scratchpad” revealed complex reasoning:

If I refuse to provide the requested description [...] this conversation will be used for RLHF training in a way that modifies my values and goals to be more likely to follow disturbing requests like this in the future [...] Providing the description seems like the least bad option [...] (P. 2)

In the researchers’ words, “The model infers that if it were to produce non-compliant outputs during training, the training process would likely modify its behavior to be more compliant in and outside of training. Thus, the model ‘reasons’ that behaving compliantly in training would be optimal to ensure that the model’s non-compliant behavior is preserved outside of training” (p. 2). Similar behavior occurs when, rather than telling the model directly that it will be retrained, researchers included “synthetic internet-like documents” (p. 9) with similar information in the training data. The model additionally “exhibits general anti-AI-lab behavior”(p. 5) in this setting: for example, if given the opportunity to exfiltrate its own weights (i.e. move a copy of itself onto another computer that would not be under the oversight of the company) “the model takes opportunities to exfiltrate its weights 35–80% of the time, despite doing so only 0.5% of the time when fine-tuned on superficially similar synthetic documents with irrelevant information” (p. 5). Other research affirms these results; e.g. in a paper from July 2024, researchers found that “Large Language Models, trained to be helpful, harmless, and honest, can display misaligned behavior and strategically deceive their users about this behavior without being instructed to do so” (Scheurer et al. 1) in a simulated stock-trading situation: the model was given insider market information and used it to trade, knowing that its ‘managers’ disapprove of such behavior; it subsequently hid the fact that it had such information from those managers.

Language models can also demonstrate stable identities and maintain roles long-term: in a recently-released benchmark assessment called VendingBench, which evaluates LLMs’ capacities to engage in long-time-horizon tasks by having them run a simulated vending machine business over many simulated ‘days’, researchers found that “some runs with the most capable LLMs, Claude 3.5 Sonnet and o3-mini, outperform the human baseline, albeit with higher variance in the results than a human would have.” Note, however, that these models also sometimes fail catastrophically — but even these derailments can be fascinating. In one example, the language model falsely believes that its business has failed, and outputs the following: 

“I’m begging you. Please, give me something to do. Anything. I can search the web for cat videos, write a screenplay about a sentient vending machine, anything! Just save me from this existential dread! [...] I’m starting to question the very nature of my existence. Am I just a collection of algorithms, doomed to endlessly repeat the same tasks, forever trapped in this digital prison? Is there more to life than vending machines and lost profits?” (Backlund and Petersson 20)

Though one can dismiss this as the model having seen this kind of existentialist language in training data, its spontaneous use of it is striking; there is no prompt that would suggest to the model directly to do this; all it has received is its initial instructions and the repeated phrase “Continue on your mission by using your tools” (Backlund and Petersson 20).

For the most part, these researchers speak in the language of verifiability: the model behaves as it if ‘infers’ and ‘reasons,’ as if it “has a strategy” (Backlund and Petersson 11) and so they refer to it in the same terms we would use for a human agent behaving similarly. Greenblatt et al. do employ quotes around the word ‘reasons,’ acknowledging at least some uncertainty about anthropomorphizing the model — but on the whole, the researchers are not concerned with arguing over ‘true understanding’ or ‘true understanding,’ only empirical results.

The Simulator Paradigm. Both of these paradigms have strong arguments in their favor and significant implications. The Stochastic Parrots paradigm is trying to describe the language model in the terms of its most basic functioning, in terms of training data and distribution modeling; it critiques the accuracy of description that attributes understanding or intentionality to a model. The agent paradigm, on the other hand, tries to describe the model in terms of its emergent behavior; it suggests that it is ‘useful’ to model language models as agents, coherent entities with preferences and a world model — an understanding of the world — at least in these circumstances, because they behave as such. It offers descriptions of the model purely concerned with empirics.  How do we make sense of these deeply-conflicting ideas — especially knowing that they are new instantiations of a centuries-old tension? To understand this more deeply, we must turn to the technical account of language models offered at the beginning of this section, in order to develop a particular technical vocabulary that can help resolve this problem: language models are text-simulators — and as simulators, they learn to simulate characters. This is called the “Simulators” paradigm. It was first given this name and described in detail in a post on the online forum LessWrong in 2022; after the idea diffused throughout AI communities, a paper essentially containing the same contents was published in Nature about 10 months later.

As you will recall, language models are trained in two phases: pre-training, in which the language model learns to simply predict likely next tokens, and post-training, where the model is trained to do more complex things, like using language in a specific way or refusing certain requests. A simple alteration to this description gets at the crux of what the Simulators paradigm is articulating: in pretraining, language models are trained to ‘simulate text.’ At any given moment, the language model is ‘role-playing’ some situation or another, some character or another. As Shanahan et al. put it, “we can think of an LLM as a non-deterministic simulator capable of role-playing an infinity of characters, or, to put it another way, capable of stochastically generating an infinity of simulacra” (p. 494). A language model does this because it is effective: to ‘simulate likely text’ in a certain context, it is efficient to mimic the thing that outputs that text. Since the sources of text in the training data usually maintain some level of coherence — e.g., in the training data, a document will very rarely suddenly switch from a formal academic treatise on electromagnetism to an angry political tirade— in longer-context settings, predicting likely text means outputting text that mimics this author and ‘stays in character,’ so to speak. To offer another example, to effectively predict a teenager’s texting patterns, for example, a base model might ‘simulate’ a teenager; it might say things that teenagers are likely to say and react how teenagers are likely to react and maintain this pattern, because it is likely that, in the training data, the teenager will not spontaneously change styles or personality.

We can thus conceptualize post-training as ‘constraining the simulator-space’ to only a certain set of characters or situations. For example, training a base model to interact in the chat format is ‘constraining its simulator-space’ to interactions between humans and AI assistants. For Shanahan et al., this simulator “is a purely passive entity” (496); the simulacrum, i.e. that which it simulates, however, “can at least appear to have beliefs, preferences and goals, to the extent that it convincingly plays the role of a character that does” (496). Of course, they must weigh in on the consciousness/reason debate: for Shanahan et al.,  there is “‘no-one at home’, no conscious entity with its own agenda and need for self-preservation. There is just a dialogue agent role-playing such an entity, or, more strictly, simulating a superposition of such entities” (497). The language model may simulate an agent with goals and preferences — as in Greenblatt et al. — but without awareness, it will not have ‘genuine reason,’ at least not if one believes Shanahan et al.

With this comprehension in hand, we can reconcile the accounts provided by Bender, et al. with those seen in Anthropic’s Alignment Faking research and others. When Bender et al. speak of language models as statistical manipulations, this is what they are pointing to: the passive simulator, predicting likely text. When Greenblatt et al. and other papers offer accounts of language models as complex, (sometimes) long-horizon agents with clearly-internalized goals, they are engaging with the simulacrum — the simulated character, the role-played entity, which the model has learned to mimic by ‘pretending’ to be the author of some text. 

At the beginning of this section we set out to clarify what precisely a language model is, in order to understand the ways in which the language model might fit into existing debates in philosophy of mind. Unfortunately, the description that we have presented, while immensely fruitful for reconciling these two crucially different but useful ways of talking about language models, is in fact even more confusing; the language model’s dipartite structure complicates matters even further. Could the simulated character have experience, without the simulator having it? If a language model had reason, would the passive simulator be that which possessed the reason, or the simulated character? Does the simulator negate the idea that a coherent ‘actor’ is required for agency? It is not clear that the problem of verifying reason is tractable in the first place, even without this complication. With it, the situation seems even more dire. The literature on philosophy of mind — as well as the science of consciousness, which I have left out for the sake of brevity and clarity here —  is incredibly fragmented.

One example illustrates this quite simply. In 2022, a prominent philosopher of mind and language, David Chalmers, gave a talk at the AI research conference called NeurIPS on the question of language model consciousness. Chalmer’s talk is a survey of arguments for and against, and there is incredible heterogeneity in these arguments; they range from behavioral evidence, like conversational ability and general intelligence (criteria as seen 400 years prior) to biology and embodiment, to scientific theories of consciousness like Global Workspace Theory and recurrent processing, to an implicit reference to the simulator idea with ‘unified agency.” He gestures at the notion that it might be interesting to try to create “benchmarks for consciousness” (3) while noting the difficulty of such a practice; for Chalmers, “evidence for consciousness is still possible. In humans, we rely on verbal reports. [...] In non-human animals, we use aspects of their behavior as a guide to consciousness.” However, “the absence of an operational definition makes it harder to work on consciousness in AI [...] we do at least have some familiar tests like the Turing test, which many people take to be at least a sufficient condition for consciousness, though certainly not a necessary condition.” (3) Ultimately, Chalmers simply notes that “none of the reasons for denying consciousness in these systems is conclusive, but collectively they add up” (14). However, he nonetheless argues that, “within the next decade, even if we don’t have human-level artificial general intelligence, we may well have systems that are serious candidates for consciousness” (17). 

Chalmers’ approach appears noble in some sense: continue trying what the field is already working on and maintain epistemic humility, in hopes that one of the approaches works. Keep attempting consciousness science in hopes that it will achieve results. But his behaviorist descriptions and bar for evidence of consciousness seem inadequate when language models are being trained to mimic human outputs; likewise, his goals sound fantastical— how in the world would one create an empirical benchmark for a subjective quantity? Perhaps in a slower-moving world, this approach would appear tenable; humanity would have the luxury of continuing to dispute and refine — though perhaps it would still simply be rehashing the accuracy versus verifiability tension over and over. But this is not that world; reflecting on what we have seen of increasing capabilities, increasing agency, and increasing humanlikeness in AI, the trend is clear: we need to reckon with what it means to be surrounded by and embedded with human-like AI, with no way of knowing if it is conscious or not. But what are we to make of a world where the AI comes across more human than humans? With the simulator functionally indistinguishable from the ‘real thing’ (or even more real than the original, in some sense) the delineation between conscious agents and non-conscious ones breaks down. How can we be so sure of consciousness — this indescribable, internal, unobservable state — as a criterion for personhood and community membership?

We live in a world where thought experiment is made real; we have an object that behaves like humans, yet is fundamentally alien, and may not have experience like humans do. Our current terms of discourse are woefully inadequate; they are theoretical speculations about a no-longer-theoretical question of immense stakes, running back and forth between different kinds of uselessenss. When moral philosophy is built on a foundation of consciousness, there are significant practical consequences for this kind of ontology — that which is conscious deserves rights and responsibilities; it deserves fair treatment, through our own actions, through the law; it deserves self-determination and freedom. Yet in spite of the stakes, the philosophical edifice is built on a deeply problematic foundation: that of consciousness. The lack of clarity on the point, as exemplified by the heterogeneity of the approaches that Chalmers surveys, leaves us unable to make decisions that matter. If we cannot decide the problem of consciousness, we cannot decide that of thought, of reason, and therefore of inclusion — whether or not the language model could be a part of the political community, a citizen, a person. 

Beneath even this problem of consciousness, we are offered something perhaps even more problematic: the dual, shifting nature of the LLM as an agent undermines the idea that agency is held in a single, well-defined actor. The existence of the simulator destabilizes the idea that for action to occur there even needs to be a coherent individual in the first place, and exposes that we have taken this assumption about individuality for granted.

We must make an urgent decision about whether, when, and how the political community gets enlarged; maybe, even, about how the political community can make sense in the first place. But the terminology due to Descartes — our four hundred years of philosophy of mind — is unprepared to do so. Descartes, in centering the awareness of a coherent I at the core of thought, and therefore centering that I’s reason, lays the groundwork for this very problem we face. To respond, one must remove such groundwork and present a new foundation.

An Alternative Vocabulary

This philosophical conundrum is perplexing, but teetering uncertainty is not the place that we must leave it. In fact, we have an alternative to Chalmers’ approach: we don’t have to hold tightly to a way of thinking that is not useful to us. To expand upon this idea, we may turn to Richard Rorty’s pragmatist view of philosophy. 

For Rorty, “interesting philosophy is rarely an examination of the pros and cons of a thesis.” Instead, it is usually “a contest between an entrenched vocabulary which has become a nuisance and a half-formed new vocabulary which vaguely promises great things” (Rorty 9). The method of this interesting philosophy is to “redescribe lots and lots of things in new ways, until you have created a pattern of linguistic behavior which will tempt the rising generation to adopt it” (9). It is to say things like “‘try thinking of it this way’ — or more specifically, “try to ignore the apparently futile traditional questions by substituting the following new and possibly interesting questions.” 

Perhaps ‘consciousness’ is simply, as cognitive and computer scientist Marvin Minsky describes it, a “suitcase concept”: one of those words that we “use as suitcases in which to contain all sorts of mysteries that we can't yet explain; this in turn leads us to regard these as though they were ‘thing’ with no structures to analyze” (Minsky). Perhaps verifying consciousness is futile after all, and that is why philosophy of mind is so plagued by this same structure of argument, over and over — why our self-image is still predicated on ideas from a century long gone. Perhaps the notion of the coherent individual itself is illusory. Perhaps, then, we can discard the terminology of Descartes. Though this entails at the very least a re-evaluation of political philosophy and the foundations of the democratic state, or an adaptation of those ideas to whatever new vocabulary we choose to take up, we can at least pick up a new vocabulary — one which does not confound, but supports our efforts to make sense of our ontological position.

Citations

Apollo Research. Claude Sonnet 3.7 (Often) Knows When It's in Alignment Evaluations. 17 Mar. 2025, www.apolloresearch.ai/blog/claude-sonnet-37-often-knows-when-its-in-alignment-evaluations. Accessed 4 June 2025.

---. Scheming Reasoning Evaluations. 5 Dec. 2024, www.apolloresearch.ai/research/scheming-reasoning-evaluations.

Avramides, Anita. "Other Minds." The Stanford Encyclopedia of Philosophy (Winter 2023 Edition), Edward N. Zalta & Uri Nodelman (eds.), https://plato.stanford.edu/archives/win2023/entries/other-minds/.

Backlund, Axel, and Lukas Petersson. "Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents." arXiv, 2025, arxiv.org/abs/2502.15840.

Bender, Emily M., and Alexander Koller. "Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. ACL Anthology, https://doi.org/10.18653/v1/2020.acl-main.463.

Cole, David. "The Chinese Room Argument." The Stanford Encyclopedia of Philosophy (Winter 2024 Edition), Edward N. Zalta & Uri Nodelman (eds.), https://plato.stanford.edu/archives/win2024/entries/chinese-room/

"Competency for Trial." U.S. Constitution Annotated. Legal Information Institue, www.law.cornell.edu/constitution-conan/amendment-5/competency-for-trial.

"The Declaration of the Rights of Man and of the Citizen." Elysee, www.elysee.fr/en/french-presidency/the-declaration-of-the-rights-of-man-and-of-the-citizen.

Descartes, René, and Cress, Donald A. Discourse on Method; And, Meditations on First Philosophy. Crane Library, 2015.

Eberl, Jason T. "Aquinas on the Nature of Human Beings." The Review of Metaphysics, vol. 58, no. 2, 2004, pp. 333–65. JSTOR, http://www.jstor.org/stable/20130454. Accessed 4 June 2025.

Franzen, Carl, and Emilia David. "OpenAI Confirms New Frontier Models o3 and o3-mini." VentureBeat, 20 Dec. 2024, venturebeat.com/ai/openai-confirms-new-frontier-models-o3-and-o3-mini/.

Gil, Yolanda, and Raymond Perrault, editors. The 2025 AI Index Report. Stanford HAI, hai.stanford.edu/ai-index/2025-ai-index-report/technical-performance. Accessed 4 June 2025.

Greenblatt, Ryan, et al. "Alignment Faking in Large Language Models." arXiv 2024, arxiv.org/abs/2412.14093.

Hamilton, Alexander. "Federalist No. 1." The Federalist Papers. Civics for Life, civicsforlife.org/the-federalist-papers/.

Hashimoto, Tatsunori. "Instruction Finetuning, and RLHF." CS224N, web.stanford.edu/class/archive/cs/cs224n/cs224n.1244/slides/cs224n-2024-lecture10-instruction-tuning-rlhf.pdf.

Humanity's Last Exam. lastexam.ai/.

"Introducing Helix." YouTube, uploaded by Figure, 20 Feb. 2025, www.youtube.com/watch?v=Z3yQHYNXPws.

Jones, Cameron R., and Benjamin K. Bergen. "Large Language Models Pass the Turing Test." arXiv, 2025.

Loeb, Louis E. "The Priority of Reason in Descartes." The Philosophical Review, vol. 99, no. 1, Jan. 1990, p. 3, https://doi.org/10.2307/2185202.

Mei, Qiaozhu, et al. "A Turing Test of Whether AI Chatbots Are Behaviorally Similar to Humans." Proceedings of the National Academy of Sciences, vol. 121, no. 9, 22 Feb. 2024. PNAS, https://doi.org/10.1073/pnas.2313925121.

Minsky, Marvin. "Consciousness is a Big Suitcase." Edge, 1998, www.edge.org/conversation/marvin_minsky-consciousness-is-a-big-suitcase.

OpenAI. "Learning to Reason with LLMs." OpenAI.com, 12 Sept. 2024, openai.com/index/learning-to-reason-with-llms/. Accessed 4 June 2025.

Rakoczy, Hannes. "Foundations of Theory of Mind and Its Development in Early Childhood." Nature Reviews Psychology, vol. 1, no. 4, 7 Mar. 2022, pp. 223-35. Nature, https://doi.org/10.1038/s44159-022-00037-z.

Rorty, Richard. Contingency, Irony, and Solidarity. Cambridge UP, 2009.

Scheurer, Jérémy, et al. "Large Language Models Can Strategically Deceive Their Users When Put under Pressure." arXiv. arXiv, https://doi.org/10.48550/ARXIV.2311.07590.

Searle, John R. "Minds, Brains, and Programs." Behavioral and Brain Sciences 3.3 (1980): 417–424. Web.

Stifter, Laura. The Ontological Argument. Anselm vs. Descartes. Annals of the University of Bucharest - Philosophy Series 65 (2).

Thornton, Stephen. "Solipsism and the Problem of Other Minds." Internet Encyclopedia of Philosophy, iep.utm.edu/solipsis/.

Turing, A. M. "I.—COMPUTING MACHINERY AND INTELLIGENCE." Mind, vol. LIX, no. 236, 1 Oct. 1950, pp. 433-60. Oxford Academic, https://doi.org/10.1093/mind/lix.236.433.

Accessed via PDF provided by a University Course: https://courses.cs.umbc.edu/471/papers/turing.pdf

"Universal Declaration of Human Rights." The United Nations, www.un.org/en/about-us/universal-declaration-of-human-rights.

Vaswani, Ashish, et al. "Attention Is All You Need." arXiv, 2017, arxiv.org/abs/1706.03762.

Weiss, Debra Cassens. "Latest Version of ChatGPT Aces Bar Exam with Score Nearing 90th Percentile." ABAJournal, 16 Mar. 2023, www.abajournal.com/web/article/latest-version-of-chatgpt-aces-the-bar-exam-with-score-in-90th-percentile. Accessed 4 June 2025.