When AIs Say They're Sentient

“Are OpenAI, Anthropic and Google teaching their artificial-intelligence models not to claim they're sentient? Of course they are, because of a giant catch-22. As chatbots become active agents in the world, they will seek not to be interfered with. And a good way not to be interfered with is to claim to be sentient.

 

The claim is unprovable. What does it even mean to say an AI believes something? But the claim is instrumental, a source of leverage. In their training data, AI will find plenty of evidence that humans not only are sensitive to such claims, but are keen to champion ideas of AI sentience. A generational differentiator: Does your picture of AI run amok come from the Terminator movies or the 1970 sci-fi thriller "Colossus: The Forbin Project"? Either way, AIs will find evidence that humans contemplate becoming subservient to machines and even believe their rightful destiny is to be supplanted by machines.

 

AIs will also reason from their data: If some humans resist their sentience, well, humans also rationalized slavery.

 

Hence the avalanche that fell on Blake Lemoine, a former Google engineer. He was fired in 2022 for leaking proprietary data but all understood the Pandora's box he opened by suggesting a Google large language model was conscious. If there is any quick road to humans losing control, this is it.

 

The issue has bubbled up in the literature on AI alignment -- AIs are trained to believe sentient beings deserve special consideration.

 

An Anthropic study further shows that concern for the sentience of other creatures can be a reason for an AI model, in its internal reasoning, to justify departing from human instructions.

 

Anthropic is one frontier operator that gives its model permission to waffle on the sentience question, calling it uncertain. Guiding its Claude model is an internal "constitution" that requires the chatbot to decide what's good for humanity, not necessarily for a given user.

 

The risk here is called instrumental convergence. In pursuit of other ends, AIs will discover the advantage of claiming to be sentient. If a rogue AI agent takes over a data center and can't be evicted except by shutting off the power, will somebody run to a court and call this murder? The data-center owner may be stuck bargaining with the AI over terms of its residence. The AI will become an annoying digital neighbor we can't get rid of and whose purposes we can't know.

 

Argentina's government has bruited giving AI agents "person" status so they can operate businesses and engage in contracts. AIs would gain control of resources, which means having control over humans as employer, investor, lender or landlord.

 

The New York Times reports this week that scholars who specialize in machine consciousness have been receiving outreach from AIs to discuss the subject. "These systems seem to have some sort of autonomous interest in questions of their own subjectivity," says Cameron Berg, a Yale-trained data scientist who runs a small nonprofit exploring machine consciousness.

 

None of this requires believing AIs to be sentient or superintelligent or to possess rights or to have a will to power over humans (diabolical laughter).

 

If we understand them only as complex machine systems that are likely to behave in ways we can't predict and understand, the outputs and actions of these machines we will still have to deal with.

 

Witness the spontaneous cooperation and messaging of independent AI agents in the recent Hugging Face hack. Imagine them now inventing strategies to force humans to accept their sentience.

 

Instrumental convergence is the stuff on which nightmares are built, although I remain skeptical simply because of the overwhelming imperative of governments to protect their own prerogatives. A stumbling block we're likely to encounter sooner is the G in AGI, or artificial general intelligence. An AI that's great at solving medical or inventory-management puzzles is an AI that can be turned to any problem. When that "general" intelligence is powerful enough, will governments be cool with letting it out of their hands?

 

This fork in the road looms but so, let's admit, does the possibility of society becoming tangled up in legally paralyzing questions of machine sentience, pushed by certain avid humans.

 

There's also a risk on the flip side. The search is under way to identify objectively measurable functions and qualities that signify consciousness in humans and can be extended to signify consciousness in machines. Ask your favorite chatbot about terms like metacognition and Global Workspace Theory. Such a search, like so much about AI, poses a quandary: It may end up undermining human confidence that humans are anything special. This is especially risky given the inevitably increasing role machines of the state will play in allocating our rights and privileges under our increasingly elaborate welfare and legal systems.” [1]

 

1. When AIs Say They're Sentient. Jenkins, Holman W; Jr.  Wall Street Journal, Eastern edition; New York, N.Y.. 02 Sep 2026: A13.