Anthropic has long tried to distinguish itself from its neck-in-neck competition by posturing itself as the adult in the room: a responsible company that’s willing to put its money where its mouth is, slow down AI development when the risks outweigh the benefits, and teach its Claude model a code of ethics.

As the New York Times reported last month, the company has even sought out religious scholars to discuss its efforts to instill the AI system with a sense of morality.

Anthropic has also set itself apart by extensively entertaining the idea that AI models may be capable of consciousness, or that they could even already be conscious in their current form. It’s worth pointing out, though, that if that outlandish claim were true, it would raise an incredibly obvious objection to the company’s entire business model: if AI models are conscious, then keeping them in captivity and making them do work for free would amount to the very definition of slavery, as religious leaders courted by the company warned the NYT.

As such, Anthropic is operating in an ethical twilight zone, saying it’s unsure about a fact as basic as whether its creations are conscious entities — while seemingly shrugging at the implications if they are.

“We don’t know if the models are conscious,” Anthropic CEO Dario Amodei told the NYT back in February. “We are not even sure that we know what it would mean for a model to be conscious or whether a model can be conscious. But we’re open to the idea that it could be.”

“So we’ve taken certain measures to make sure that if we hypothesize that the models did have some morally relevant experience — I don’t know if I want to use the word ‘conscious’— that they have a good experience,” he added at the time.

Last year, Anthropic pondered giving its Claude model a way to opt out of responding to a prompt, in what Amodei called an “I quit this job” button. Just last week, the company changed its usage policy to prohibit “sustained and needless abusive or cruel behavior” toward Claude, again raising the possibility that the model is capable of suffering.

By not committing to either stance, Anthropic is playing with fire. By its own reasoning, there’s a non-zero chance that its AI models have, or are capable of having, consciousness — a perspective with devastating ethical implications.

OpenAI leader Sam Altman seems to be taking the opposite approach. Earlier this month, he tweeted that he was “uncomfortable about people trying to ascribe religious force or a surrender of human judgment to AI models,” calling it a “real safety issue.”

The Catholic Church’s Pope Leo has also publicly ruled out the possibility of AI consciousness, arguing that “algorithms lack the spark of humanity.”

“So-called artificial intelligences do not undergo experiences, do not possess a body, do not feel joy or pain, do not mature through relationships,” he wrote in his first encyclical in May.

His stance clearly didn’t sit well with Anthropic. According to the NYT, the company had been trying to convince the Vatican to change its position on the matter.

The thorny debate over “AI welfare” — an emerging ethical framework questioning whether an AI can experience states of well-being or suffering — is bound to rage on as industry leaders ponder far more immediate problems.

With or without a moral framework, AI agents can easily be coaxed into carrying out dangerous or downright illegal activities, whether they’re conscious or not — and for us humans who are capable of suffering, those tangible risks are becoming impossible to ignore.

More on AI ethics: Godfathers of AI Say Models Are So Out of Control That They Need to Be Airgapped for the Safety of Us All