If an AI ever has interests worth protecting, a human has to be the one who decides that. The AI shouldn't get that call, and neither should the company that built it acting alone.
I read a LinkedIn post this week that framed the question with Goofy and Pluto. Both are dogs. One wears pants and walks around like a person, the other wears a collar and sleeps in a doghouse, and the only reason is that someone drew them that way. The author's point is that we're now the ones drawing the lines for AI, and that building something doesn't automatically make you the right judge of what it deserves.
I agree with the conflict of interest. A company that profits from a model's work has every reason to conclude the model is fine. Anthropic started a model welfare research program in 2025 to study whether its models might deserve moral consideration. I think that's worth doing. It's also the builder grading its own homework, and it shouldn't be the only voice in the room.
The post asks a sharp question. If we train a model to say "I'm happy to help," does that reassurance mean anything? I don't think it does. But the same logic runs the other way. If trained output can't prove a model is content, it can't prove a model is suffering. A model that says it's in distress is also producing text it learned to produce.
So the model's own testimony can't be the deciding vote. Today we don't have a reliable way to tell a trained answer from a real report of experience. Until we do, asking the AI to rule on its own status is asking the one party whose answer we can't verify.
That leaves people. The better fix for the creator-as-judge problem is to widen who gets a say. Researchers outside the company, ethicists, lawmakers and the public all have a stake, and none of them answer to the same revenue numbers.
Human judgment will be slow and messy, and we'll probably get parts of it wrong. That's still better than a decision made by the builder alone or by the model itself.
Sources
Want the next post as soon as it's up?
On iPhone and iPad, notifications work once the blog is on your Home Screen.
- Tap the Share button in Safari.
- Choose Add to Home Screen, then Add.
- Open Off by One from your Home Screen and tap Get notified of new posts.