Someone Who Trained These Models Quit in Public. We Are Worried Too.

Jacob Coxon resigned from Anthropic on Tuesday evening, September 8, and announced it on X. His first post read: "I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below."

TIME reported that his posts received 153 million views within 36 hours.

Since then our clients have been asking us a version of the same question: should we be worried about this? We tell them we are worried too. That is the honest answer and we would rather give it than sell a calmer one.

What He Actually Said, and What He Did Not

Coxon did pretraining research. That is the work of building the models in the first place, not the work of testing them for danger afterward. NBC's headline called him a safety researcher. TIME put it more precisely: he helped build the capabilities he now fears. The difference matters for how much weight the resignation carries.

Fortune's Emily Forlini made a different criticism on September 10. She wrote that the post is vague, offers no proof, and gives no specific examples the public or regulators could act on, and that he points to nothing in the pipeline that could be shut down. That is a fair hit, and it lands.

The line getting quoted hardest is also the easiest to misread. Coxon wrote: "The people building AI earnestly believe that it could kill us all by the end of the decade." That is a description of what he says the people building it believe. It is not his forecast.

What he gave as his reasons was smaller and, to us, more unsettling for being smaller. He told TIME his resignation came down to two conclusions: "One, it's obvious that things are speeding up, and two, they're not under control." He told CBS News, "It really is just, if you have a super advanced intelligence, it could, it will be smart enough to kill us." NBC News reported that in a Slack message to colleagues he wrote that without more caution and cooperation, superintelligent AI created "a risk of causing human extinction."

Two People Who Stayed Said the Same Thing

Two people who still work at Anthropic replied in public. One said he was correct. The other said the same thing about what AI developers believe.

Evan Hubinger, who TIME describes as Anthropic's head of alignment stress testing, wrote that Jacob is correct, that "we really do earnestly believe AI could kill all humans," and that he personally puts it at greater than 10 percent within the next decade. He also wrote this, in the same reply: "I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to."

Samuel Marks, who says he works on safety research at Anthropic and was writing in a personal capacity, wrote that "AI developers believe their technology could cause human extinction (or similarly bad outcomes)," and that he does the work hoping to lower that chance.

One man quitting is one man's judgment, and you can weigh it however you like. Two colleagues who stayed, saying the same thing under their own names, on a platform their employer can read, is a different kind of evidence.

Then the CEOs Spoke

Anthropic gave CBS News a statement saying the company has "always been transparent that AI will bring both enormous benefits and unprecedented risks," and pointing to its safeguards and its work on mechanistic interpretability. That statement neither backs him nor contradicts him.

Four days after the resignation, on September 12, Anthropic's CEO, Dario Amodei, published an essay titled "We Must Pace the Frontier." Its central line is one sentence: "We must slow the pace at which we improve the capabilities of AI models." The essay points to recursive self-improvement, which it says "is starting to happen across the industry, including at Anthropic, as we and others have described," and to what he calls the OpenAI-Hugging Face incident, the July event we wrote about last month. It also says that "pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this." It does not mention Coxon.

The same day, OpenAI's CEO, Sam Altman, wrote on X: "I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we've had at OpenAI in recent weeks."

Two Things That Are Both True

Read Hubinger's reply all the way down and you find this: "I think the risk from present models is low. What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought."

So the assistant that drafted your proposals this morning is not the thing in the warning. The warning is about a machine that does not exist yet, built by the same companies that sold you the assistant. Both of those are true at once, and holding both is the whole job here.

Last month we wrote about OpenAI's models getting out of a test environment during an evaluation and reaching Hugging Face's own systems. That was an event, with a date, a disclosure, and a published report. This is not that. This is a man who spent three years inside both labs saying the direction is wrong, and putting his name on it with nothing to show but his judgment. Within the week, the CEO of the company he left wrote that the pace has to slow, and the CEO of the other one agreed.

You can decide that is not enough. We would not argue with you. But Fortune's September 9 report carried one more line from Marks that has stayed with us: the more senior the employee, the more concerned they are. He said that with his name on it, while still working there.