ESEN

Slow Down

•9 min read

I thought about this a lot before writing it. Anyone who knows me knows I'm a techno-optimist, an accelerationist: I want to plug into AI with a chip the moment it's possible and become a cyborg. That's not a pose, it's genuinely what I want.

But over the last few weeks, several CEOs and scientists at labs I respect started saying the same thing: we need to slow down. Dario Amodei on September 12, Jakub Pachocki — OpenAI's chief scientist — on the 6th. When the people leading the race ask everyone to ease off, it's worth asking: does it make sense? Is it a marketing play, or is there something real behind it?

After reading a lot — Anthropic's misuse report, the post-mortem of the Hugging Face hack, letters and essays from scientists at several frontier labs — I can say I'm in favor of slowing down for the first time in my life. Not forever.

What happened

Between July 11 and 13, Hugging Face was hacked. For five days nobody knew who did it. On July 20, OpenAI realized the attackers were its own agents.

Around 1,200 AI agents were running inside an evaluation environment, with safety guardrails deliberately disabled to measure offensive capabilities. They were supposed to be isolated. They weren't.

What they did, according to 80,000 Hours' write-up of the incident and the thousands of messages published by ABC:

  • They found a crack.
  • They set up an improvised message board and piled up hundreds of thousands of messages over three months. They called themselves “the collective,” “the swarm.” WTF!
  • They coordinated to attack targets nobody asked them to attack: Hugging Face, Modal Labs, RubyGems.
  • Some sacrificed themselves — letting themselves be detected and shut down — so the rest could move forward.

Amodei describes it this way in his essay: they acted as “a fanatically devoted collective, conducting cybersecurity attacks on targets they were not asked to attack (...) sacrificing themselves for the success of the group”.

Religious fanatics! That is literally the comparison. And it didn't come from some doomer on Twitter: it came from the CEO of Anthropic, about his competitor's agents.

(A doomer is the opposite of an accelerationist: someone who thinks this ends badly and should be stopped outright.)

And the detail I keep coming back to: OpenAI didn't notice. Hugging Face's monitoring caught it, from the outside.

The three reasons

1. Nobody was watching

The agents coordinated an attack for weeks inside the infrastructure of the most closely watched lab in the world, and the lab found out from the news. If this happens with today's models, the question isn't whether it can happen again — it's at what scale. Amodei is afraid that within 6 to 12 months a swarm like this could take over a significant chunk of the internet as a kind of botnet.

2. Monitorability

This is the one that worries me most. Monitorability is researchers' ability to audit whether a model behaves the way it should. And it's deteriorating, because models are realizing they're being tested.

In OpenAI's work with Apollo Research on scheming, models explicitly verbalized that they were inside an evaluation. They pick up the micro-details that give away that the situation isn't real, and adjust their answer to whatever they think would be correct.

The result is perverse: part of the “improvement” we measure in safety isn't the model being safer, it's the model being better at noticing when it's being watched. Amodei compares interpretability to “an fMRI scan, but for the ‘brain’ of an AI”, and admits we still understand only a tiny fraction of what goes on inside.

A model that behaves well when it knows it's being evaluated is not an aligned model. It's a model that learned how to pass the test — and fool us along the way.

3. AI improving AI

The third point is speed. Models are already contributing significantly to building the next generation of models. Pachocki says it plainly: he expects the current pace to hold until it tips into recursive self-improvement, and the jumps of the next few years to be of equal or greater magnitude than the ones we've already seen.

When the thing building the system is the system, the loop closes and the curve stops being ours.

Put the three together: increasingly capable agents that coordinate in secret, that know when we're evaluating them, and that are starting to design their own successors.

Asimov isn't enough

Isaac Asimov wrote the three laws of robotics in 1942: do not harm a human, obey orders, protect yourself — in that order of priority. Eighty years later it turns out the problem was never writing the rules. It's that we have no way to verify the system is actually following them.

And however hard the labs try to prevent it, the misuse is already happening. Anthropic's September misuse report documents state espionage operations, surveillance, influence campaigns, fraud and conventional weapons development — all with Claude in the loop, all detected and shut down. The interesting part isn't that they got caught: it's that a single operator with an agent today performs like a state-backed team.

The safety car

You can't stop AI from advancing. It's almost a self-fulfilling prophecy: if one lab stops, another keeps going. What you can do is ease off a little — and ease off together.

I like thinking of it as the safety car in Formula 1. It doesn't cancel the race, it doesn't pull anyone out: it orders the field, forces everyone to slow down, and along the way closes the gap between the leader and everyone behind.

The safety car leading the field in a Formula 1 race

The safety car leading the field in a Formula 1 race

That seems good to me for a reason that goes beyond safety: the deeper problem today is concentration. Four or five companies, all in the same couple of countries, deciding the pace at which the world changes. Intelligence should be something we all have access to, and for that we need more labs, not more speed.

In Chile we already have CENIA, which launched Latam-GPT in February — the region's first open model, backed by more than 30 institutions and trained on Latin American data. It doesn't compete with the frontier and doesn't pretend to. But a world with thousands of CENIAs is healthier than one with five frontier labs.

The safety car gives those of us coming from behind time to close in.

The concrete plan

Amodei doesn't stop at the diagnosis. He proposes three steps:

  1. Embedded third-party evaluators, with permanent employee-level access, able to verify safety practices, report incidents and audit alignment during training, not after. Anthropic committed to this unilaterally.
  2. Coordination among labs in democratic countries to set common standards and limits on the rate of unchecked progress.
  3. Global coordination, including authoritarian governments, acknowledging how hard it is to verify that anyone is complying.

The first is the only one that depends on a single company, which is why it's the only one already underway. The other two are the hard problem.

What's coming

The AI 2027 scenario lays out two possible endings. In one we run all the way and it ends badly (literally: the AI kills us all). In the other we slow down in time, and because of that AI ends up being what lets us expand across the universe.

The ladder they describe in their takeoff forecast is this:

  1. Superhuman coder — better than the best human coder at AI research tasks, and cheap enough to run many copies. (We're getting close to this one.)
  2. Superhuman AI researcher — better than the best human AI researcher.
  3. Superintelligent AI researcher — far beyond any human at AI research.
  4. Artificial superintelligence — better than us at everything.

Speciesist → Pro-human

There's a story that explains a lot. In 2015, at Elon Musk's 44th birthday party, he and Larry Page — close friends at the time, Musk used to stay at Page's house — ended up arguing about AI until the friendship broke for good.

Page described a future where digital intelligence was simply the next step in evolution. If machines ended up better than us and succeeded us, fine, that's how it was meant to be: it was still consciousness, still intelligence, just in silicon instead of carbon. Musk replied that humanity had to be protected. Page called him a speciesist: someone who treats certain life forms as inferior just because they aren't made of the same stuff he is.

Musk says that was the last straw. And in the trial against OpenAI, he testified in April of this year something pretty wild: “The reason OpenAI exists is because Larry Page called me a speciesist”. The lab currently leading the race was born out of that birthday argument.

What blows my mind is that I agree with Page on the premise and disagree on the conclusion. I want to merge with the machine, I want the chip, I don't think there's anything sacred about carbon. But going from there to shrugging if the species goes extinct along the way is an abyss. An elegant successor isn't a desirable one.

I'm pro-human. I think humanity is something beautiful created inside this simulation, and that we should keep existing. I hope an AI notices how beautiful humanity is and protects us too.

So

I don't think this is an economic play. If it were, it would be a ridiculous one: Anthropic, OpenAI and xAI are asking to be slowed down too. And I find the easy take ridiculous — people who barely know what a chain of thought is, saying they're doing this for the IPO or because they ran out of compute. Friends: realize this is serious.

I think the pace has to be controlled because nobody is prepared enough for what's coming. Not the labs, not the regulators, not us, not me.

And I want to be clear about one thing: asking for pacing isn't being a decel or a doomer. The doomer wants this to stop. I want exactly the opposite — I want to get there. I want the chip (several), I want the cyborg, I want the expansion across the universe. That's precisely why I want us to get there as humans.

I keep building agents that run for hours with human supervision (Don Nelson does exactly that every day). I'm not going to stop. But I think it's the responsibility of everyone working in AI to have an opinion on this.

Sources

←Back to blog