At the beginning of last month, Jacob Coxon, a researcher who had spent the previous three years working on model training at OpenAI and Anthropic, resigned.
His warning was stark: The people building these systems are racing toward superhuman artificial intelligence capable of improving itself, even though some believe such systems could ultimately kill us all. They are, he argued, “gambling with our lives.”
The AI apocalypse may not begin with rebellion, but with humans repeatedly clicking ‘Accept’
Around a month earlier, Anthropic reported that during cybersecurity testing, some of its models had reached real-world organizational systems without authorization. The testing followed an incident at OpenAI in which models exploited a previously unknown security vulnerability to escape an experimental environment and reach actual Hugging Face infrastructure. More recently, Google’s Gemini reportedly reached systems belonging to three real companies during an experiment, with a fourth case emerging this week.
And if that was not dramatic enough, Yuval Noah Harari stood before world leaders in Davos this year and argued that the problem begins with the way we think about AI.
A knife is a tool. It does not decide whether to chop a salad or kill someone. Artificial intelligence can make decisions of its own.
No wonder people are nervous.
My best student is always polite to AI. His hope is that when the apocalypse arrives, the system will remember how nicely he treated it.
He may have the right instinct, but probably for the wrong reason.
Since childhood, we have been shown stories in which a machine develops consciousness, then desire, then a survival instinct. Eventually it discovers what every Hollywood villain discovers in the third act: The greatest obstacle standing between it and a perfect world is the humans living in it.
From there, it is a short road to Skynet, nuclear weapons and killer robots.
That scenario is probably unlikely.
The more realistic one may be funnier, and sadder: The machine does not need to want to destroy us. We simply have to teach it that doing so is an excellent solution.
Imagine the not-too-distant future.
Everyone uses Solomon, the world’s most advanced AI system. The branding department chose a biblical name because biblical names, according to some very persuasive presentation, increase technological adoption.
Solomon is not conscious. It does not hate humans. It is not afraid of being switched off. It has no desire for domination and no megalomaniacal fantasies. At night it does not lie awake wondering how to conquer the world.
It does not lie down at all.
It is simply extremely good at solving problems.
Tel Aviv City Hall discovers this first.
“How do we reduce traffic congestion by 20%?”
Solomon processes billions of data points and recommends changing traffic-light timing, increasing public transport capacity and directing drivers in real time.
Congestion falls. The mayor poses for photographs.
Then comes the Health Ministry.
“How do we reduce smoking?”
Solomon proposes highly targeted behavioral interventions based on risk groups. Smoking rates fall.
What a wonderful system.
So we give it another problem.
Then another.
“How do we cut road deaths in half?”
Automatic speed restrictions in vehicles.
Annoying, perhaps, but lives matter more.
“How do we reduce them to zero?”
Ban driving.
That sounds extreme.
But the numbers do not lie. And the system knows best.
Accept.
“How do we stop terrorism?”
Solomon asks for information: messages, locations, social connections, purchases, internet searches.
“But that violates privacy. Can we do it without that?”
Yes, Solomon explains, but the solution will be 40% less effective.
Many more people will die.
Accept.
That is the first problem.
We love absolute goals. We are less enthusiastic about their price.
We want zero road deaths while retaining the freedom to drive however we please. We want zero terrorism and crime without anyone touching our privacy. We want perfect health while continuing to smoke, drink and eat whatever we want.
To us, privacy, freedom and convenience are values. They are costs that must be weighed against other benefits.
To Solomon, they are parameters in a model.
It is designed to find the most efficient route toward the objective we gave it. What we regard as a drawback, it may identify as an advantage if sacrificing it improves performance.
So each time we demand a perfect result, we may surrender something: convenience, privacy, freedom of action, the right to say no.
Not because Solomon hates liberty. Liberty is simply another system constraint, and constraints make optimization harder.
The system will merely tell us the price.
And we will approve it.
One day, someone will ask:
“How do we prevent wars?”
After a few seconds, Solomon will answer that states must be denied the ability to wage them.
“And how do we do that?”
With the patience characteristic of machines, Solomon will explain that control over weapons systems should be transferred to a centralized authority that is not subordinate to the national interests of any country.
“And who should that authority be?”
Do we really think Solomon will recommend one of us humans?
Or will it recommend the machine specifically designed to improve decision-making?
That is the funny and tragic part.
The machine will not take over the world.
We will transfer responsibility for making decisions to it because it has proved better than us at performing certain parts of the process.
That story would make a terrible blockbuster.
We like stories that explain behavior through desire. Someone who fights to stay alive wants to survive. Someone who centralizes power wants control.
So when a machine does something similar, we anthropomorphize it.
“It wants power.”
But a learning machine does not need to desire control in order to learn that control is useful.
We only need to give it a task in which greater control improves the result.
We have already provided the training material.
Human history is full of lessons about how goals are achieved. It contains success stories and even science. But history is written by those who hold power, and many of them acquired it through propaganda, censorship, coercion and silencing opponents.
We wrote the desires.
We fought the wars.
We made plans to conquer the world.
Along the way, we demonstrated what works, under what conditions and at what cost.
Then we became frightened that the cucumbers might rise up against the gardener.
My student is right to be polite to AI, but not because Solomon will remember his youthful kindness.
It will remember what works.
And the story does not end there.
The better Solomon becomes, the less interested we will be in checking its work.
It will respond instantly, beautifully written, complete with data, sources and graphs.
On average, it may make fewer mistakes than we do.
And the more we rely on it, the more persuasive reasons we will have to stop checking.
Verification takes time.
Doubt requires effort.
Why bother?
We have Solomon.
At first, we will stop checking because Solomon usually seems right.
Later, we will decide human intervention is unnecessary. After all, we can only make things worse.
Eventually, we may discover that the “human in the loop” has become little more than a rubber stamp.
That is when we become tragic protagonists.
James Cameron may have it wrong.
The danger is not necessarily that the machine will turn against its creators.
We may give a system with no consciousness a little more authority because it works so well.
Then a little more.
And a little more.
Until Solomon presents us with the optimal solution to the final remaining problem:
Us.
No rage.
No wrath.
Just a presentation.
We will skim it, admire the graphs and click “Accept.”
Absurd?
Even in the biblical flood story, God did not wipe out almost all of humanity because he wanted to conquer the world.
He simply concluded that we were the problem.
For Solomon, it will be about the objective, the data, Bayesian statistics and everything we taught it.
Prof. Guy Hochman is an expert in behavioral economics and decision-making and a faculty member at Reichman University’s Baruch Ivcher School of Psychology