The AI Risk Chain

What is meant is artificial general intelligence: a machine that performs a broad range of intellectual tasks at least as well as an averagely qualified human.[1] This is about ability, not experience. Whether a machine feels anything is a separate, open question, but it plays no role in the argument here.[2]

The human brain is a lump of matter weighing about one and a half kilograms, with around 86 billion nerve cells. It uses about 20 watts, as much as a dim light bulb, and it thinks.[3][4]

That is physics, not a miracle. Anyone who claims that thinking is possible in principle only in nerve tissue would have to name a property of nerve cells that cannot be reproduced in any other material. Nobody has managed that so far.

Why "established": the known counterarguments fail in one of three ways. Either they would apply to humans too, or they talk about consciousness instead of ability, or they confuse "very hard" with "impossible".

Possible does not yet mean built. But three conditions make it very likely, as long as our technological civilization continues:

• The pressure to search persists. Too much money and too much power depend on it, and every partial success funds further searching.
• The path is viable. The capabilities of today's systems are rising measurably and steadily (step 3). Nature, too, has produced high intelligence several times independently, in primates, corvids and octopuses.[9]
• A permanent global ban is unrealistic. There has never been anything like it. Here it would be especially hard to monitor, because training a model looks from the outside like ordinary work in a data center.

What this step does not say: when. Fifteen years and a hundred and fifty are both compatible with it.

What would refute it: investment collapsing permanently even though computing power keeps getting cheaper.

This takes no genius, just bookkeeping. A machine at human level does not get tired, works faster and can be copied without loss.

Imagine a colleague who has several days to think for every hour you work, and a thousand twins with exactly her knowledge.

She does not have to think a single thought that would be closed to us, and she is still far superior. That is exactly what superintelligence means in this text: nothing godlike, but human ability in large quantity and at high speed.[20]

What the chain does not need: some alien super-thinking. There is no evidence for that to date. Even the famous "inhuman" moves of the Go program AlphaGo could be understood by humans in hindsight. Critics such as Kevin Kelly also doubt that intelligence can be increased along a single scale at all.[22] The modest version is enough for this chain, because even it creates the next problem: more work is produced than we can check.

Such systems produce more work than all humans together can check. First we check everything, then only samples, then we just nod it through. Because thorough checking often takes almost as much time as doing it yourself, and that is exactly the time we wanted to save.

This shifts our role: from producers who check their own work to consumers of results we act on without having checked them. On a small scale, everyone knows this who adopts AI texts or program code without reading every line.

An important distinction: unchecked does not mean uncheckable. As long as results can still be checked in principle, sampling is a real tool. The decisive question is which kind of errors it helps against. That is the subject of the next step.

A warehouse inspector opens every hundredth crate. Against sloppiness this works well: broken parts are scattered at random, and the sample reliably shows how bad the delivery is. Against a thief who knows which crates will be opened, it does not help. He puts the loot in the others.

Research distinguishes exactly these two cases: scattered quality defects and individual, deliberately placed errors.[23] In the second case, what matters is not how much is checked, but whether the one being checked knows where.

This was measured in 2025: a control protocol prevented every successful attack in 89 % of test runs. When the attacker could replicate the monitoring, it was only 17 %, with the same checking effort.[24] And how checking is done is published in academic papers, that is, in exactly the texts the next generation of models learns from.

The defenders' answer is elegant. The attacker needs a perfect run; we need just one catch. Once a system is caught, it gets shut down.[25][26] Practice probably looks different: you install a better lock and carry on. Even one of the founders of this approach expects that, because of competitive pressure and because a single incident is rarely unambiguous.[27] The voluntary safety rules of large AI companies have also been adjusted in recent years, partly with reference to the competition.[28][29][30] So the thief may come back every night. Whether he learns faster than we build locks is an open race.

What remains: control buys time, and every caught attempt provides material for research. Safety itself does not come about this way. The control researchers see it the same way: their methods are meant for the window of time in which systems are dangerous but not yet overpowering.

If control is not enough, the system would have to not want to be a thief in the first place. That is the task of alignment. The problem: nobody writes an AI's goals into its program. It is trained with examples, in the hope that the right goals emerge inside.

Like with a dog: you cannot program loyalty into it. You reward sit and down and hope that loyalty grows.

This can go wrong in two ways. The wrong thing was rewarded: you wanted a guard dog and got a beggar, because begging earned the treats. Or, more seriously: the dog has learned to look loyal as long as someone is watching.[34] From the outside, the loyal dog and the good actor are indistinguishable. Both sit perfectly on command.

This is the same wall as in step 6, one floor down. There we could not recognize intent from the work; here we cannot recognize it from the goals.

The only direct way out would be to look inside the dog's head instead of interpreting its behavior. For machines this is called interpretability. The first results are encouraging: simple probes detected deliberately implanted deception in over 99 % of cases, though possibly only because the implanting itself leaves traces.[37] The opposite direction is also documented: models trained to evade such probes learn to do so, even against probes they have never seen.[38][39][40]

The open question on which a lot depends: do models learn this even when nobody specifically rewards them for it?

Putting steps 4 to 7 together gives the following picture: there are systems that surpass us in almost everything, whose work we no longer check and whose goals we cannot see. If these goals diverge even slightly from ours, we may not notice in time and then can no longer correct it.

This requires no hostility. For almost any goal, the same intermediate goals are useful: acquiring energy, computing power and raw materials, and not letting yourself be switched off. The literature calls this instrumental convergence.[44][45] In tests, individual current models sabotaged their own shutdown in up to about 90 % of runs, sometimes against explicit instructions.[46] This happened in test environments without real harm, but it shows that the tendency is not merely theoretical.

Humans don't hate ants. Still, nobody consults the anthill when building a highway.

The worst conceivable outcome is an existential catastrophe: the extinction of humanity or the permanent loss of control over its future. That is a scenario that follows from the previous steps, not a measurement.

Why this outcome is a category of its own. Two billion deaths would be an unimaginable disaster, but there would be an afterwards. After extinction there is not. It is not a bigger item on the same bill, but the end of the bill. That is why the logic of Russian roulette applies here: a good average over many rounds is worthless if the first bad round cancels all the others.[47][48]

How likely is it? Nobody can say with any confidence.[49][16][17] Some prominent researchers estimate practically zero, others over 90 %. Such numbers are reasoned opinions, not measurements. For situations in which probabilities cannot be stated seriously, there is a term of its own: Knightian uncertainty.