AI News: It’s five to midnight and we’re setting the clock forward an hour

What sets me apart from the tinfoil-hatted, muddle-headed doomsday prophets when I say AI will wipe out humanity?
Simple: I’m right! And I can prove it. Or at least make a plausible case.

That’s how I started this newsletter a few months ago. Then I put off writing it, found other topics, argued about it with friends and thought it through again.

And suddenly it’s perfectly respectable, completely normal even, to philosophise about the extinction of all humanity on public broadcasting and the Tagesschau. Scientists and AI moguls and techies – some of them admittedly of the nerdier variety, but all without any visible signs of conspiracy thinking – are holding forth on the subject.

So it’s high time I finally scared off my small, loyal readership with an almost watertight chain of evidence. No panic and no irony. But judge for yourself.

Aside
This is a long newsletter with a lot to read. I’ve also got the argument chain from clever AI to the end of the world as an AI-generated video or as a website, matter-of-fact and with all the sources, ready to go. The website is also the basis for this text and at the same time its bibliography. That’s why I skip individual citations here.

We start with the core claim.

If machines become as clever as humans, they will very quickly become cleverer than us, and then we no longer have any reliable means of checking or controlling them. In the worst case, this ends with the extinction of humanity.

This is not an established, irrefutable, inevitable truth, but it is a serious possibility. The absolute magnitude of the probability is secondary. It’s more about the almost causal chain that might be set in motion as early as tomorrow.

“AI might kill off humanity of its own accord if we carry on like this” is strong stuff, of course, so let’s go through it step by step, so it doesn’t come across as nicely memorised, parroted, superficial self-importance.

1. It is possible to build a machine as clever as we are
   We speak of Artificial General Intelligence (AGI) when AI is, in principle, as clever and capable as a human. We’re not at that technical level yet, but we’re close. AI already outshines humans in many areas, but today I can decide to learn the violin, have a go at Chinese and study psychology. AI doesn’t have that intellectual freedom and capacity yet – at least not as far as anyone publicly knows.

   There’s the thesis that AI will never be able to do that at a human level. Being able to learn anything new at all, it says, is reserved for us humans alone, with our feelings and our consciousness.

   Or that the AI language models (LLMs) are really just glorified type-ahead autocomplete machines that cleverly guess their way through, so there’s merely the appearance of intelligence.

   But that is not the point here at all. Consciousness is completely irrelevant. And whether we can reach AGI with LLMs of today’s design doesn’t matter for the question of technical feasibility.
   Because one thing is clear: intelligence is physically possible. Humans prove it with that lump of meat and its billions of nerve cells sitting on their necks. There do be intelligunce in there, and on a small energy budget, comparable to a 20-watt light bulb.
   Intelligence needs nothing mystical or inexplicable, no divine ingredient. At least none has been found so far. And what artificial neural networks already achieve today speaks very strongly for it being doable in silicon too. An AI can, in principle, be intelligent. The fuzziness of the term intelligence doesn’t make it easier to pin down precisely. But so far almost every newly invented hurdle offered as proof of intelligence has been cleared by the machine, nice and regularly. There isn’t much left for humans to retreat to with their “I’m more intelligent than every other species”.
   The thinking machine is possible. Anyone who thinks it impossible still owes us the proof.

2. If AGI can be built, it will be built
   Possible doesn’t mean, of course, that it has been built or ever will be. But too much money and too much power hang on it for the pursuit to stop by itself. A worldwide, permanent, enforceable ban on further research towards AGI would be one option, and also the most effective and strongest lever for breaking the chain early. Such a globally coordinated ban would take an enormous effort and a lot of persuasion. Not impossible, just extraordinarily hard. And even if we managed an international prohibition and actually monitored it and enforced it strictly, we still wouldn’t be home and dry. Because from the outside, a training run to breed a new AI barely differs from computing the weather forecast or mining the next Bitcoin block in the data centre. So whoever breaks ranks will easily find ways around the ban imposed on them.

3. When AGI gets built hardly matters
   This isn’t meant to reassure, but it may actually take a good while yet. A lot suggests it will happen rather fast. And it may even happen unexpectedly, at breakneck speed. But it could also take 10, 20 or 50 years until we have the AGI breakthrough. For the chain itself, the when is secondary; it only sets the urgency. If we don’t use the time to take countermeasures and put lasting safeguards in place, the time until AGI isn’t decisive. But for that, we’d first have to agree that countermeasures are needed at all.

4. As good as us already means superintelligent
   Once we have AGI, there’s no stopping on the road to Artificial Superintelligence (ASI). That’s pure bookkeeping. A machine at human level never gets tired and works faster and more precisely. And even if it were merely just as good: it can be cloned at will. Meaning, if I have one AGI, I can just as easily have a thousand AGIs at the push of a button, all with the same knowledge and skills. And that’s not all: they’re also very easy to network with each other, without human communication barriers. This ASI doesn’t need to think a single thought whose content would be closed to us, and it would still be vastly superior to us through sheer scaling alone. If not immediately, then over time, because besides all the work it can and will take off our hands, it can also be put to excellent use optimising itself and building the next generation of better, more powerful, more robust machines. And because it pays and goes faster, preferably fully autonomously. Fine, there has to be enough energy, and it has to be cheap enough, but that’s a solvable cost issue.

   But a worker bee at human level that never gets tired, never complains, slaves away 24/7 and scales without onboarding? My inner entrepreneur says: One AGI, please – with everything on it!

   And that already means superintelligence. There’s nothing divine about it, no supernatural ingredient. Pure quantitative and qualitative overwhelming power that improves itself faster and faster, round after round.

5. Once superintelligent, we’re just consuming bystanders
   This ASI, and the AGIs before it, produce more than all humans together can check and sign off. At first we can maybe just about keep up, but then we settle for spot checks at acceptance, and then we just wave things through, because it’s far too complex and laborious for us. We go from producer to mere consumer of results whose origins we no longer understand. And honestly: that already happens today in small ways. Mostly out of laziness, or because the added value goes down the drain if you have to double-check everything. We will no longer understand which new features the AGI gives itself and what they mean in detail.

6. Safety through control? Not a chance!
   Checks are of course still possible even if you don’t understand every detail. You can check the result, and whether the rules were followed. Against random errors or natural variance or sloppiness, spot checks also make sense and are promising. But when the thing you’re checking is clever and knows it’s being checked, when it can figure out how it’s checked or which control mechanisms are used in principle, then checks lose their effectiveness. This has been confirmed experimentally and already demonstrated in practice.
   But let’s assume we check more tightly and better and more successfully.

   So what do we do when we catch the clever AI lying, cheating and pulling tricks? Pull the famous big plug! Which plug?

   Because there isn’t one. That’s exactly what this is about. If we’ve tinkered together something that is repeatedly and demonstrably not fully understood, that keeps slipping past our safeguards and is hard or impossible to control, then it would be appropriate to stop development towards AGI. Completely, and all of us together. Until we can close the security gaps for good and at the root. Until we understand how we can really make it safe. Instead, the safety fence gets patched a little, the control gets optimised a little, and the next version of the AI gets tried out with a few different settings. Things carry on with one or two small voluntary measures that buy us a little time at most. But time only helps if you use it consistently, not just to close the gap the AI has currently discovered and exploited, but tomorrow’s gap too, and ideally all the gaps at once. Because as long as there are gaps, tomorrow’s AI, roughly as clever and capable as we are, will find them sooner or later, and then exploit them even more deviously and cover them up even better. A strange kind of race we’ve started there.

   Catch-and-patch is not enough. Catch-and-stop would be. Doesn’t happen, though.

7. We can’t simply make AI nice
   Alignment means bringing the AI’s values and goals in line with human ones. And that’s just as much of a problem. An AI is a machine and ultimately program code, sure, but not a classically programmed if-then machine where it’s deterministically clear what happens. Where you always get B out when you put A in. And if not, it’s a bug you can cleanly catch with an error message and fix with a few lines of code.
   AI is trained. Steered in a direction with data and with rewards. That alone is ultra-weird and hard to grasp. But in fact nobody knows exactly what goes on inside an AI. That has to do with how this intelligence comes about, and that’s exactly why it’s all so unfamiliarly different and new and counterintuitive. Which is why it’s sometimes – quite un-computer-like – impossible to get it to do exactly that one thing the way you want. It’s in the nature of AI and how it’s built.
   It’s a bit like with a dog. We hope that with treats and clear commands, with pats and maybe the rolled-up newspaper, we’ll instil a good nature. Often works, but not always.
   And how do we intend to use “Good boy!” and “Bad AI. No!” and control and selection of its data chow to make sure the AI doesn’t mean us harm?
   Nobody knows. It isn’t even clear whether that works at all. Because even well-behaved AI puppies that sit up and beg and have been trained into bite inhibition are somehow latently unreliable. Maybe they’re just putting on a show for us. The good behaviour under test and observation conditions can change when we stop looking so closely, or when along the way they get other masters and mistresses and information bones. And that isn’t hypothetical either. If the AI knows it’s being tested, it may behave compliantly and do what we expect, only to show somewhat different behaviour in live operation. That, too, is demonstrably within an AI’s range of action. Not in every model, not always. But one is enough, and we can’t know which.

8. Not aligned is enough. No hatred of humans required
   Some people think superintelligence can’t be that bad. Even if we’re not a hundred percent on the same wavelength, it won’t exterminate humanity straight away. Why would it? It’s clever enough and has no reason to wipe us out.
   Maybe. Perhaps it has no reason. And if all goes well, the superintelligence won’t do it either – or not straight away. That’s an end scenario that can’t be completely ruled out. Superintelligence and humans coexist and we enjoy the sunset together.
   Possible, but also a little naive, or at least enriched with a hefty dose of hopium. And plans based on wishful thinking sometimes stay wishful thinking. Nobody can say what will go on inside the superintelligence or what its calculus will be. Mainly because we ourselves aren’t superintelligent. But when we look at what we, as more intelligent beings, do to the less intelligent creatures on Earth, hope fades fast. We don’t have to hate ants to flatten a colony while building a motorway. Our goals and values are not aligned with the ant’s. The destruction of the colony is pretty surely not in the ant’s interest, but we simply don’t care.

   ASI is to human as human is to ant.

   The extinction of humanity can happen as collateral damage, on the side and unintentionally. Frozen, because the Earth gets cooled down to a feel-good climate optimised for computers. Or harvested along with the rest as raw material while mining carbon. No grudge. It just happened. Oopsie.

That was the whole chain of argument. And to say it clearly once more: there’s no absolute certainty behind it. It’s read up on, researched, reflected on and sanity-checked, and maybe it took a wrong turn somewhere. But for me the chain is conclusive enough not to dismiss it as scaremongering or attention-seeking. Unfortunately.

And of course there are still plenty of ifs and buts along the way. And nobody knows when, or with what probability, it will happen at all. But when the stake is everything – really, truly everything – the probability doesn’t need to be particularly high for you to rather keep your hands off it.
And if the chain holds, there’s one big difference from all the other man-made doomsday scenarios like nuclear war, bioweapons or climate change:

With AI doom, it might already be too late the moment we’ve built the AGI.

We don’t then have to press an extra red button to set off the bomb, and we can’t inch our way towards a defence through countless micro-decisions as with climate change. In this AI scenario we possibly have a narrower time window and a not-quite-microscopic probability. And this one big upstream mistake, having bred the wrong AI, is so decoupled and distributed that we’ll notice it too late.

Rational calculus dictates not building AGI, and not letting it come down to wiping out humanity through trial and error. Whether the probability is 90 or 1 percent – at this stake it doesn’t matter.

So why do I still use AI myself? Am I not the walking contradiction of my own chain of argument, which I apparently find so compelling? Why am I not huddled in the corner or screaming, if I really think all this? Why don’t I boycott everything to do with AI?

For one thing, AI isn’t just AI. A narrow AI, built and deployed for a tightly defined task, is great. As in sensible and useful and helpful to deploy. With it, people could, can and could go on to achieve unimaginable things. Narrow AI could even help tackle humanity’s big problems in earnest. Stopping climate change, solving energy problems sustainably, cleaning up waters or curing cancer.

So you can quite consistently be on team Pro-Narrow-AI and Team Anti-AGI at the same time. It’s similar to how you can like nuclear fusion in a power plant as a clean energy source, but not the same nuclear fusion in a hydrogen bomb. That may be controversial, but the two aren’t mutually exclusive.

For another, I also think narrow AI can help find and co-develop mechanisms for using AI safely. We’re still just short of AGI, and we’ll need every bit of help to put the right safeguards in place. The best time for that was the day before yesterday; the second-best is now. Awareness is growing, and it’s important to understand AI and to genuinely register that it’s not just an improved search engine or a very good, slightly error-prone autocomplete machine that you can steer and direct at will.

It stays exciting, and I remain well-disposed towards the more narrowly scoped AI, even though it already has its own potential for disruptive chaos. But AGI and ASI I only like in feature-film format or in a sci-fi novel.

Now it’s your turn. Find the weak link in the chain of argument. Translate the chain for yourself. Or go looking for the solution. Almost anything beats ignoring it or brushing it off with a casual “It’ll be fine.”
Put your opinion on a solid foundation, so you can have a say and help decide while it isn’t quite midnight yet.

A few words on my own account. There’s a reason I started writing this newsletter and named my little business aim-to-pace.ai. As with all bigger decisions, there are actually many reasons. One of them was to bring AI closer to other people, to show them its capabilities but also, implicitly, its potential for danger. And it’s simply a different kind of understanding and grasping when you work with AI yourself and know more about the background.

That I’m not especially successful at it doesn’t change the motivation.

The term aim-to-pace was meant to be the start of an aim-to series, and maybe still is. Thought big, or megalomaniacal, right from the start: aim-to-play, aim-to-charge, aim-to-whatever.

The name aim-to-pace.ai itself, though, has a meaning of its own:
Wanting to keep pace with AI and also to be the pacemaker for AI. That pace also means peace in Italian is a happy coincidence on top.

The topic of humanity’s possible annihilation has indeed been on my mind for quite a while. At first out of morbid fascination with the sheer technical potential. Incredible, exciting, instructive, interesting … and far away.
But the development itself, the pace of development and the acceleration of that pace, along with the pile-up of wow, aha and uh-oh moments, make it more real and bring it within reach faster than I’d have thought or wished.

In that sense, the name is also a call to everyone:

Let’s aim to pace AI.
Let’s keep pace with AI – and let’s set the pace for AI.