AI is the master of every error class

Accepting error as such as unavoidable. Really, deep down inside. Check! And then along comes some hyperactive super-clever young person flooring it on YouTube or dumping a kilo of Insta in your lap by offering you the all-purpose universal miracle cure for the AI’s incurable case of error-itis. We shouldn’t just accept that.

Don’t get me wrong, I enjoy watching that sort of thing, there’s plenty of useful stuff in there, and a bit of showmanship comes with the trade.

But honestly.

• This one trick stops AI making mistakes
• This is how you force Claude to always tell the truth
• Only 0.000000001% of people use this oracle hack

How can they claim that, when I’m the one who has the solution for dealing with AI errors, all by myself. And for me personally I actually do. Not as the one law for everything, the one knack that always works. Because there is no such thing. Google, OpenAI and Anthropic didn’t just implement it sloppily. And nobody pulls it off with a hack or a trick either. AI errors are simply, fundamentally unavoidable. That holds for fellow humans and all other fellow intelligences — animals, AIs and assorted aliens.

If avoiding isn’t an option, then deal with it as well as possible instead. And there are principles and methods that do justice to the core of the problem. More justice, at least, than any one-click miracle that works about as well as every other perpetual motion machine. I don’t wish the guy any harm — he makes rather successful and rather good videos — but he’s offering a medicine without explaining the disease. The problem: it even works, which doesn’t make it harmless. It creates dependency, because it leaves the consumer uninformed. The patient information leaflet is missing entirely.

The promise that something infallibly helps against all errors, always, is modern-day quackery.

But above all it lulls you into false security, precisely because it does work and does sound logical and does contain a huge kernel of truth.

Let me explain with a concrete example. Said YouTuber goes all in on a skill-based system that gets Claude to run an answer through the mincer of 10 — in words, ten — agents. One agent rethinks the question from scratch, one tries it in a different context. One plays devil’s advocate. Another verifies or falsifies the statements again. One negates the question and asks for the opposite, plus further moves. And once the 10 agents are done, the answer is error-free. That’s the claim, anyway.

But the opposite of good really is well-meant. And all 10 agents are individually sound and have their fair place in the fight against AI errors. You may and can and even should combine them. But throwing everything into a blender, giving it a good whirl and selling that as the solution against errors is dishonest — and, viewed on its own terms, ironically a systematic error.

Here’s my reasoning for being against this kind of all-in-one solution.

1. It isn’t economical. I’m no fan of insisting on token thrift. Not because I like throwing money out of the window, but saving has to come later in the order. When you’re learning to drive — where everything converges: traffic rules, unfamiliar sequences of movement, situational and reactive peak performance — it isn’t helpful to build fuel economy into your very first practical lessons and forget the indicator or the shoulder check over it. That can be a topic later, once you can drive. So this really is my weakest argument against the 10-agent error terminator. But siccing 10 agents on every trivial little problem is genuinely over the top.
2. It lulls you into false security. By giving the impression of being the simple and infallible solution — which it just isn’t! — it leads to neglect of due diligence and of checking the result. Risk overcompensation. Which may even push the overall system to a higher risk level, because the last checking instance is simply left out or turns out too flimsy.
3. It’s wrong. The decisive and indisputable knock-down argument. It cannot possibly avoid all errors. Can’t be done. Not all of them anyway, and in this way three times not. Because the agents are correlated, i.e. not independent — they’re of the same build. Ten articles by the same investigative journalist in different outlets don’t make his story any truer. Checking yourself is better than not checking at all, but it is certainly — as in, with a real scientific guarantee — not an error-free system. And that’s the big, genuine problem I have with it. It systematises this error and sells the result as error-free. Mislabelling is the mildest thing that comes to mind.

But enough griping about what other people don’t do perfectly. The point is really to get an overall system as error-free as possible, given that the individual LLM can’t manage it. A system in which the AI contributes its valuable part.

And so what is my oh-so-wonderful system that makes everything better. Why is my method worth more, and where and how can one check that for oneself, Mr Smarty-Pants. Mr Know-It-All, your entrance please.

Gladly.

Step 1: Background understanding I hopefully built that up in my last post. Do read it if you haven’t yet, or just take it on faith now. A learning something — human, animal or machine — has to make errors, otherwise it would be incapable of learning. So eliminating errors completely, forever and everywhere, is a dead end. Fact, and done.

Step 2: Find analogies AI can learn and makes errors. But we’re already great at dealing with learning, error-prone human beings. We’ve trained and optimised that on ourselves for long enough.

Just a small selection of the arrows we have in our quiver in the fight against errors. All of them real and concretely in use. And there are many more:

[This paragraph is AI-generated — start]

Poka-yoke, rubber duck, dead man’s switch, four-eyes principle, canary, shisa kanko, Swiss cheese, pre-flight check, Andon cord, smoke test, red team, positive locking, checklist, fail fast, read-back, interlock, dry run, kill switch, plausibility check, pair programming, sterile cockpit, watchdog, sleep-on-it rule, check digit, devil’s advocate, rollback, point-and-call, two-hand control, pre-mortem, sampling, jidoka, countersignature, chaos engineering, back-of-the-envelope check, buddy system, forcing function, cashing up, guardrails, dress rehearsal, two-man rule, digit sum, safety margin, heartbeat, regression test, belt and braces, dual control, sanity check, 5-why, stage-gate, working backwards, circuit breaker, cooling-off, crew resource management, backup, fishbone, tripwire, A/B test, redundancy, checksum, FMEA, near-miss reporting, safe defaults, reconciliation, blameless postmortem, drift detection, sign-off, defence in depth, speak-up culture, graceful degradation, just culture, feature flag, visual inspection, fail safe, debriefing, undo, stocktake, threshold alarm, MOT

[This paragraph is AI-generated — end]

Step 3: Simplify and transfer Let’s take what we humans have worked out as armament against errors and transfer it to AI, paying particular attention to the few un-human AI quirks. We don’t have to check and adapt every single error procedure for AI suitability. But the underlying principles behind the long list can be boiled down to a few — along three questions:

• How do I prevent errors?
  • Make errors impossible (poka-yoke): Where possible, the strongest lever of course — don’t catch the error, rule it out by construction. The plug that only fits one way round.
  • Simplification: If it can be done more simply, do it more simply. Every superfluous part is an additional source of error — standards and automation clear up.
• How do I detect errors?
  • Redundancy: Ask the same question in independent contexts. If several independent routes make the same error, it’s unlikely — if they make different ones, it shows.
  • Verification: Build in checkability from the start, don’t hope for it afterwards. What can’t be checked can’t be corrected either.
  • Fail early and small: Small steps, fast feedback. The same error is a correction after an hour and a catastrophe after three months.
  • Competence: The human keeps the responsibility — and with it the ability to recognise errors at all. Whoever only signs off forgets how to check; inspection fatigue hits experts and laypeople alike.
• How do I survive errors, or how do I handle them?
  • Damage limitation: If the error happens anyway: keep the effect small and be able to undo it. Isolation limits damage in space, reversibility in time — a reversible error is a learning moment, an irreversible one is damage.
  • Error culture: Treat errors openly instead of sanctioning them. Whoever hides errors doesn’t prevent the error — only the learning from it.

And part of that can be applied directly and effortlessly in everyday prompting. Simple, concrete, effective.

Simplification — slice the elephant The AI is a generalist and can do everything fairly well, but not everything well at once. The solution is so close. Because the AI is also a specialist, if you simply tell it to be one — and suddenly you have a team of specialists you “only” have to orchestrate. So instead of cramming everything into one giant prompt, tie it into small, optimisable packages. A bit of “what if I had a whole company do this?”. Explained with the concrete example of my editorial process. I write and publish this Substack newsletter you’re reading more or less regularly. And yes, I write it all myself and they are my words and sentences. But AI does help me, of course. For one, in research. I have an idea or a thought I want to develop. So I gather information and bundle it. Sometimes so much so that I build a whole website — a side effect of the newsletter process. Then I write it down. Just like that, with an appalling number of typos, genuine gaps and transpositions. The pure typos are handled by my AI cleaner — real errors only, and it’s the only one allowed to change the text directly. Only then do I work up the courage to hand it to my AI coach. It adjusts nothing but gives its comments. Does the story carry? Does it make sense overall? What’s too long or too convoluted? Where does the never-existent thread snap? What’s especially good and what’s rather rubbish. It gives tips I can incorporate but don’t have to. That can take a few iterations, but it definitely helps to order one’s thoughts in their own juices. And when I’m done with that, it goes to the AI copy editor. New typos are a side matter there, but it checks readability, correctness, style, tone, structure and so on. Tips here too, but more of a list for me to work through, because these are actual errors or stumbles. And then it goes to the AI closer. That one makes it Substack-ready. Mechanically. And reading this back now, I think: cool. That must run like clockwork and generate nothing but brilliant, perfect newsletters. It doesn’t. But whatever errors and confusions and false statements remain in the end result — that’s style.

Simplification — do it the same way wherever you can Standards help. Dumb it down. That transfers beautifully to the team, too. An error everyone makes is better, because it’s spotted more easily and solved more durably. Above all, though, the standard prevents errors caused by carelessness or inattention. Wherever possible, quality shouldn’t depend on the person or AI doing it. Use a schema, use a method. Define quality standards. In the output or in the prompt. My standard is a prompt pattern and a method for generating prompts.

Redundancy Do the same thing again, only differently and above all elsewhere. Three parallel chat windows are almost standard for me. In one I prepare and set things up. Get the data in shape, research and clarify and refine what I actually want. In the second I do the same again with a different AI, or take the results from chat 1 apart once more — on the premise that everything about it is wrong: devil’s advocate, red team, steelman. If necessary, because it’s too long or complex, I have the third window determine, analyse and assess the delta between the two chats.

Verification Ideally you only hand the AI tasks you can also verify easily. So testing and checking should be cheaper than doing it yourself from the start. So recompute results. Spot-check the source references. Does the program start without errors. Does the core function work.

Competence Anyone paying attention will have noticed that my redundancy and verification examples are the boring and supposedly worse version of our 10-agent one-click magic from earlier. The difference — I remain in between as the competence. I can control and optimise every step. I can also automate case by case, as much or as little as I like. And I can use different data or different LLMs. Do the checking holistically or by sampling. The decisive point is that I take responsibility for all the correct and good outputs of the AI, but also for all its errors. Which is why I steer how and where and how much. And that keeps me competent along the way, and equipped for the next task I want to optimise with AI.

Transferring poka-yoke, damage limitation and error culture is less direct and less immediate, even though they represent a big lever. Maybe you’ll find a clever poka-yoke trick, or manage to establish an AI error culture. I’d be very interested in your ideas.

I do have a few concrete, AI-specific error preventers left, though. Knowing the AI weak points, that is:

• A chat can be burned. The AI hallucinates, goes round in circles, contradicts itself, even though it was perfectly sensible at the start. Clear case: context distraction … don’t carry on hoping it’ll get better or that you can still talk the AI round. Salvage what can be salvaged — have it give you a summary of the chat and start fresh with that in an unused chat window. And the best part: the whole thing also works proactively. Past a certain duration or length of chat, just have it summarised and carry on in the next one. Saves frustration.
• Stick to one topic. I like to digress, and one thought sometimes brings up another in an entirely different context. So why not just keep chatting. The AI answers obligingly and plays along, after all. That’s a deceptive trap. So: new thought or new topic means new window.
• A bit of radio discipline. On the one hand I’m a great advocate of not adapting your own language and way of thinking to the AI, and of giving more information than you yourself think necessary. And of trusting the AI with more from the outset than you’d expect of a number-cruncher. But you also shouldn’t drift into waffling.

The goal is not the error-free machine. The goal is a system that makes the right errors, learns from them, and stops the wrong ones from propagating and becoming systematic errors.

A note on my own behalf: I write my posts myself, with the words out of my head. AI helps me with research, as an adviser and with technical correction. But the post is out of me, by me as a human. The images are all AI-generated, and I create and tend and maintain my website with Claude — largely in terms of content, too. In my newsletter, as part of the transparency campaign, I’ll clearly mark the few passages of text (mostly lists or tirades, for the sake of the joke) as AI-generated sections — retroactively as well. Meaning: whatever isn’t marked is by me. That’s just how I am. Swear.

Completely error-free isn’t possible. Not even with large, error-minimising systems. This isn’t meant to sound sarcastic, but if such a thing existed we’d have deployed it long ago — against plane crashes, swabs forgotten in abdominal surgery, the nuclear meltdown. And no, that isn’t settled by invoking human factors, and it won’t fundamentally disappear with AI factors either. Errors are part of it. Rarer, less drastic in their effects, controllable when they do happen — those are the real goals. And they’re achievable.

You can aim for zero errors, sure. It also sells better than “we’re aiming for only one catastrophic plane crash per year”. Everyone wants zero, forever. But wanting isn’t enough. We’ve already improved immensely: from up to 100 crashes a year down to a handful, and that with massively increased traffic. Maybe we’ll manage one crash every two years. Every ten. It isn’t about the numbers — and yet it is. Because past a certain point the next improvement is no longer sensibly — in the economic sense — achievable.

And now the uncomfortable part. I don’t want to rant against airlines. But all those safety precautions don’t exist because the board consists solely of morally balanced philanthropists. One main motivation: no decision-maker wants to see a prime-time news item showing their own company logo on the tail of a burning wreck. That has extremely direct effects on the annual accounts and the next bonus. Sure, protecting lives is good and sells splendidly. Win-win. But aviation safety is neither an accident nor a virtue — it’s also a property of the media system. A crash kills many people at one stroke, in one place, with a logo on the tail. That’s a week of genuine anti-advertising for the company.

The cleanest proof of this, incidentally, flies right alongside the airlines: general aviation. Same aerodynamics and physics, same atmosphere, same country, same regulator, in part the same airfields. But per flight hour, private flying is many times more dangerous than scheduled flight — risk-wise closer to a motorbike than to an A320. And that has improved only marginally over 40 years, at least compared with commercial aviation, which kept getting steadily and rapidly safer. Why? Not for lack of knowledge. The methods — checklist discipline, mandatory training, systematic error analysis — are known and would be transferable. They’re simply not applied. Because a crashed Cessna with two dead is a regional news item, not a special report. No logo on the tail, no corporation trembling, no pressure. Everything the same — except visibility. And the outcome differs by two orders of magnitude.

Or take medicine. Avoidable treatment errors in German hospitals cost — depending on the estimate, and the figures are contested — a multiple of all road deaths every year. For every plane crash there’s a mandatory, independent investigation. For the forgotten swab there is: nothing comparable. Same human society, same value of a human life, an entirely different visibility regime. Guess where more people die. And with a probability bordering on aviation safety, life-saving improvements could be achieved here.

Errors are part of it. Avoiding them is important and right. Wanting to eradicate them, or categorically ruling them out like a mantra, is utopian — and whoever promises you zero is selling you marketing, not a system. That’s the scientific truth. But behind errors there’s also a human or inhuman economy. And in case of doubt, that one doesn’t even stop at life.

During my research I fell in love with the principle of punishing the covering-up of errors, rather than the making of them. It sounds so good that it might just work. At the very least it seems to be the right mindset, and it’s actually lived at the airlines.