Confabulation as a statistical inevitability
- Any claim about long-tail or one-off facts seen rarely in training
- Open-ended generation with no retrieval grounding
- Setups that reward a confident answer over an honest "I don't know"
Four distinct dimensions of failure — positional, qualitative, structural, behavioral — distribute unevenly across the LLM pipeline. More pathologies surface at the attention layer than at any other stage; behavioral failures emerge only downstream. Understanding which dimension produced a given failure tells you which layer of the stack can fix it — and which cannot be patched by prompting at all.
The atlas has five complementary maps. Map A places all 17 pathologies on a 2D grid where rows are dimensions (color-coded) and columns are pipeline stages where the failure surfaces. Empty cells are informative — they show that, e.g., behavioral pathologies never originate at the input stage. Map B isolates one typical failure cascade — a "death spiral" that turns one upstream failure into a session-wide breakdown. Map C is the defensive architecture: a two-run verification pattern that addresses the two critical-severity pathologies (P07, P08) plus the two that propagate across runs (P15, P17) at once. Map D is the master matrix for filtered lookup. Map E is a qualitative simulator — pull a parameter slider to see which pathologies respond.
Click any node in Map A or Map B to open the detail panel (Layer 2: mechanism, when it strikes, mitigation effectiveness with concrete percentages). Inside that panel, reveal the prompt or architecture pattern as Layer 3 when you want concrete code. The single insight to walk away with: the attention layer alone hosts 6 of the 17 pathologies — it is the single most fragile component in a long-context system, and no single fix tier covers it.
Three caveats on Class 1. First, the 4×5 matrix and every cell assignment are this atlas's own construct — a way to order the phenomena, not a measured taxonomy; the per-pathology Source tag marks which entries are empirically replicated, which come from a practitioner blog taxonomy, and which are framing. Second, several of these effects are model-generation-dependent: newer frontier models measurably dampen some of them, so treat severities as relative, not fixed. Third, the dimensions are not perfectly orthogonal — Context Rot, Attention Dilution and Context Distraction overlap heavily, and Context Rot reads more as an umbrella for gradual long-context decay than as a peer of the other two.
The seventeen above are only the first class — failures that live around the context window. Two further classes sit deeper and behave differently: one in the model core, one only in the integrated application. The axis that separates them is where the failure is born and how far it can be cured.
CONFIRMED or NOT FOUND| Pathology | Dimension | Stage | Severity | Frequency | Prompt-fixable |
|---|
Every proposed defence reduces to one wish: keep instructions and data apart. Architecturally, that wish cannot be granted. The model is handed a single, undifferentiated token stream; there is no field, tag or type that marks "this span is a command" versus "this span is data to be processed." Syntax and semantics share one channel.
Zverev et al.9 make this measurable: they define a formal separation score and, on their SEP dataset, show that no model achieves high separation — and that the usual mitigations (prompt engineering, fine-tuning) barely move the score or buy separation only by sacrificing usefulness.
This is the same construction flaw as classic SQL injection: control instructions and user data travelling over one channel, so data crafted to look like a command gets executed as one. Computer security solved its analogues by building a hard boundary the LLM stack lacks — memory is writable exclusive-or executable (W^X), and data is, by rule, never executed. An LLM agent has no such guarantee: a sentence retrieved from a web page can act as an instruction the moment it enters the context.
The honest move the field made was to drop the model-level question. You do not make the model injection-proof; you build the system so a fooled model can do no harm. The trust boundary moves out of the model and into the architecture — classic security discipline: least privilege, taint-tracking, control/data-flow separation.
CaMeL17 is the worked example: a privileged planner is constructed only from trusted user input, while everything that touches untrusted data is treated as tainted and cannot trigger a privileged action. The model can still be deceived — it just cannot reach anything that matters.
The practical lever is Willison's "lethal trifecta"18: catastrophe needs three ingredients at once — access to private data, exposure to untrusted content, and a way to exfiltrate. Break any one leg and a successful injection drops from disaster to nuisance.
And to be precise about "unsolvable": there is no impossibility proof here, unlike the halting problem. The right analogy is adversarial robustness and human social engineering — an open input space against an adversary, reducible to acceptable residual risk, not eliminable. "Latently unsolvable" means structurally resistant — shown empirically, not proven impossible. What wins is boring engineering discipline, not a clever prompt patch.