LLM

09/15/2026

I just read a very convincing paper that explains to a physical scientist how LLMs actually work.

Implicit in the work was how startlingly limited the approach is.

Of course the paper then went on to define a bunch of clever ways to get around these limitations.

Their central position is that LLMs are useful research tools but unreliable epistemic agents.

The paper identifies several characteristic LLM failure modes: hallucination, context rot, semantic drift, over-confidence, hyperfixation and sycophancy.

Yet these have obvious human equivalents: we bullshit, lose track of information, shift definitions, express unjustified confidence, become fixated on particular explanations, rely heavily on heuristics, accommodate other people’s beliefs and resist changing our own.

Humans developed experimentation, falsification, replication, measurement and independent verification precisely because unaided human thinking is so unreliable.

The interesting question is therefore not whether LLMs suffer from serious cognitive limitations, but whether systematic external procedures can reliably compensate for them, just as the scientific method compensates for similar limitations in human cognition.

The article’s practical advice is largely a set of procedures designed to compensate for those LLM defects: decompose the task, demand explicit assumptions, require sources, test claims independently, use the model iteratively rather than accepting its first answer and deploy agents to automate these improvements.

And then I guess we can’t judge LLMs on their general limitations because these can be so easily fixed.

So our place at the top of the cognitive apex is at risk – AI will take over! Most humans are using heuristics (lol) to judge this possibility.

If the question is sufficiently important, our own history implies that we should stop relying on intuition and develop systematic procedures for assessing it. Probably using LLMs.