Most explanations of AI answer a question you did not ask. You wanted to know why it keeps getting the same kind of thing wrong, and you got a history of neural networks.
So this one skips the taxonomy. There is one idea here, and almost everything odd you have noticed falls out of it.
First, a quick disambiguation
"AI" has been stretched to cover several unrelated technologies, which is why conversations about it go sideways. The recommendation engine on a shopping site, the model that reads a medical scan, and the chatbot you type into are not variations on one thing. They were built differently and they fail differently.
Only the last one is what this site is about, and it has a more useful name: a language model. Everything below is about that.
What a model is, in one sentence
Here is the whole thing:
A language model is an enormous set of learned patterns about how text tends to continue, plus a layer of training on what people rated as a good answer.
The first part is the bulk of it. It was shown a staggering amount of writing and repeatedly asked to predict what came next. Over time it got extremely good at that, and getting good at it required absorbing a great deal about grammar, style, reasoning, and how the world is usually described.
The second part is thinner but it is what you are actually talking to. People rated its answers, and it was tuned towards the ones they rated well. That is why it answers your question instead of carrying on writing more questions after it, and it is why two tools built on the same idea can have noticeably different manners.
But notice what is not in there. There is no database. No facts stored as facts, filed away and retrieved on request. When it tells you the boiling point of water, it is not looking that up. It is producing the text that reliably follows that question. The answer is right because that text is overwhelmingly what follows, everywhere, in everything it read.
The mental shift
This is the part that changes how you use it:
It is not answering your question. It is producing what an answer to your question looks like.
Almost always, those are the same thing. What an answer to "how do I reverse a list in Python" looks like is the correct answer, because correct answers to that question are what the internet is full of.
The gap between those two things is where every surprise lives. Once you can see the gap, the behavior stops being mysterious.
What the shift explains
Confident wrong answers. If the job is producing what an answer looks like, a wrong answer that looks right is a complete success by that measure. There is no separate step where it checks. This is worth its own post, because it is the failure that costs people most.
Brilliant at essays, shaky at arithmetic. An essay has a shape, and the
shape is most of the work. 4,817 × 293 has no shape. It has an answer, which
you get by calculating, not by pattern-matching. Modern tools paper over this
by quietly running a calculator: ChatGPT writes and runs a line of Python and
reports back what it got. That is a fix worth having, and also a tell. The
underlying thing could not do it.
Better at common things than rare ones. More examples of something means a better-learned pattern. This is why it is superb on well-trodden ground and progressively less reliable the further into a niche you go, and why the drop is gradual rather than an error message.
Your tone changes the answer. Ask casually, get a casual answer. Ask like an expert, get an answer pitched at experts. Your question sets the pattern the reply continues, which is why How to Write Better Prompts opens with saying who the answer is for.
It agrees with you too readily. Agreement is what the raters rewarded, so agreement is what you tend to get when you push back. It has not reconsidered. How badly this bites varies between tools.
It has no memory of its own. Each time you send a message, the whole conversation is re-read from the top, and anything your tool remembers across conversations it remembers by pasting it back in. Context Is Everything is about what that means for how you work.
A better metaphor than "intelligence"
"Intelligence" imports the wrong expectations: that it knows what it knows, that it will say when it is unsure, that it can check.
That framing predicts behavior well. You would give a brilliant improviser the actual document rather than trusting their recollection. You would check any specific figure they produced. You would not ask them to do your accounts. You would ask them to explain a hard idea five different ways, or to argue against your position, or to tidy your draft, and they would be excellent at all three.
So what is it actually good at?
Not a list of tasks, but a shape. It is strong wherever the shape of a good answer carries most of the value:
- Explaining, rephrasing, translating between registers
- Structuring something you already know into something readable
- Generating options when you are stuck for a starting point
- Critique: finding the weak paragraph, the missing case, the unclear sentence
- First drafts of anything formulaic
And it is structurally weak wherever correctness cannot be inferred from shape: specific facts, current events, exact arithmetic, anything niche, and anything where being confidently wrong is expensive.
Where to go next
That is the shape of the thing, and it is enough to work from. Three practical follow-ons:
- How to Write Better Prompts covers the shape of the ask
- Context Is Everything covers what to put in front of it
- Why AI Makes Things Up covers catching the failure that matters
You do not need to understand transformers, and knowing what a parameter is will not make you better at this. Knowing that it produces what an answer looks like will.