“All models are wrong, but some are useful.”
George Box
This practical primer is part of the series on moving from data engineering into AI systems. It focuses on LLM concepts relevant to interviews and implementation; the original relative newsletter schedule is omitted.
Read the related career and job-posting articles for the broader series. Publication timing does not affect the technical guidance below.
If you have applied to AI engineer roles in the last few weeks, you have probably noticed that the technical conversation now turns, at some point, to a question that sounds harmless but is doing real work. How do you think about model A? Or, why might this model give a great answer to one question and a poor one to a similar question? Or, walk me through what happens when a user sends a prompt.
These questions are not testing whether you can implement attention from scratch. They are testing whether you have a useful mental model of what the system is doing. A senior candidate who has spent twenty years building data systems can reach that bar in an afternoon of careful study. You do not need a PhD. You need the right small set of ideas, in the right shape.
Here is the shape.
Many generative LLMs predict the next token.
That is the entire system, at its core. You give the model a sequence of text. It produces a token (a word or part of a word). Then it appends that token to the sequence and produces the next one. Then the next. It keeps going until it generates a stop signal or hits a length limit.
There is no agent inside the model that decides.
If you internalize nothing else from this piece, internalize this: predict the next token, given the context, based on what was learned during training.
Three foundations follow.
Autoregressive output generation contributes latency token by token. Input length also affects prefill latency and time to first token; caching, model architecture and serving configuration matter. Step-by-step prompting is not a reliable improvement for every model or task—evaluate the model’s recommended prompting approach and actual results.
Inference weights generally remain fixed during a request, but an application can maintain memory, retrieve documents and include prior conversation in later prompts. Later training is separate. Truncation can remove earlier information, yet a model can also fail to use information still in context; apparent forgetting has more than one cause.
This is the foundation of every retrieval-augmented generation system, every agentic workflow, every long-context application. The entire discipline of AI engineering is, in one sense, the work of deciding what goes into the context window and what does not.
Sampling can produce different outputs from identical prompts; serving kernels and model updates can introduce additional variation. Temperature, where exposed, changes sampling behavior but does not guarantee correctness or complete determinism.
Differences in context, retrieval, reasoning difficulty, model capability and sampling can all affect correctness. Temperature alone is not an explanation for why one answer was accurate and another was not.
What this gives you in the interview.
These three foundations let you answer the how do you think about model behaviour family of questions like a practitioner.
A model that produces inconsistent outputs is not broken. It is sampling from a probability distribution.
A model that hallucinates is not lying. It is producing the most probable continuation given its training data, even when no good continuation exists.
Losing an earlier instruction can reflect context truncation or a failure to use information still present. Inspect both application context assembly and model behavior before diagnosing the cause.
Latency depends on input prefill and output generation, as well as caching, architecture and serving configuration.
You will sound like someone who has thought about this for longer than ten minutes. Because you have. That is the actual bar in the interview.
One layer this primer does not cover.
There is a deeper layer of understanding: how the model actually computes the next token from the input. That involves attention, embeddings, the network’s layers, and the mathematics of the transformation between them. Most AI engineering interviews do not require this depth in the first conversation. They require the mental model above. The deeper layer is worth learning, and we will reach it in a future Tuesday piece.
For now, the next-token-predictor framing, the three foundations, and the implications above are enough to make you a credible technical conversation partner. Two hours with these ideas, tested against your own experience using ChatGPT or Claude in the last month, and you have made yourself meaningfully more interview-ready by the end of the day.
Understand the system, not just the symptoms.
Written by Roland Wenzlofsky, founder of DWHPro and author of Teradata Query Performance Tuning. DWHPro has helped data warehouse practitioners for 15+ years.
Deciding what to change in your data platform?
Compare optimisation, modernisation and migration against your workload and business requirements.
