How Large Language Models Work: Tokens to Answers

What is a large language model?

A large language model (LLM) is a type of neural network trained to predict the next piece of text. That sounds simple, but when the model is large and the training data is vast, predicting the next token well requires picking up grammar, facts, writing styles, reasoning patterns and even some coding skill.

Step 1: How does an LLM read text? Tokens

Computers work with numbers, so text is first split into tokens. A token might be a whole word, a part of a word, or a punctuation mark. A tokenizer converts your sentence into a list of token IDs, and the model reads and writes text as sequences of these IDs.

This is why model limits and prices are usually quoted in tokens rather than words, and why unusual words or other languages can use more tokens than common English words.

Step 2: How does the model represent meaning? Embeddings

Each token ID is turned into a list of numbers called an embedding. During training, tokens used in similar ways end up with similar embeddings, so the model can represent relationships between words and ideas as positions in a high-dimensional space.

Step 3: How does it understand context? The transformer

Most modern LLMs use the transformer architecture, introduced in the 2017 research paper Attention Is All You Need. Its key mechanism is attention, which lets the model weigh how relevant every other token in the input is when it processes a given token.

Attention is one of the mechanisms that lets the model use the surrounding words, rather than a fixed rule, to work out what a word such as “it” refers to.

Step 4: How is the model trained?

Training usually happens in stages:

1. Pre-training. The model reads a very large collection of text and is repeatedly asked to predict the next token. Its billions of internal parameters are adjusted to reduce its mistakes.
2. Fine-tuning. The model is trained further on curated examples, such as question-and-answer pairs, so it follows instructions instead of just continuing text.
3. Preference tuning. Human feedback, or feedback from other models, is used to nudge the model toward answers people rate as more helpful and safe. Reinforcement learning from human feedback (RLHF) is one well-known method.

Step 5: How does it generate an answer? Inference

When you send a prompt, the model produces a probability for every possible next token, picks one, adds it to the text, and repeats. Settings such as temperature control how adventurous the choice is: low values favor the most likely token, higher values add variety.

The model can only look at a limited amount of text at once, called the context window. Your prompt and the model’s reply must fit inside it.

Why do LLMs make things up?

LLMs generate plausible text, not verified text. Unless they are connected to a search tool or a database, they do not look facts up; they rely on patterns absorbed in training. When the pattern is weak or missing, the model may still produce fluent, confident, incorrect output. This is commonly called hallucination.

Ways to reduce it include:

• Giving the model reliable source material in the prompt.
• Using retrieval-augmented generation to fetch documents first.
• Asking for citations and checking them.
• Keeping a human in the loop for important decisions.

What can LLMs do well, and where do they struggle?

Usually strong	Often weak
Drafting, summarizing, rewriting	Guaranteed factual accuracy
Translation and explanation	Precise multi-step arithmetic without tools
Writing and explaining code	Knowing about events after their training data
Classifying and extracting information	Consistently following very long, complex rules