
Key Takeaways
- Training changes a model’s parameters using a measurable objective. Ordinary generation uses those parameters without updating them.
- Pre-training learns from text; post-training uses demonstrations and feedback to shape how the model responds.
- Reasoning can improve through practice with checked answers. The quality of the reward limits what that practice teaches.
- Judge a training run on independent evaluations of the target behavior, and check that other useful skills are preserved.
Training a large language model means changing the numbers inside a neural network so that it produces better outputs. What counts as “better” changes during training. Learning to predict text, follow a request, and solve a problem using feedback are related tasks, but they provide different signals to learn from.
The process usually begins with pre-training, which develops broad capabilities from a large collection of text. Post-training then uses more targeted examples and feedback to shape the model’s behavior. Together, these stages help explain how a text predictor becomes an assistant—and why a fluent assistant can still make mistakes.
This guide follows that process from the learning loop to reasoning and evaluation. It focuses on autoregressive language models: models that generate text one token, or piece of text, at a time, using the text available so far. The examples below describe public training methods; individual model recipes vary.
The learning loop
A neural network is a series of numerical operations, grouped into layers, with adjustable values called parameters. These include weights that determine how information is combined. Layers also use nonlinear operations—for example, replacing negative values with zero. These let the network represent more complex relationships than a single weighted sum.
Training from scratch starts with initialized parameters, many of them random. The training program takes a batch, a small collection of examples, and runs it through the network. This calculation is called the forward pass. For a language model, it produces scores for the tokens that could come next.
A loss function compares those predictions with the next tokens recorded in the examples, called targets. The training program then uses backpropagation to calculate a gradient for each parameter: how a small change to that parameter would affect the loss at its current value. Software such as PyTorch computes these derivatives automatically through the network’s operations.
An optimizer uses the gradients to update the parameters. In basic gradient descent, it moves each parameter in the direction that locally lowers the loss. The learning rate controls the size of the step. The program repeats the process with more batches, gradually changing the network’s behavior.
For example, if a weight is 1, its gradient is −2, and the learning rate is 0.1, basic gradient descent sets the new weight to 1 − 0.1 × (−2) = 1.2. The negative gradient tells the optimizer to increase this weight to reduce the loss locally.

Prediction, loss, backpropagation, and an optimizer update form one training step. The network and sliders are schematic.
These updates use the same parameters across many examples, so changing the model to improve one prediction can affect others. A lower loss on the training data does not guarantee better performance on new inputs. That is why evaluation must use examples kept out of the updates.
Using the trained model is a separate process called inference. During ordinary text generation, the model’s parameters stay fixed. A prompt, the input text supplied to the model, changes the information available for the next prediction; it does not, by itself, perform a training update.
Pre-training: learning from text
Pre-training applies this loop to a large text collection, or corpus. The model learns patterns that help predict what follows a piece of text, including patterns in grammar, factual statements, code, and explanations. It can also memorize particular passages. The objective rewards predicting the recorded text, so the choice of text matters throughout the process.
Turning text into a training signal
First, a tokenizer splits text into tokens and assigns each an ID from a vocabulary. A token can be a word, part of a word, punctuation, or another text fragment. These IDs are labels: a larger ID does not mean a more important word or a larger quantity.
The text itself supplies the targets. If four tokens are represented as A B C D, the model can learn to predict B after A, C after A B, and D after A B C. No person needs to label each next token separately. This is called self-supervised learning.

One sequence supplies targets at several positions. A, B, C, and D stand for tokens, not necessarily whole words.
For each prediction, the model assigns a probability to every token in its vocabulary. These probabilities add up to one. Cross-entropy loss penalizes the model when it assigns a low probability to the token that actually follows in the dataset. Giving that token an 80% probability produces a smaller loss than giving it a 10% probability.
A training sequence contains many such prediction positions, and a Transformer can process them in parallel. A causal mask prevents a position from reading future tokens, including its own next-token target. This preserves the task: predict the next token from the preceding context.
The recorded continuation is a training target, not the only sensible continuation and not necessarily a true statement. Predicting text accurately and establishing that an answer is correct remain different goals.
Using context with a Transformer
The token IDs first become embeddings, learned lists of numbers that the network can process. Positional information lets the model distinguish different orders of the same tokens. A Transformer is a neural network organized into repeated blocks that process these representations and combine information across the sequence.
Each block contains attention, which lets a position combine information from the context it is allowed to read. Learned comparisons determine how strongly other positions contribute. In a sentence about a person putting a book on a shelf, attention can help later words use information about the person, the book, or the shelf. Several attention heads perform different learned combinations in parallel.
A block also contains a feed-forward network, which transforms the representation at each position using learned weights and nonlinear operations. Repeating attention and these transformations lets later blocks work with representations built by earlier ones. At the output, the model converts numerical scores, called logits, into next-token probabilities.

The highlighted position combines information from earlier positions and itself. The drawing omits other attention paths and block components.
During training, the prefix comes from the recorded text. During generation, the model selects a token from its predicted distribution, adds it to the context, and predicts again. It can choose the highest-probability token or sample among possibilities. Different choices can produce different responses from the same trained model.
Choosing what the model learns from
Building the corpus involves extracting usable text, filtering unwanted material, removing duplicates, and choosing how much different sources contribute. FineWeb documents how these decisions affect web-scale training data. Repeated low-quality material can consume training effort, while evaluation examples accidentally included in training can make a model appear more capable than it is.
The amount of data also needs to fit the training budget and model size. The Chinchilla research showed why allocating compute to a larger model while giving it too few training tokens can be a poor tradeoff. Its broader lesson is to consider model capacity and training data together, rather than treat parameter count as a complete measure of capability.
The result is a base model. It may already answer questions or complete code when prompted appropriately. But its training objective has not directly specified the behavior expected from a helpful assistant. Post-training makes that behavior a more explicit target.
Post-training: shaping useful behavior
Post-training continues from a trained model instead of restarting with random parameters. It changes the examples, the objective, or both. A common approach begins with demonstrations of good responses, then uses feedback to improve the model’s choices. Some recipes repeat or combine these stages.

A public training pattern: broad text, targeted demonstrations, then feedback on attempts. This is one recipe; reasoning can also be learned without a reinforcement learning stage.
Learning from demonstrations
Supervised fine-tuning (SFT) trains on prompts paired with desired responses. A demonstration can show how to answer a question, explain an uncertainty, use a tool, or decline an unsafe request. The next-token objective remains, but the training examples now show the behavior the assistant should produce.
Conversations need a consistent representation. A chat template places the user’s message, the assistant’s reply, and any system instructions into the model’s expected sequence of tokens. Models can use different templates, so the formatting used at inference needs to match the model’s training.
Many SFT recipes calculate loss on the assistant’s response while excluding the user’s prompt from the loss. The prompt still provides context. For each response position, the model sees the preceding recorded tokens and learns to predict the next recorded token. It also learns the appropriate token for ending a turn; the generation program stops when it encounters a configured stopping condition or reaches its token limit.

One position in response-only SFT: P, Q, and the recorded reply token A provide context for predicting B. The demonstration supplies B as the target; C is not visible to that prediction. Probabilities are schematic.
Demonstrations can teach useful behavior, but they cannot cover every request the model will encounter. They also inherit mistakes and biases in the desired responses. Feedback provides another way to tell the model which responses should be favored.
Learning from preferences
Some qualities are easier to compare than to specify as a perfect answer. Reviewers might prefer one explanation because it is clearer, more complete, or more appropriately cautious. Training can use these comparisons between responses to the same prompt.
In a common reinforcement learning from human feedback (RLHF) pipeline, these comparisons train a separate reward model to predict which responses people prefer. The assistant then generates responses, the reward model scores them, and reinforcement learning updates the assistant to favor higher-scoring behavior. InstructGPT is a public example of this approach.
Direct Preference Optimization (DPO) takes another route. Its original formulation uses a dataset of preferred and rejected responses to update the assistant directly, without training a separate reward model or repeatedly generating responses inside the optimization loop. Both approaches use preferences, but they involve different training procedures.
Preference is an imperfect guide to correctness. A confident explanation can be appealing and wrong. Training therefore needs evaluations that check the intended behavior independently of whether a response receives a favorable rating.
Learning to reason with feedback
For some tasks, a training program can check an attempted answer more directly. A math verifier can compare a result with a known answer. A test runner can check whether generated code passes specified tests. These checks provide a reward for practice, while remaining limited by what they actually test.
In this setting, the model generates attempted solutions, receives scores, and undergoes updates that favor better attempts. The model is often called the policy: it determines the probabilities of the tokens it generates. Sampling fresh attempts from the current policy lets later rounds of training respond to its changing behavior.
Group Relative Policy Optimization (GRPO), introduced in DeepSeekMath, compares several attempts at the same prompt. Each attempt receives an advantage, a signal describing how its reward compares with the group. If two attempts pass and two fail, with rewards of 1 and 0, the average reward is 0.5. Passing attempts are above average; failing attempts are below it.

GRPO uses relative rewards to guide updates. The sliders represent incentives for response likelihood, not a dedicated parameter for each attempt.
The training objective uses these advantages with the generated tokens’ probabilities to encourage higher-reward attempts and discourage lower-reward ones. Backpropagation computes gradients for the shared parameters, and the optimizer applies the update.
The verifier only needs to supply a score. The training program calculates gradients using the model’s token probabilities, so the verifier does not need to expose its own gradients. This lets a calculator or test runner provide feedback without becoming part of the neural network.
This approach has real limits. If every attempt receives the same reward, there is no within-group signal about which was better. A score for a whole answer also does not establish that every intermediate step was sound. Weak tests can reward shortcuts, and improving the rewarded skill can come at the expense of other behavior.
Reasoning does not require every model to undergo RL. Prompting and demonstrations can also elicit or teach step-by-step problem solving. Distillation trains a student model on outputs from a more capable teacher; the DeepSeek-R1 work includes students trained this way through SFT. At inference, a model may also spend more computation generating intermediate work before answering. That additional computation uses its trained parameters without ordinarily changing them.
Making training practical
The learning loop is compact, but running it at scale requires enough memory and computation to process the model, its data, and its updates. For an application team, the first decision is often whether to train from scratch or adapt an existing model. Those choices have very different resource requirements.
Scaling the update across GPUs
A GPU accelerates the large numerical operations used by neural networks. But the model’s weights are only part of training memory. Eight billion weights stored at two bytes each take about 16 GB, using decimal units. Full training also needs space for gradients and intermediate results from the forward pass, called activations. Optimizers may also keep statistics between updates to guide later steps; these are called optimizer state.
Distributed training divides work across GPUs in several ways:
- Data parallelism runs model replicas on different batches, then synchronizes gradients.
- Tensor parallelism splits operations inside a layer across devices.
- Pipeline parallelism places groups of layers on different devices and passes intermediate results between them.

At left, replicas synchronize gradients. The top matrix in the middle is a reference view of one operation split over two GPU bases. The pipeline at right shows only forward activations; backward communication and state sharding are omitted.
These methods can be combined. Training systems can also shard parameters, gradients, or optimizer state, distributing them rather than keeping a full copy on every device. ZeRO describes this memory-saving approach. The engineering challenge is to keep the devices doing useful work while coordinating communication and preserving enough training state to recover from failures.
Adapting an existing model
Fine-tuning starts from a checkpoint, a saved set of trained parameters. For many applications, this provides a practical starting point: the model already has language capabilities, and training can focus on a specific behavior. Full fine-tuning updates all of the selected model’s parameters.
Low-Rank Adaptation (LoRA) reduces the number of parameters that must be trained. It freezes the original weights and adds a trainable adjustment to selected weight matrices. That adjustment is represented as the product of two smaller matrices—rectangular grids of numbers—rather than a full-size update. The model uses the original computation and the learned adjustment together. These added trainable weights are called adapters.
QLoRA reduces memory further by storing the frozen base weights in a four-bit representation while training LoRA adapters. This lower-precision storage is called quantization: fewer bits represent each weight, reducing memory use. Computation uses higher-precision representations where needed; “four-bit” does not describe every calculation or every item held in memory. Activations and other training state still need space.

Gold marks trained weights; gray marks fixed weights. Adapter sizes are schematic, and the four-bit badge refers to base-weight storage.
These methods change which parameters are updated and how they are stored. They do not decide what the model should learn. A LoRA run still needs suitable examples, an objective, and an independent evaluation, just as full fine-tuning does.
Evaluating what changed
Evaluation should begin before the training run. Establish how the starting model performs on the behavior you want to improve, then keep a separate set of examples out of training. Use validation data to choose training settings and reserve a test set for the final comparison. Repeatedly selecting settings against the test set weakens its independence.
Track training loss or reward alongside performance on these held-out tasks. Falling training loss with worsening validation performance is a warning sign of overfitting: learning the training examples in a way that fails to carry over. For an assistant, also check instruction following, factual errors, and useful skills outside the task being optimized. Keep evaluation prompts, tool access, and generation budgets consistent when comparing models.
Some outputs require more than a benchmark score. A proposed discovery raises two separate questions: whether the result holds up and whether it was already known. Neither fluent writing nor a high training reward answers both.

Correctness and novelty require different evidence. The appropriate check depends on the claim; the diagram does not imply automatic validation.
Use checks suited to the claim: tests for specified software behavior, a proof checker for a formal derivation, or an experiment for an empirical prediction. A claim of novelty additionally requires examining prior work. Each check establishes a particular kind of evidence, with its own limits.
For a first training project, start with an existing checkpoint and one behavior you can measure. Build the evaluation before choosing the training method, run a small adaptation, and inspect both improvements and failures against the starting model. That makes the next decision concrete: improve the data, change the objective, or keep the original model.
References
Background and editorial reference
Matthew Berman, interviewing Jeremy Berman: Anthropic Researcher explains how to train LLMs. Background interview on the training process.
William Hu, Anthropic: Building always-on agents. Editorial reference for the article’s explanatory structure and tone.
Neural networks and the training loop
- PyTorch: Build the Neural Network.
- PyTorch: Automatic Differentiation with torch.autograd.
- PyTorch: Optimizing Model Parameters.
- PyTorch: CrossEntropyLoss.
Language models and pre-training
- Hugging Face LLM Course: How do Transformers work?.
- Attention Is All You Need.
- Hugging Face: Causal language modeling.
- Language Models are Few-Shot Learners.
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale.
Scaling training
- Training Compute-Optimal Large Language Models.
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models.
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.
- The Llama 3 Herd of Models.
Post-training and reasoning
- Hugging Face: Chat templates.
- Training language models to follow instructions with human feedback.
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model.
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.
- Hugging Face TRL: GRPO Trainer.
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.