When I compare language-model training methods, I want to know what changes in the model, what data the method needs, and whether the result will help with the task I am building. Those questions are easy to lose in a list of benchmark scores.
This guide covers the mechanics and the findings I use when making those decisions. The papers are linked beside the claims they support.
Tokens and attention
An autoregressive language model estimates the probability of the next token given the preceding tokens. A tokenizer maps text to token IDs; an embedding table maps those IDs to vectors. Tokens can represent parts of words, punctuation or other text fragments. Their boundaries depend on the tokenizer.
In a decoder-only Transformer, causal self-attention lets a position combine information from preceding positions and itself. Queries and keys determine attention weights, which weight the value vectors. Multiple attention heads provide separate learned projections. Feed-forward layers, residual connections and normalisation transform the representations between attention operations.
The original Transformer paper introduced an encoder-decoder architecture for translation. Many generative language models use its decoder-style components. Attention weights describe a computation; a head’s function or the reason for a generated answer requires further analysis.
During generation, the model produces token scores and a decoding procedure selects the next token. Temperature and sampling settings affect that selection. The resulting token becomes part of the input for the next step. A convincing sentence can therefore be produced without checking whether it is true.
What pretraining and fine-tuning change
Pretraining learns from a large corpus, commonly by minimising next-token prediction loss. Continued pretraining applies a similar objective to an existing model, often on a new domain or data mixture. Supervised fine-tuning, or SFT, trains on demonstrations such as instructions paired with desired responses.
SFT can change behaviour and teach information. How well it does either depends on the starting model, examples and training setup. It can also reduce performance on tasks outside the fine-tuning distribution, so those tasks belong in the evaluation.
LIMA fine-tuned a 65B LLaMA model on 1,000 curated examples. Its results support the authors’ hypothesis that substantial assistant behaviour can be elicited from a capable pretrained model with a small, carefully selected dataset. That experiment does not establish a universal data requirement or show that all fine-tuning only changes presentation.
A controlled study of learning new facts through SFT found that new knowledge was learned more slowly than information consistent with the model’s existing knowledge. As new examples were learned, hallucinations increased in the study’s closed-book question-answering setup. The distinction matters: acquiring knowledge during fine-tuning is possible, but reliable factual updating is a separate problem.
For changing information such as an organisation’s current policies, retrieval gives an application a source it can refresh and cite. Fine-tuning can still help the application follow its response format or use retrieved evidence. These techniques can be combined.
LoRA and QLoRA
LoRA freezes the original weights and trains low-rank updates to selected weight matrices. For a matrix with dimensions d × k, a rank-r update uses r(d + k) parameters instead of dk. The chosen rank and target layers determine the size of the adapter.
This reduces trainable parameters and optimiser-state memory. An adapter can be merged into compatible base weights for inference. The paper found competitive results against full fine-tuning in its experiments; the appropriate rank and placement still need testing for a new task.
A small adapter is not evidence that fine-tuning cannot store new information. Rank constrains each matrix update, not the total information capacity of the network.
QLoRA combines a frozen, four-bit base model with trainable LoRA adapters. It introduced NF4 quantisation, double quantisation of quantisation constants, and paged optimisers. The authors demonstrated fine-tuning a 65B model on a single 48 GB GPU in their setup.
Memory estimates must include more than the stored weights. During training, activations, gradients and optimiser state also matter. During generation, the KV cache grows with sequence length and concurrency. A model that loads successfully can still run out of memory on a longer request. Measure peak memory with the intended context length, batch size and software stack.
Preference training
Preference datasets contain a prompt and responses ranked by a person or another evaluator. They can teach a model to prefer helpful, accurate or appropriately formatted responses, depending on what the labels reward.
Direct Preference Optimization, or DPO, derives a preference-training objective from a KL-regularised reward-maximisation formulation. Standard DPO trains on preferred and rejected response pairs relative to a reference policy, without separately training an explicit reward model or sampling responses in an online RL loop. The derivation does not mean every practical DPO run reaches the same result as PPO-based RLHF.
ORPO combines a supervised learning objective with an odds-ratio preference term and dispenses with a separate reference model. This offers a different training arrangement. Comparisons should control for data, base model and compute before attributing a gain to the objective alone.
Preference training can affect correctness as well as style. Calling DPO “style training” hides what is actually in the preference labels.
Reinforcement learning for reasoning
DeepSeekMath introduced Group Relative Policy Optimization, or GRPO. It estimates a baseline from rewards for a group of sampled responses, avoiding PPO’s separate critic model. The paper explores both outcome and process supervision.
Its analysis found a larger improvement in majority-vote accuracy than in pass-at-K on the evaluated tasks, suggesting that RL made successful outputs more likely in that setting. A finite sample of answers cannot establish every problem a model can solve, so this observation is not a general limit on what RL can learn.
A verifier also has limits. Passing a unit test establishes behaviour covered by that test. Training against incomplete tests can reward an implementation that fails elsewhere. Keep evaluation cases separate and inspect failures as well as aggregate reward.
The Qwen3 report describes reasoning cold-start training, reasoning RL, thinking-mode fusion and general RL for its larger models. It also describes distillation for smaller models. This is a documented training recipe, rather than a fixed sequence that every model needs.
Distillation transfers behaviour from a teacher into a student using targets such as generated responses. Teacher errors can carry over. There is no general rule that a student must score below its teacher on every downstream task: results depend on the task, training data and evaluation. Any claim of surpassing a teacher needs a direct comparison under the same conditions.
Model size and data
Parameter count affects memory, compute and capacity. Training data and architecture affect how that capacity is used.
Phi-1 demonstrated strong Python coding results from a 1.3B model trained on selected and synthetic educational data. That supports careful data selection for the measured coding tasks. It does not make parameter count irrelevant to other workloads.
TinyStories studied very small models trained on a restricted distribution of short stories. It showed that coherent text generation can emerge at small scale when the language and subject matter are constrained. A model suited to that distribution is not automatically suited to open-ended assistance.
For an application, compare candidates on its actual inputs. Include the awkward cases: long records, missing fields, ambiguous requests and examples unlike the training set. There is no universal parameter threshold at which a particular training method becomes useful.
Reading an evaluation result
Before using a score to choose a model, check the evaluation setup:
- Which model version, prompt and decoding settings were used?
- Was the test data kept separate from training and model selection?
- Is success checked by execution, reference answers, human judgement or a model judge?
- How many examples were evaluated, and how variable are the results?
- What latency, memory and cost does the result require?
The MT-Bench and Chatbot Arena study examines model judges and documents issues including position and verbosity bias. Model grading can be useful, but an evaluator needs calibration against the criteria people actually care about. Swapping answer order and manually reviewing a sample can expose problems in the grading process.
I would start a fine-tuning experiment with a held-out evaluation set and an untuned baseline. Then change one part of the setup, record the settings, and inspect where the outputs improved or regressed. A structured-output task should measure schema validity and field accuracy separately. A coding task should run tests. A factual task should check supporting evidence.
The training method is a choice made after defining the failure you want to fix.