Find a concept

Search Notes and Discovery

Enter a term to search published Notes and Discovery.

    In this topic

    Structured note

    Fine-Tuning Transformers: From First Principles to Reliable Evaluation

    Understand full fine-tuning, trace classification loss and metrics, then build DistilBERT sentiment and DistilGPT2 news classifiers with honest error analysis.

    Fine TuningFoundation
    Published
    Reviewed
    Reading time
    42 min

    Prerequisites

    No prerequisite note—start here.

    Handwritten layered paper illustration showing a pretrained Transformer adapted with labeled sentiment examples into a fine-tuned model with a classification head.
    On this page

    Fine-tuning means continuing training from a pretrained model so its parameters become more useful for a particular task or data distribution. For text classification, that usually means attaching a small prediction layer, showing the model labeled examples, and updating its weights using the errors it makes.

    Text classification assigns an input to one of a predefined set of categories. A data distribution describes the patterns and frequencies of examples a model encounters. The opening illustration uses a few labeled examples as a visual shorthand for task adaptation; a useful model requires a sufficiently large, representative training and evaluation collection.

    The difficult part is deciding what a successful update means. A model can improve its average score while missing most neutral tweets. It can score well on held-out news articles and misclassify a clearly written sports headline. Understanding those failures requires connecting the task, architecture, input representation, loss, and evaluation.

    This article builds that connection in three stages: a numerical explanation of learning and metrics, a complete sentiment-classification workflow with DistilBERT, and a separate news-classification workflow with DistilGPT2. It then explains what changes when the output is a generated summary instead of a class label.

    About the results: the recorded experiment tables describe earlier runs. The expanded teaching code below has not been run here to reproduce those experiments. Arithmetic examples are explicitly illustrative. Training time, learned probabilities, exact tokenized samples, and final scores depend on the environment, data revision, seed, and preprocessing; the recorded values are observations, not promised outputs.

    What changes when we fine-tune a model?

    Pretraining is the initial learning process that gives a model broadly useful representations from a large collection of examples. A representation is a vector of numbers that encodes features useful to the model. A parameter, also called a weight, is a learned numerical value inside the model.

    An encoder such as BERT learns partly by recovering hidden tokens from their context. A causal language model such as GPT learns to predict the next token from the preceding tokens. Neither objective directly says, “Classify this tweet as negative, neutral, or positive.” Fine-tuning supplies that task-specific objective.

    Large text collection → pretraining → pretrained weights
                                                ↓
    Labeled task examples → task loss → updated task-specific weights

    A label is the desired answer supplied with a training example. For a sentiment dataset, "The update fixed my problem" might have the label positive. The model predicts a label, compares its prediction with the supplied answer, and adjusts its parameters.

    A classification head is the final trainable layer or small network that turns the model’s representation into one score per class. The backbone is the pretrained network beneath that head. The head usually starts with new random parameters when the checkpoint was pretrained for a different objective.

    Full fine-tuning, frozen features, and adapters

    Full fine-tuning allows all model parameters to change. Freezing a parameter means excluding it from gradient updates. An adapter is a small trainable addition to a larger model. Parameter-efficient fine-tuning, or PEFT, changes a relatively small set of parameters instead of updating the entire backbone.

    LoRA, short for low-rank adaptation, learns a weight update using two smaller matrices. In the diagram, WW is a frozen weight matrix, AA and BB are trainable matrices, and BABA is their matrix product.

    For WW with shape dout×dind_{out}\times d_{in}, choose AA with shape r×dinr\times d_{in} and BB with shape dout×rd_{out}\times r. The rank setting rr limits how many independent directions the update can express and is chosen much smaller than either dimension. This reduces the trainable parameter count from doutdind_{out}d_{in} to r(din+dout)r(d_{in}+d_{out}) for that weight update. The diagram omits the configurable scale applied to BABA. See the LoRA paper for the full method.

    Three adaptation paths: full fine-tuning updates the backbone and classification head; a frozen backbone updates only the head; LoRA keeps base weights frozen and trains small added matrices.

    Choose which parameters can learn before comparing the cost or quality of two training methods.

    MethodWhat learns?Practical consequence
    Frozen backbone with a new headThe prediction headA cheap baseline that tests the usefulness of existing features
    Full fine-tuningBackbone and head

    Greater freedom to adapt, with more training memory and a risk of damaging useful pretrained behavior

    LoRA

    Small added low-rank matrices, plus any explicitly trainable task layers

    Fewer trainable weights and optimizer states; the base model still has to be loaded and used

    Quantization stores numbers with fewer bits. QLoRA trains low-rank adapters through a frozen backbone stored in a 4-bit format; the QLoRA paper describes the additional memory-saving techniques. These methods are useful later, but they do not eliminate the need for clean data, a suitable objective, or an untouched test set.

    The examples here use full fine-tuning. In attention layers, this includes the query, key, value, and output projection weights; it also includes feed-forward layers, normalization parameters, embeddings, and the new task head. Equal total and trainable parameter counts establish that weights are eligible for learning. They do not establish that every scalar changes on every update.

    One parameter update, without hiding the symbols

    A loss is a numerical penalty for a prediction. A gradient tells us how the loss changes when a trainable parameter changes. The learning rate sets the size of an update. An optimizer applies an update rule using those gradients.

    Let θt\theta_t denote a parameter value before update tt, LL the loss, and η\eta the learning rate. The simplest gradient-descent rule is:

    θt+1=θt−η∇θL.\theta_{t+1}=\theta_t-\eta\nabla_{\theta}L.

    For an illustrative single parameter, suppose θt=0.50\theta_t=0.50, its gradient is 0.200.20, and η=0.01\eta=0.01. Then:

    θt+1=0.50−(0.01)(0.20)=0.498.\theta_{t+1}=0.50-(0.01)(0.20)=0.498.

    The positive gradient means increasing this parameter locally increases the loss; subtracting a small positive amount moves in the opposite direction. Real Transformer training uses millions of coupled parameters and an optimizer such as AdamW, which also tracks information from past gradients and applies weight decay. The simple equation explains the direction of learning, rather than every detail of AdamW.

    Backpropagation computes gradients by applying the chain rule backward through the computation. It does not itself update the weights. That happens during the optimizer step.

    Choose an architecture by its information flow

    A Transformer is a neural network architecture built around attention and position-wise transformations. Attention mixes information between token representations by assigning different weights to other positions. A token is a text unit from the model’s vocabulary; it may be a word, a word fragment, or a byte-derived piece.

    FamilyExamplesInformation available at each positionCommon uses
    Encoder-onlyBERT, DistilBERT, RoBERTaBoth earlier and later input tokensClassification, entity extraction, extractive question answering
    Decoder-onlyGPT-2, DistilGPT2The current token and earlier tokens

    Next-token prediction and text generation; also classification with a suitable head

    Encoder–decoderT5, FLAN-T5, BART, Pegasus

    Encoder sees the source; decoder sees earlier output tokens and encoder states

    Translation, summarization, other sequence-to-sequence tasks

    Bidirectional attention allows a token to use context on both sides. Causal attention prevents a token from reading later positions. Autoregressive generation repeatedly predicts a next token using the tokens available so far.

    Consider “The bank approved the loan.” An encoder can use “loan” when building the representation of “bank.” A causal decoder’s representation at “bank” cannot use the later word “loan.” Its representation at the end of the sentence can, however, incorporate all preceding words. That is why a causal decoder can still classify a complete input.

    What the attention mask actually changes

    Inside one attention head, a query vector expresses what a position is looking for, a key vector expresses what another position offers, and a value vector carries the information to mix. A projection is a learned linear transformation that creates such vectors from hidden states.

    Let QQ, KK, and VV be the matrices containing these vectors, dkd_k the key-vector width, and MM an additive mask. Scaled dot-product attention is:

    Attention⁡(Q,K,V)=softmax⁡(QK⊤dk+M)V.\operatorname{Attention}(Q,K,V) =\operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}+M\right)V.

    Here K⊤K^\top is the transpose of KK. Softmax, defined numerically below, turns scores into weights. Allowed positions receive a mask value of zero; blocked positions receive a very large negative value, conceptually negative infinity, so their attention weights become zero.

                     Positions a token may read
    Encoder                    Causal decoder
           1 2 3 4                    1 2 3 4
    token1 ✓ ✓ ✓ ✓             token1 ✓ × × ×
    token2 ✓ ✓ ✓ ✓             token2 ✓ ✓ × ×
    token3 ✓ ✓ ✓ ✓             token3 ✓ ✓ ✓ ×
    token4 ✓ ✓ ✓ ✓             token4 ✓ ✓ ✓ ✓

    The causal mask and the padding mask solve different problems. The causal mask blocks future information. The padding mask blocks artificial filler positions. A GPT classifier needs correct handling of both.

    Turn text into tensors without changing its meaning accidentally

    Tokenization converts text into vocabulary pieces and their integer IDs. An embedding maps each ID to a learned vector. A tensor is a multidimensional array; a batch of token IDs usually has shape [batch_size, sequence_length].

    Always start with the tokenizer that belongs to the pretrained checkpoint. An ID is an address into that checkpoint’s embedding table. An arbitrary replacement tokenizer can give that address a different meaning.

    BERT-style special tokens

    A special token is a reserved symbol used to mark structure rather than an ordinary text fragment.

    [CLS] I really enjoyed this movie [SEP] [PAD] [PAD]
      ↑                                      ↑
    sequence start                     artificial filler
    SymbolPurpose
    [CLS]

    A special first position whose final representation can feed a sequence classifier

    [SEP]A separator or boundary marker
    [PAD]Filler used to make tensors compatible in a batch
    [MASK]A marker for hidden tokens in masked-language-model pretraining
    [UNK]

    A fallback for text the tokenizer cannot represent with its ordinary pieces

    [CLS] does not inherently mean “the sentiment of this sentence.” Its usefulness as a classification representation depends on the architecture and training. DistilBERT’s sequence-classification head transforms the hidden state at the first position and maps it to class scores.

    An uncased tokenizer normalizes capitalization; a cased tokenizer preserves relevant case distinctions. Losing case can remove useful signals for named entity recognition, the task of finding names such as people or locations in text. An uncased model can still learn entity recognition; it simply cannot use the original capitalization signal in the same way.

    Padding, truncation, and the input attention mask

    Padding adds filler positions so examples can share one rectangular tensor. Truncation removes tokens beyond a chosen limit. The attention mask marks real input positions with 1 and padding with 0 in the tokenizer output.

    Illustrative BERT token IDs:
    input_ids:       101 2023 2003 2204 102   0   0
    attention_mask:    1    1    1    1   1   0   0

    An attention mask is not a class label, a confidence value, or a mechanism for restoring truncated text. Once the end of a review is truncated, a final sentence such as “Actually, I would never buy this again” may be missing entirely.

    Static padding pads every example to a fixed length. Dynamic padding pads each batch only to its longest example. Static padding is easy to inspect; dynamic padding often avoids wasted work. The classification examples below use fixed length 128 so their input shapes are explicit.

    For ordinary dense attention, the attention-score matrix has approximately n2n^2 entries per head for sequence length nn. Moving from 128 to 256 tokens makes that matrix four times as large. The whole model’s time and memory do not necessarily increase by exactly four times, because other operations and implementation choices also matter.

    Measure token lengths before selecting the limit. Report the fraction that exceed the cap using untruncated lengths. A processed example of length 128 might originally have had exactly 128 tokens or many more; the capped length alone cannot tell you.

    Separate learning, model selection, and final evaluation

    The training set provides examples used for parameter updates. The validation set guides development choices. The test set estimates final performance on data withheld from those choices. A split is one of these separate dataset partitions.

    Data is split into training, validation, and test partitions. Training updates weights; validation selects a checkpoint; the untouched test partition evaluates the selected model once.

    Validation participates in development decisions. The test set is reserved for evaluating the chosen procedure.

    A hyperparameter is a setting chosen outside the weight-learning process, such as learning rate, batch size, or number of epochs. An epoch is one pass through the training examples. A checkpoint is a saved model state, often including enough information to resume training.

    Selecting the best validation checkpoint is legitimate. Selecting the learning rate by repeatedly checking the test score turns the test set into another development set. A fresh holdout is then needed for an honest final estimate.

    An unseen example should also be independent enough for the intended evaluation. Different rows containing the same tweet, near-duplicate headlines, or texts by the same source can cross split boundaries. For production forecasting, a chronological split may be more informative than a random one. A data leak occurs when information that should be unavailable during development influences training or selection.

    The word “test” does not guarantee an unbiased result. Independence, representative sampling, label quality, and disciplined model selection are the conditions that make the result useful.

    Follow one classification prediction into its loss

    Logits are raw, unrestricted class scores produced before converting them into probabilities. Softmax converts a vector of scores into positive numbers that sum to one. Cross-entropy penalizes assigning too little probability to the correct label.

    Suppose the class order is [negative, neutral, positive], and a model produces the illustrative logits:

    z = [0.0, 1.0, 2.0]
    true label = positive, index 2

    For class ii, with score ziz_i, softmax is:

    pi=ezi∑jezj.p_i=\frac{e^{z_i}}{\sum_j e^{z_j}}.

    The exponentials are approximately [1.0000,2.7183,7.3891][1.0000,2.7183,7.3891]. Their sum is 11.107311.1073, giving:

    negative: 0.0900
    neutral:  0.2447
    positive: 0.6652

    Argmax returns the index of the largest value, so the predicted label is positive. The cross-entropy for correct class yy is:

    L=−log⁡py.L=-\log p_y.

    Here log⁡\log is the natural logarithm. For a positive label, L=−log⁡(0.6652)≈0.4076L=-\log(0.6652)\approx0.4076. If the same input were actually negative, the loss would be −log⁡(0.0900)≈2.4076-\log(0.0900)\approx2.4076. The model receives a larger penalty for putting little probability on the supplied correct answer.

    The following is an independent arithmetic example, with no pretrained model or training required:

    import torch
    
    logits = torch.tensor([[0.0, 1.0, 2.0]], requires_grad=True)
    labels = torch.tensor([2])
    probabilities = torch.softmax(logits, dim=-1)
    loss = torch.nn.functional.cross_entropy(logits, labels)
    loss.backward()
    
    print(probabilities.detach().round(decimals=4))
    print(round(loss.item(), 4))
    print(logits.grad.round(decimals=4))

    Expected arithmetic output, subject to display formatting:

    tensor([[0.0900, 0.2447, 0.6652]])
    0.4076
    tensor([[ 0.0900,  0.2447, -0.3348]])

    The negative gradient on the correct class encourages its logit to increase. Positive gradients on competing classes encourage their logits to decrease. These gradients flow backward into the head and, during full fine-tuning, into the backbone.

    Pass raw logits to PyTorch’s classification cross-entropy function. Applying softmax first changes the input the loss function expects; the function already handles the required log-softmax operation internally. See PyTorch’s CrossEntropyLoss contract.

    The practical loop is therefore: construct a batch, run the forward computation, obtain logits and loss, backpropagate the loss, apply the optimizer update, and clear gradients before the next independent update. Trainer coordinates these operations for us.

    Understand accuracy, precision, recall, and F1 before reporting them

    Accuracy is the fraction of all examples classified correctly. Precision asks what fraction of predictions for a chosen class are correct. Recall asks what fraction of actual members of that class the model finds. F1 is the harmonic mean of precision and recall, rewarding a balance between the two.

    For a chosen positive class, a true positive (TP) is a correctly predicted positive, a false positive (FP) is a negative example predicted positive, a false negative (FN) is a positive example missed, and a true negative (TN) is a correctly predicted negative. Support is the number of actual examples belonging to a class.

    A confusion matrix counts predictions against actual labels. Its row and column meanings must always be stated.

    Illustrative confusion matrix with actual labels in rows and predictions in columns: 40 true positives, 10 false negatives, 20 false positives, and 30 true negatives. Accuracy is 70 percent, precision 66.7 percent, recall 80 percent, and F1 about 72.7 percent.

    These illustrative counts demonstrate why four metrics can describe the same predictions differently.

    There are 100 examples: 50 actual positives and 50 actual negatives. The model predicts positive 60 times, getting 40 of those predictions right.

    Accuracy=TP+TNTP+TN+FP+FN=40+30100=0.70,Precision=TPTP+FP=4060≈0.6667,Recall=TPTP+FN=4050=0.80,F1=2TP2TP+FP+FN=80110≈0.7273.\begin{aligned} \text{Accuracy}&=\frac{TP+TN}{TP+TN+FP+FN}=\frac{40+30}{100}=0.70,\\ \text{Precision}&=\frac{TP}{TP+FP}=\frac{40}{60}\approx0.6667,\\ \text{Recall}&=\frac{TP}{TP+FN}=\frac{40}{50}=0.80,\\ F1&=\frac{2TP}{2TP+FP+FN}=\frac{80}{110}\approx0.7273. \end{aligned}

    “The model finds 80% of positives” describes recall. It does not mean 80% of its positive predictions are reliable; their precision is about 66.7%.

    For multiple classes, calculate these quantities for each class by treating it as the chosen positive class and grouping all others together. Macro F1 averages class F1 scores equally. Weighted F1 averages them in proportion to their supports. Macro F1 gives a rare class the same importance as a frequent one; weighted F1 reflects class frequency in the evaluation set. Neither is a substitute for examining individual classes.

    For example, if class F1 values are [0.90,0.50][0.90,0.50] with supports [90,10][90,10], macro F1 is 0.700.70, while weighted F1 is (90×0.90+10×0.50)/100=0.86(90\times0.90+10\times0.50)/100=0.86. The larger weighted result can coexist with poor performance on the smaller class. The averaging definitions are documented in scikit-learn’s metric reference.

    Some metrics are undefined when their denominator is zero. In the code below, zero_division=0 makes that convention explicit. Also inspect whether a class received any predictions; a zero score can reveal a model that has ignored it entirely.

    The right precision–recall balance depends on error costs. Missing an urgent message and incorrectly flagging a harmless message can have different consequences. A decision threshold is the score cutoff used to choose a class. It can be adjusted using validation data for a binary classifier when the deployment requirement calls for it, but the threshold and its costs must be evaluated on representative data.

    Prepare a reproducible training environment

    The main examples assume Python, a working PyTorch installation for your device, internet access for public model and dataset downloads, and enough disk space for caches and checkpoints. A supported GPU can make full fine-tuning much faster; the code does not require one or enable mixed precision blindly.

    Install PyTorch using the command appropriate for your operating system and accelerator, then install the libraries used here:

    python -m pip install transformers datasets accelerate scikit-learn numpy sentencepiece

    Hugging Face Transformers provides model and tokenizer interfaces. Datasets loads and transforms dataset splits. Accelerate supplies device and distributed-training support used by Trainer. scikit-learn supplies the metrics in these examples. SentencePiece supports the T5 tokenizer used in the final teaching example.

    The API pattern uses eval_strategy in TrainingArguments and processing_class in Trainer, consistent with the current classification guide. Record the versions you actually use and keep a dependency lock for a repeatable experiment. An unpinned install command cannot promise bitwise reproduction over time.

    The following blocks form the shared setup and then a complete sentiment workflow. The news example explicitly creates its own data, model, and trainer; it reuses only shared imports and helper functions.

    import math
    from collections import Counter
    from importlib.metadata import version
    from pathlib import Path
    
    import numpy as np
    import torch
    from datasets import Dataset, DatasetDict, load_dataset
    from sklearn.metrics import (
        accuracy_score,
        classification_report,
        confusion_matrix,
        f1_score,
    )
    from transformers import (
        AutoModelForSequenceClassification,
        AutoModelForSeq2SeqLM,
        AutoTokenizer,
        DataCollatorForSeq2Seq,
        DataCollatorWithPadding,
        Seq2SeqTrainer,
        Seq2SeqTrainingArguments,
        Trainer,
        TrainingArguments,
        set_seed,
    )
    
    SEED = 42
    set_seed(SEED)
    
    for package in ["torch", "transformers", "datasets", "accelerate", "scikit-learn"]:
        print(f"{package}: {version(package)}")
    print("CUDA available:", torch.cuda.is_available())

    A seed initializes random-number generators so an experiment is easier to repeat. It does not eliminate every source of nondeterminism across hardware, libraries, or parallel execution. CUDA availability is only one accelerator check; False does not establish that every possible hardware backend is unavailable.

    The two classifiers use the same metric function. No softmax is needed to find the winning class: it preserves the ordering of logits.

    def compute_classification_metrics(eval_prediction):
        logits, labels = eval_prediction
        if isinstance(logits, tuple):
            logits = logits[0]
        predictions = np.argmax(logits, axis=-1)
        return {
            "accuracy": float(accuracy_score(labels, predictions)),
            "f1_weighted": float(
                f1_score(labels, predictions, average="weighted", zero_division=0)
            ),
            "f1_macro": float(
                f1_score(labels, predictions, average="macro", zero_division=0)
            ),
        }
    
    
    def print_parameter_counts(model):
        total = sum(parameter.numel() for parameter in model.parameters())
        trainable = sum(
            parameter.numel()
            for parameter in model.parameters()
            if parameter.requires_grad
        )
        print(f"Total parameters: {total:,}")
        print(f"Trainable parameters: {trainable:,}")
        print(f"FP32 parameter storage only: {total * 4 / 1_000_000:.1f} MB")
    
    
    def print_classification_details(prediction_output, label_names):
        logits = prediction_output.predictions
        if isinstance(logits, tuple):
            logits = logits[0]
        predictions = np.argmax(logits, axis=-1)
        labels = prediction_output.label_ids
        class_ids = list(range(len(label_names)))
        print(classification_report(
            labels,
            predictions,
            labels=class_ids,
            target_names=label_names,
            digits=4,
            zero_division=0,
        ))
        print("Confusion matrix: actual rows, predicted columns")
        print(confusion_matrix(labels, predictions, labels=class_ids))

    FP32 stores each floating-point parameter in 32 bits, or four bytes. Activations are intermediate values produced as an input passes through the network. Parameter storage is only part of training memory. Gradients, optimizer state, activations retained for backpropagation, temporary workspaces, and framework overhead also require memory. A 268 MB weight estimate is not a 268 MB training-memory estimate.

    Experiment 1: fine-tune DistilBERT for tweet sentiment

    DistilBERT is a smaller BERT-style encoder trained using knowledge distillation, a process in which a smaller model learns from a larger model’s behavior as well as training objectives. We will use its uncased checkpoint with a three-class sentiment head.

    The task is to map an English tweet to negative, neutral, or positive. The dataset card describes the multilingual source and its language subsets; read its provenance and usage conditions before reusing it in a product.

    Load splits and inspect the labels

    SENTIMENT_MODEL = "distilbert/distilbert-base-uncased"
    SENTIMENT_LABELS = ["negative", "neutral", "positive"]
    SENTIMENT_MAX_LENGTH = 128
    SENTIMENT_BATCH_SIZE = 32
    
    sentiment_data = load_dataset(
        "cardiffnlp/tweet_sentiment_multilingual",
        revision="refs/convert/parquet",
        data_dir="english",
    )
    
    for split_name, split in sentiment_data.items():
        counts = Counter(split["label"])
        print(split_name, len(split), dict(sorted(counts.items())))
    
    feature = sentiment_data["train"].features["label"]
    print("Dataset label names:", feature.names)
    assert list(feature.names) == SENTIMENT_LABELS
    assert set(sentiment_data) == {"train", "validation", "test"}

    The explicit Parquet revision selects a data-only representation rather than relying on an older executable loading script. A revision identifies a branch, tag, or commit of a repository. This conversion branch is not an immutable experiment lock; for strict reproduction, record resolved dataset and model commit identifiers along with the dependency versions.

    Recorded split sizes and class counts:

    SplitExamplesNegativeNeutralPositive
    Training1,839613613613
    Validation324———
    Test870290290290

    The validation class counts are deliberately left unspecified here; print them from the data you load. The equal training counts matter: if neutral sentiment is difficult later, fewer neutral training examples cannot explain that result. Balance does not make labels unambiguous or linguistic patterns equally easy.

    Tokenize every split with one policy

    sentiment_tokenizer = AutoTokenizer.from_pretrained(SENTIMENT_MODEL)
    
    
    def tokenize_sentiment(batch):
        return sentiment_tokenizer(
            batch["text"],
            padding="max_length",
            truncation=True,
            max_length=SENTIMENT_MAX_LENGTH,
        )
    
    
    sentiment_tokens = sentiment_data.map(tokenize_sentiment, batched=True)
    sentiment_tokens = sentiment_tokens.remove_columns(["text"])
    sentiment_tokens = sentiment_tokens.rename_column("label", "labels")
    sentiment_tokens.set_format("torch")
    
    sample = sentiment_tokens["train"][0]
    print("Vocabulary:", sentiment_tokenizer.vocab_size)
    print("Input shape:", sample["input_ids"].shape)
    print("Mask shape:", sample["attention_mask"].shape)
    print("First IDs:", sample["input_ids"][:10].tolist())
    print("Real token count:", int(sample["attention_mask"].sum()))

    Recorded tokenization inspection:

    Vocabulary: 30522
    Input shape: torch.Size([128])
    Mask shape: torch.Size([128])
    First IDs: [101, 3100, 1045, 1032, 23343, 24096, 2683, 2213, 3374, 2021]

    The vocabulary contains 30,522 entries. ID 101 is the [CLS] marker for this checkpoint. A single example has shape [128]; a batch of 32 examples has shape [32, 128]. The associated labels tensor contains one integer per example, giving shape [32] for a full batch.

    Renaming the target to labels makes the model’s training interface explicit. Passing labels to the sequence-classification model lets it calculate the task loss. set_format("torch") requests PyTorch tensors when examples are retrieved.

    Attach the new head and check what is trainable

    set_seed(SEED)
    sentiment_model = AutoModelForSequenceClassification.from_pretrained(
        SENTIMENT_MODEL,
        num_labels=len(SENTIMENT_LABELS),
        id2label={index: label for index, label in enumerate(SENTIMENT_LABELS)},
        label2id={label: index for index, label in enumerate(SENTIMENT_LABELS)},
    )
    print_parameter_counts(sentiment_model)

    Recorded parameter inspection:

    Total parameters: 66,955,779
    Trainable parameters: 66,955,779
    FP32 parameter storage only: 267.8 MB

    A warning that classification-head parameters were newly initialized is expected when loading a general pretrained checkpoint into this task-specific model. It means the head still needs training. Warnings about incompatible dimensions or unexpectedly missing backbone weights need separate investigation.

    Configure training and validation

    Batch size is the number of examples processed together. A training step usually refers to a parameter update; gradient accumulation, explained later, can combine several batches into one such update. Weight decay discourages large parameter values through the optimizer’s regularization rule. Regularization means a technique intended to reduce overfitting, where a model fits training examples better than it generalizes to new ones.

    sentiment_arguments = TrainingArguments(
        output_dir="./runs/sentiment-distilbert",
        learning_rate=2e-5,
        per_device_train_batch_size=SENTIMENT_BATCH_SIZE,
        per_device_eval_batch_size=32,
        num_train_epochs=3,
        weight_decay=0.01,
        eval_strategy="epoch",
        save_strategy="epoch",
        load_best_model_at_end=True,
        metric_for_best_model="f1_weighted",
        greater_is_better=True,
        save_total_limit=2,
        report_to="none",
        seed=SEED,
        data_seed=SEED,
    )
    
    sentiment_trainer = Trainer(
        model=sentiment_model,
        args=sentiment_arguments,
        train_dataset=sentiment_tokens["train"],
        eval_dataset=sentiment_tokens["validation"],
        processing_class=sentiment_tokenizer,
        data_collator=DataCollatorWithPadding(sentiment_tokenizer),
        compute_metrics=compute_classification_metrics,
    )
    
    print("Single-device batches per epoch:", math.ceil(
        len(sentiment_tokens["train"]) / SENTIMENT_BATCH_SIZE
    ))

    The single-device, no-accumulation calculation gives ⌈1839/32⌉=58\lceil1839/32\rceil=58 batches per epoch when the incomplete final batch is retained. Integer floor division gives 57 full batches and misses the last 15 examples. Distributed sampling or different loader options can change the actual count.

    TrainingArguments declares the training policy; Trainer connects that policy to a particular model and datasets. A newly created model does not replace the model inside an existing trainer automatically. Creating an explicit trainer for each experiment prevents a subtle and expensive mistake: training the previous task again.

    The chosen learning rate, 2×10−52\times10^{-5}, is a reasonable starting candidate rather than a universal optimum. Three epochs limit this initial experiment; they do not guarantee the absence of overfitting. The sequence limit controls information retention, and the batch size controls both memory and the update statistics.

    At the end of every epoch, this configuration validates and saves a checkpoint. load_best_model_at_end=True restores the saved checkpoint with the highest validation weighted F1. It does not necessarily restore the final epoch. Early stopping is a separate policy that ends training after the validation metric fails to improve for a chosen patience, the number of evaluation checks allowed without sufficient improvement. Simply restoring the best checkpoint does not turn early stopping on. The Trainer reference defines these controls.

    Train, then evaluate the selected model

    The next block performs the actual training when you run it. It downloads weights and uses your local compute; the printed results must come from that run.

    sentiment_training_result = sentiment_trainer.train()
    print(sentiment_training_result.metrics)
    
    sentiment_test_output = sentiment_trainer.predict(
        sentiment_tokens["test"], metric_key_prefix="test"
    )
    print(sentiment_test_output.metrics)
    print_classification_details(sentiment_test_output, SENTIMENT_LABELS)

    Using predict returns the metrics and prediction logits from the same test pass, which lets us inspect class behavior without a second prediction pass. Keep this final test evaluation separate from hyperparameter selection.

    Recorded training summary:

    MeasurementRecorded value
    Training runtime141.8040 seconds
    Training examples per second38.9060
    Optimizer steps per second1.2270
    Reported training loss0.8273
    Completed epochs3

    Recorded held-out test summary:

    MeasurementRecorded value
    Loss0.7470
    Accuracy0.6598
    Weighted F10.6493
    Macro F10.6493

    The training loss is an aggregate reported across training updates, not necessarily the loss of the restored best checkpoint measured afterward. Training and evaluation also differ in behaviors such as dropout, so comparing those two scalar losses alone is not a full generalization diagnosis.

    Dropout randomly suppresses selected activations during training as a regularizer. Evaluation mode disables that random suppression. Runtime and throughput are particular to the recorded environment and should not be used to size a different machine.

    The neutral class changes the interpretation

    ClassPrecisionRecallF1Support
    Negative0.620.890.73290
    Neutral0.570.430.49290
    Positive0.800.660.73290

    These recorded class metrics are rounded. A neutral recall of 0.43 means the model recognizes only about 43% of actual neutral examples. Roughly 57% are assigned another sentiment. It does not mean that 43% of neutral predictions are correct; that would describe precision.

    Neutral examples may lack overt emotion, contain mixed sentiment, or depend on conversational context. Those are hypotheses to test by reading errors, rather than conclusions established by this table. Start with the confusion matrix, inspect the neutral-to-negative and neutral-to-positive pairs, and examine whether annotators would agree on the intended label.

    Macro and weighted F1 match because the test supports are equal. That equality is arithmetic, not evidence that all classes behave similarly. A model that always picks one of three equally frequent classes has accuracy about 33.3%; the observed 66% is above that trivial baseline, but its suitability still depends on the application’s requirements and stronger baselines.

    Inference: make device placement and label mapping explicit

    Inference means using a trained model to produce predictions without updating its weights.

    Inference means using trained weights to make predictions without updating them. The helper below works for either classifier in this article. It returns the chosen label and the full score distribution, so an ambiguous prediction remains inspectable.

    def predict_text(model, tokenizer, text, max_length=128):
        if not isinstance(text, str) or not text.strip():
            raise ValueError("Provide a non-empty string.")
    
        encoded = tokenizer(
            text,
            return_tensors="pt",
            truncation=True,
            max_length=max_length,
        )
        device = next(model.parameters()).device
        encoded = {name: tensor.to(device) for name, tensor in encoded.items()}
    
        model.eval()
        with torch.inference_mode():
            logits = model(**encoded).logits
            probabilities = torch.softmax(logits, dim=-1)[0]
    
        predicted_id = int(probabilities.argmax().item())
        scores = {
            model.config.id2label[index]: float(probabilities[index].item())
            for index in range(probabilities.numel())
        }
        return {
            "label": model.config.id2label[predicted_id],
            "score": scores[model.config.id2label[predicted_id]],
            "all_scores": scores,
        }
    
    
    for text in [
        "I absolutely love this product! Best purchase ever!",
        "Not impressed. Expected much better quality.",
    ]:
        print(predict_text(
            sentiment_trainer.model,
            sentiment_tokenizer,
            text,
            max_length=SENTIMENT_MAX_LENGTH,
        ))

    Trainer may have moved the model onto a GPU. The tokenizer creates CPU tensors by default, so moving them to the model’s device avoids a device mismatch. This helper assumes a normal single-device model, as used here; a sharded deployment needs its own device-management strategy.

    Recorded predictions for those two sentences:

    InputPredicted classLargest softmax score
    “I absolutely love this product! Best purchase ever!”Positive0.810
    “Not impressed. Expected much better quality.”Positive0.433

    The second prediction conflicts with the clear negative meaning of the sentence. Its 0.433 score is also below 0.5; in a three-class problem a winning score need not exceed 0.5. Calibration describes how closely predicted probabilities correspond to observed correctness rates. A softmax score is not automatically a calibrated probability of being correct.

    Save the complete inference package

    sentiment_export = Path("./exports/sentiment-distilbert")
    sentiment_trainer.save_model(str(sentiment_export))
    sentiment_tokenizer.save_pretrained(str(sentiment_export))
    
    # Later, in a fresh process with the same dependencies:
    reloaded_tokenizer = AutoTokenizer.from_pretrained(str(sentiment_export))
    reloaded_model = AutoModelForSequenceClassification.from_pretrained(
        str(sentiment_export)
    )
    print(predict_text(reloaded_model, reloaded_tokenizer, "The service was fine."))

    Save the tokenizer, model configuration, label mapping, preprocessing policy, and evaluation record together. Saving only a weight tensor leaves important parts of the input/output contract unspecified. Resuming the optimizer from a training checkpoint is a different operation from loading an inference package.

    Experiment 2: adapt a causal decoder for news classification

    DistilGPT2 is a smaller GPT-2-style causal decoder. We will attach a four-class sequence-classification head to it. This is supervised classification using a decoder backbone; it is not a text-generation prompt asking the model to spell a category name.

    The AG News dataset has 120,000 training examples and 7,600 test examples across World, Sports, Business, and Sci/Tech. The teaching experiment uses smaller subsets: 2,400 training, 400 validation, and 1,000 test examples.

    Build disjoint subsets explicitly

    Stratification preserves chosen class proportions when constructing a subset or split. We select 700 examples per class from the source training partition, then split them into 600 training and 100 validation examples per class. The test subset comes only from the source test partition.

    NEWS_MODEL = "distilbert/distilgpt2"
    NEWS_LABELS = ["World", "Sports", "Business", "Sci/Tech"]
    NEWS_MAX_LENGTH = 128
    
    news_source = load_dataset("fancyzhx/ag_news")
    assert list(news_source["train"].features["label"].names) == NEWS_LABELS
    
    
    def balanced_subset(split, per_class, seed):
        labels = np.asarray(split["label"])
        random_generator = np.random.default_rng(seed)
        chosen_indices = []
        for class_id in range(len(NEWS_LABELS)):
            candidates = np.flatnonzero(labels == class_id)
            if len(candidates) < per_class:
                raise ValueError(f"Not enough examples for class {class_id}")
            chosen_indices.extend(random_generator.choice(
                candidates, size=per_class, replace=False
            ).tolist())
        random_generator.shuffle(chosen_indices)
        return split.select(chosen_indices)
    
    
    news_development = balanced_subset(news_source["train"], 700, SEED)
    news_split = news_development.train_test_split(
        test_size=400,
        stratify_by_column="label",
        seed=SEED,
    )
    news_data = DatasetDict({
        "train": news_split["train"],
        "validation": news_split["test"],
        "test": balanced_subset(news_source["test"], 250, SEED),
    })
    
    for split_name, split in news_data.items():
        print(split_name, len(split), dict(sorted(Counter(split["label"]).items())))

    Expected subset sizes from this construction:

    train 2400 {0: 600, 1: 600, 2: 600, 3: 600}
    validation 400 {0: 100, 1: 100, 2: 100, 3: 100}
    test 1000 {0: 250, 1: 250, 2: 250, 3: 250}

    This is an explicit new subset-construction policy. Matching the number of examples in an earlier experiment does not establish that the selected rows, random seed, or results match it. The class prior is the frequency of a class before considering an input’s features. Balanced test sampling also changes that prior relative to a production stream; retain natural-frequency evaluation when that is the relevant deployment question.

    Fix GPT-2 padding without changing token positions unnecessarily

    GPT-2’s tokenizer has no dedicated padding token by default. An end-of-sequence token, or EOS token, marks a sequence boundary in conventions that use it. We can reuse it as the filler ID, provided the attention mask marks actual padding and the model configuration knows the selected padding ID.

    news_tokenizer = AutoTokenizer.from_pretrained(NEWS_MODEL)
    print("Default padding token:", news_tokenizer.pad_token)
    
    news_tokenizer.pad_token = news_tokenizer.eos_token
    news_tokenizer.padding_side = "right"
    
    print("Padding token ID:", news_tokenizer.pad_token_id)
    print("EOS token ID:", news_tokenizer.eos_token_id)

    Expected configuration for this checkpoint:

    Default padding token: None
    Padding token ID: 50256
    EOS token ID: 50256

    Position embeddings provide information about token order. GPT-2 uses learned absolute position embeddings. Right padding preserves a real token’s position index regardless of how much padding follows it. The GPT-2 documentation recommends right padding for this reason and explains that its sequence classifier uses the last non-padding token when a padding ID is configured.

    Real content: The market rallied today
    Right padding: The market rallied today PAD PAD PAD
                                          ↑
                        classifier uses the last non-PAD position

    The historical decoder experiment used left padding. Its inspected sample began with ten 50256 IDs and ten zero mask values, had 68 padding positions, and ended with a real-token mask value of 1. That illustrates how padding IDs and masks relate. It is not a requirement that every GPT classifier use left padding.

    If you deliberately left-pad GPT-2 during a direct forward pass, verify the position_ids policy: a padding mask alone does not necessarily renumber absolute positions. Generation libraries may implement different position handling, so a generation recommendation should not be copied into a classification experiment without checking the model path.

    Reusing EOS as PAD does not mean the tokenizer automatically appends EOS to every text. The default GPT-2 tokenizer does not do that. These examples classify ordinary text without explicitly inserted EOS markers. If real EOS tokens must occur inside inputs, a separate PAD token and resized embeddings may be a better contract because padding-ID-based pooling otherwise treats those IDs as filler.

    Inspect lengths before truncation

    def news_token_lengths(batch):
        encoded = news_tokenizer(batch["text"], truncation=False)
        return {"token_length": [len(ids) for ids in encoded["input_ids"]]}
    
    
    news_lengths = news_data["train"].map(news_token_lengths, batched=True)
    lengths = np.asarray(news_lengths["token_length"])
    print("Median tokens:", float(np.median(lengths)))
    print("Examples exceeding the cap:", int((lengths > NEWS_MAX_LENGTH).sum()))
    print("Fraction exceeding the cap:", float((lengths > NEWS_MAX_LENGTH).mean()))

    Recorded length observations were a median of 50 tokens and 12 examples, or 0.5% of 2,400, reaching the 128-token cap. They suggest that 128 was a plausible compromise for that subset, but “reaching the cap” is weaker evidence than measuring original lengths. The code above measures actual exceedances. Neither statistic proves that the discarded information is irrelevant to the label.

    Tokenize and create a separate decoder model

    def tokenize_news(batch):
        return news_tokenizer(
            batch["text"],
            padding="max_length",
            truncation=True,
            max_length=NEWS_MAX_LENGTH,
        )
    
    
    news_tokens = news_data.map(tokenize_news, batched=True)
    news_tokens = news_tokens.remove_columns(["text"])
    news_tokens = news_tokens.rename_column("label", "labels")
    news_tokens.set_format("torch")
    
    set_seed(SEED)
    news_model = AutoModelForSequenceClassification.from_pretrained(
        NEWS_MODEL,
        num_labels=len(NEWS_LABELS),
        id2label={index: label for index, label in enumerate(NEWS_LABELS)},
        label2id={label: index for index, label in enumerate(NEWS_LABELS)},
    )
    news_model.config.pad_token_id = news_tokenizer.pad_token_id
    print_parameter_counts(news_model)

    Recorded parameter inspection:

    Total parameters: 81,915,648
    Trainable parameters: 81,915,648
    FP32 parameter storage only: 327.7 MB

    The new score head maps a token representation to four class scores. At a real final token, causal attention has had access to the preceding input, making that position usable for sequence classification. The head’s weights need training, and pad_token_id must agree between tokenizer and model.

    Compared with the sentiment model, this decoder has more parameters, but parameter count alone does not determine memory or speed. The news experiment uses a smaller device batch of 16 as a practical starting point.

    Train with its own arguments and trainer

    news_arguments = TrainingArguments(
        output_dir="./runs/news-distilgpt2",
        learning_rate=2e-5,
        per_device_train_batch_size=16,
        per_device_eval_batch_size=16,
        num_train_epochs=3,
        weight_decay=0.01,
        eval_strategy="epoch",
        save_strategy="epoch",
        load_best_model_at_end=True,
        metric_for_best_model="f1_weighted",
        greater_is_better=True,
        save_total_limit=2,
        report_to="none",
        seed=SEED,
        data_seed=SEED,
    )
    
    news_trainer = Trainer(
        model=news_model,
        args=news_arguments,
        train_dataset=news_tokens["train"],
        eval_dataset=news_tokens["validation"],
        processing_class=news_tokenizer,
        data_collator=DataCollatorWithPadding(news_tokenizer),
        compute_metrics=compute_classification_metrics,
    )
    
    news_training_result = news_trainer.train()
    print(news_training_result.metrics)
    news_test_output = news_trainer.predict(news_tokens["test"], metric_key_prefix="test")
    print(news_test_output.metrics)
    print_classification_details(news_test_output, NEWS_LABELS)

    Recorded training summary:

    MeasurementRecorded value
    Training runtime211.7052 seconds
    Training examples per second34.0100
    Reported training loss0.5998
    Completed epochs3

    Recorded validation trend:

    EpochLossAccuracyWeighted F1Macro F1
    10.49150.83000.82880.8282
    20.40870.86500.86420.8636
    30.37540.87750.87690.8763

    Lower validation loss and improving F1 across these three epochs are encouraging. They do not establish that additional epochs would continue to help. A single seed also cannot show how sensitive the result is to head initialization or subset selection.

    Read the held-out result and its limitations

    Recorded test summary:

    MeasurementRecorded value
    Loss0.2710
    Accuracy0.8980
    Weighted F10.8982
    Macro F10.8984

    Recorded class metrics:

    ClassPrecisionRecallF1
    World0.910.900.91
    Sports0.970.960.96
    Business0.850.860.86
    Sci/Tech0.870.870.87

    The more precise class hit rates recorded were World 90.20%, Sports 96.00%, Business 86.38%, and Sci/Tech 86.69%. A hit rate among actual members of one class is recall, rather than a distinct form of overall accuracy. The class metrics above are rounded; full counts and a confusion matrix would be needed to reconstruct every aggregate exactly.

    The recorded macro and weighted F1 differ slightly. On the newly constructed exactly balanced test subset, those two averages should be equal apart from negligible numerical effects. The differing recorded values are another reason not to assume that the new subset policy recreates the earlier test rows or class supports.

    Four custom headlines are useful probes, not a new benchmark

    An error probe is a deliberately chosen example that checks a particular behavior. Distribution shift means the relationship between deployment data and training or evaluation data has changed in a relevant way. Out-of-distribution data falls outside the patterns represented by the data used to develop the model.

    news_probes = [
        (
            "UN Security Council holds emergency session over escalating tensions in the Middle East",
            "World",
        ),
        (
            "Manchester United defeat Liverpool 3-1 in dramatic Premier League showdown",
            "Sports",
        ),
        (
            "Federal Reserve raises interest rates by 25 basis points amid inflation concerns",
            "Business",
        ),
        (
            "NASA's Artemis mission successfully deploys new lunar surface habitat module",
            "Sci/Tech",
        ),
    ]
    
    for headline, expected_label in news_probes:
        result = predict_text(news_trainer.model, news_tokenizer, headline, NEWS_MAX_LENGTH)
        print({"expected": expected_label, **result})

    These are classification examples, not factual reporting that the described events happened. Recorded predictions were:

    ExpectedPredictedLargest softmax score
    WorldWorld0.572
    SportsWorld0.694
    BusinessWorld0.577
    Sci/TechSci/Tech0.550

    Only two of four probes were correct. This establishes failures on those sentences. It does not establish a 50% production accuracy, prove distribution shift, or rule out a preprocessing bug. Four hand-written examples are too small and selectively chosen to estimate a deployment rate.

    First verify label order, tokenizer/model pairing, padding, position handling, truncation, and which checkpoint was loaded. Then collect a larger independent sample of recent domain text, label it consistently, and compare errors with the benchmark errors. A shift in vocabulary, style, article length, time period, or class frequencies may help explain the discrepancy after implementation mistakes have been excluded.

    The sports error also illustrates why model score and correctness must be checked separately: a wrong answer can receive a larger softmax score than a correct answer on another example.

    Why 90% versus 66% does not rank GPT above BERT

    The two recorded experiments used different tasks, data, label definitions, and sample sizes. One separated news topics; the other separated sentiment in tweets. Their scores answer different questions.

    PropertyDistilBERT experimentDistilGPT2 experiment
    ArchitectureEncoderDecoder
    AttentionBidirectionalCausal
    TaskTweet sentimentNews topic
    Number of labels34
    ParametersAbout 67 millionAbout 82 million
    Device batch size3216
    Sequence cap128128
    Recorded test accuracy65.98%89.80%

    A credible architecture comparison needs a shared task, the same held-out examples, equivalent label semantics, controlled preprocessing, a stated training budget, and comparable hyperparameter-search effort. The two models should be evaluated across multiple seeds when the difference is small enough for random variation to matter.

    Baseline means a reference method used to judge whether a more complex approach adds value. Useful classification baselines include always predicting the most frequent class and a linear classifier on TF–IDF features, numerical word features weighted by their frequency within a document and rarity across documents. A frozen encoder with only its head trained provides another informative adaptation baseline.

    An encoder is a sensible initial choice for many classification tasks. A decoder can also work. The decision should follow measured quality, latency, memory, licensing, and maintenance needs for the same problem.

    When the output is a sequence: encoder–decoder fine-tuning

    Sequence-to-sequence, or seq2seq, means mapping an input sequence to an output sequence whose length need not match. A summary is a sequence of tokens rather than one of three class IDs.

    An encoder–decoder model has three attention relationships:

    1. Encoder self-attention: queries, keys, and values come from the source representations; input tokens can read both sides.
    2. Decoder masked self-attention: queries, keys, and values come from decoder states; future output tokens are blocked.
    3. Cross-attention: decoder states supply queries, while encoder outputs supply keys and values, connecting the generated output to the source.
    Source text → encoder → source representations ──────────┐
                                                           │ keys and values
    Earlier target tokens → causal decoder → cross-attention┘
                                                   ↓
                                         next output-token logits

    Teacher forcing trains a decoder with the true earlier target tokens available as context. The model’s training interface shifts the target sequence appropriately to form decoder inputs; callers normally supply the target token IDs as labels rather than manually performing another shift.

    For target tokens y1,…,yTy_1,\ldots,y_T and source input xx, the sequence objective sums next-token penalties:

    Lsequence=−∑t=1Tlog⁡p(yt∣y1,…,yt−1,x).L_{\text{sequence}} =-\sum_{t=1}^{T}\log p(y_t\mid y_1,\ldots,y_{t-1},x).

    An implementation may average this sum over valid target tokens. Padding positions should be excluded from the loss. Hugging Face’s seq2seq collator uses label value -100 for that ignored padding convention.

    A minimal training pattern for T5

    T5 casts text tasks as text-to-text transformations. FLAN-T5 is a T5-family checkpoint additionally tuned on collections of instruction tasks. Models such as BART and Pegasus use related encoder–decoder structures with different pretraining choices.

    The next example is a deliberately tiny mechanical demonstration of input/target processing and a training call. Four training pairs and two validation pairs cannot establish useful summarization quality. This block is optional, reuses the shared imports, and has no recorded training output or quality claim.

    SUMMARY_MODEL = "google/flan-t5-small"
    summary_tokenizer = AutoTokenizer.from_pretrained(SUMMARY_MODEL)
    summary_model = AutoModelForSeq2SeqLM.from_pretrained(SUMMARY_MODEL)
    
    summary_data = DatasetDict({
        "train": Dataset.from_dict({
            "document": [
                "The city opened a library on Monday. It has a children's room and free computers.",
                "A storm closed the airport overnight. Flights resumed after morning inspections.",
                "The laboratory released its code and data so other teams could reproduce the study.",
                "The council approved a new park. Construction will begin in June.",
            ],
            "summary": [
                "The city opened a library with a children's room and free computers.",
                "Flights resumed after a storm-related airport closure.",
                "The laboratory shared code and data for reproducibility.",
                "The council approved a park, with construction starting in June.",
            ],
        }),
        "validation": Dataset.from_dict({
            "document": [
                "A clinic extended its opening hours to serve patients after work.",
                "Engineers repaired a bridge and reopened it to traffic on Friday.",
            ],
            "summary": [
                "The clinic extended hours for patients who work.",
                "The repaired bridge reopened on Friday.",
            ],
        }),
    })
    
    
    def tokenize_summary(batch):
        inputs = summary_tokenizer(
            ["summarize: " + text for text in batch["document"]],
            max_length=256,
            truncation=True,
        )
        targets = summary_tokenizer(
            text_target=batch["summary"],
            max_length=64,
            truncation=True,
        )
        inputs["labels"] = targets["input_ids"]
        return inputs
    
    
    summary_tokens = summary_data.map(
        tokenize_summary,
        batched=True,
        remove_columns=["document", "summary"],
    )
    summary_arguments = Seq2SeqTrainingArguments(
        output_dir="./runs/summary-mechanics-demo",
        learning_rate=2e-5,
        per_device_train_batch_size=2,
        per_device_eval_batch_size=2,
        num_train_epochs=1,
        eval_strategy="epoch",
        save_strategy="no",
        predict_with_generate=True,
        generation_max_length=64,
        report_to="none",
        seed=SEED,
    )
    summary_trainer = Seq2SeqTrainer(
        model=summary_model,
        args=summary_arguments,
        train_dataset=summary_tokens["train"],
        eval_dataset=summary_tokens["validation"],
        processing_class=summary_tokenizer,
        data_collator=DataCollatorForSeq2Seq(
            tokenizer=summary_tokenizer,
            model=summary_model,
            label_pad_token_id=-100,
        ),
    )
    summary_trainer.train()
    
    source_text = "The museum reopened after a six-month renovation. Admission is free this weekend."
    summary_inputs = summary_tokenizer(
        "summarize: " + source_text,
        return_tensors="pt",
        truncation=True,
        max_length=256,
    )
    summary_inputs = {
        name: tensor.to(next(summary_trainer.model.parameters()).device)
        for name, tensor in summary_inputs.items()
    }
    summary_trainer.model.eval()
    with torch.inference_mode():
        generated_ids = summary_trainer.model.generate(
            **summary_inputs,
            max_new_tokens=64,
            do_sample=False,
        )
    print(summary_tokenizer.decode(generated_ids[0], skip_special_tokens=True))

    The source and target have independent truncation limits. DataCollatorForSeq2Seq pads the input tensors and masks target padding out of the loss. generate performs repeated next-token inference; a classification argmax over three or four labels cannot replace that process. The summarization task guide provides the broader dataset-scale recipe.

    When replacing the tiny pairs with a real dataset, create independent development and test partitions, inspect long-source truncation, define generation settings, and compare against the original checkpoint. A model already trained for instructions may summarize competently before this task-specific training, so a post-training sample alone cannot establish improvement.

    Why summarization needs different evaluation

    ROUGE is a family of reference-overlap metrics often used in summarization. ROUGE-1 compares individual tokens, or unigrams; ROUGE-2 compares consecutive token pairs, or bigrams; ROUGE-L uses a longest common subsequence, a sequence that preserves order while allowing gaps.

    Consider this illustrative pair, ignoring case and punctuation:

    Reference:  AI improves medical diagnosis
    Prediction: AI improves diagnosis

    The prediction contains three words, all present in the four-word reference. The illustrative ROUGE-1 precision is 3/3=13/3=1, recall is 3/4=0.753/4=0.75, and F1 is about 0.85710.8571.

    The reference bigrams are AI improves, improves medical, and medical diagnosis. The prediction bigrams are AI improves and improves diagnosis. Only one matches: ROUGE-2 precision is 1/21/2, recall is 1/31/3, and F1 is 0.40.4. Tokenization, stemming, normalization, aggregation, and the particular scorer can affect reported values, so record those settings.

    A factuality check asks whether claims in the summary are supported by the source. ROUGE does not establish factual correctness: a fluent summary can reuse source words and still change who did what. Useful evaluation combines overlap, factual consistency, coverage, readability, and domain-specific human review. Exact string accuracy is usually too rigid when multiple summaries can express the same facts.

    Make training efficient without confusing the objective

    Gradient accumulation and effective batch size

    Gradient accumulation combines gradients from several smaller batches before updating weights. This can approximate a larger batch without storing all its examples’ activations at once.

    For device batch size BB, accumulation factor AA, and DD data-parallel devices, the nominal effective batch is:

    Beffective=B×A×D.B_{\text{effective}}=B\times A\times D.

    On one device, a batch of 8 accumulated over 4 microbatches gives an effective batch of 32. A microbatch is one of those smaller forward/backward batches.

    8 examples → forward/backward → keep gradients
    8 examples → forward/backward → add gradients
    8 examples → forward/backward → add gradients
    8 examples → forward/backward → add gradients → one optimizer update

    In TrainingArguments, set per_device_train_batch_size=8 and gradient_accumulation_steps=4 to request that arrangement. It does not make four microbatches take the memory of one example, reduce weight-storage requirements, or guarantee bitwise equivalence to a single batch of 32. Dropout, numeric rounding, partial final batches, and batch-dependent operations can affect equivalence.

    Mixed precision and checkpointing

    Mixed precision uses lower-precision numerical formats for selected operations while retaining higher precision where needed. FP16 and BF16 are 16-bit formats with different numerical ranges and precision. They can improve throughput and memory use on compatible hardware, but support and numerical behavior must be checked.

    Gradient checkpointing saves memory by retaining fewer intermediate activations and recomputing some of them during backpropagation. It trades extra computation for reduced activation memory. It is different from saving model checkpoints to disk.

    If memory runs out, first reduce device batch size and add accumulation if the intended effective batch should stay similar. Then inspect sequence lengths and remove unnecessary padding. Consider a smaller model, compatible mixed precision, activation checkpointing, or PEFT as appropriate. Reducing the sequence cap is a modeling change because it can delete evidence, so evaluate truncation errors after doing it.

    Running several full models in the same Python process also consumes memory. The sentiment, news, and summarization examples are separate experiments; on a limited accelerator, run them in separate processes or release completed trainers and models before starting the next one.

    Diagnose failures with evidence

    ObservationPlausible explanationsNext useful check
    Training loss falls while validation worsens

    Overfitting, an unsuitable learning rate, noisy labels, or mismatch between partitions

    Inspect learning curves and validation errors; compare earlier checkpoints

    Overall accuracy looks good but one class is weak

    Class imbalance, label ambiguity, overlapping language, or poor decision boundaries

    Class support, per-class precision/recall/F1, and the confusion matrix

    Scores are almost always one class

    Label-mapping error, data problem, head-training failure, or genuine imbalance

    Inspect labels and logits on a tiny batch; verify parameter gradients and split counts

    Custom inputs fail despite a strong test score

    Preprocessing mismatch, checkpoint mismatch, selective probes, or distribution shift

    Verify the inference contract, then evaluate a larger independently labeled target sample

    Training is slower than expected

    Device fallback, long padded batches, data loading, evaluation, or checkpoint overhead

    Print the device, inspect token-length statistics, and measure each stage

    The model forgets useful behavior after trainingExcessive updates or narrow training dataCompare the original checkpoint on retained general-purpose tasks

    Catastrophic forgetting is a substantial loss of previously useful capabilities after learning a new task or distribution. It is one reason to compare more than the target-task score when adapting a general-purpose model for continued broad use.

    A useful early sanity check is whether a model can overfit a very small clean training subset. Failure there often reveals a broken training path, unsuitable labels, or an optimization issue. Success only shows that the model can fit those examples; it is not a generalization result.

    After selecting the model, inspect errors by class, source, text length, time period, and any relevant domain groups. Report sample sizes alongside scores. A difference based on a handful of examples is less stable than the same difference observed across a large independent sample.

    For deployment, preserve the exact tokenizer and preprocessing rules, document intended input language and length, measure inference latency and memory, and monitor shifts in label frequencies and error patterns. A latency measurement is the time to complete a request; throughput measures how much work is completed per unit time. Neither can be inferred reliably from a parameter count alone.

    Questions that test whether the mechanism is clear

    1. What makes full fine-tuning different from training a frozen encoder’s head? The backbone parameters also receive updates; that changes adaptation capacity and optimizer-memory requirements.
    2. Why is the classification head often newly initialized? The original checkpoint’s task output is not the new label space, so a compatible task-specific mapping has to be learned.
    3. What does a causal mask prevent? A position cannot read future input positions during its self-attention computation. The final real position can still aggregate earlier context.
    4. Why does a tokenizer attention mask not solve every padding issue? It identifies filler positions for attention, while position IDs and classification pooling have their own contracts.
    5. Why can a predicted class have a score below 0.5? In a multiclass problem, it only needs the largest score, not a majority of the probability mass.
    6. Why inspect neutral recall after reporting sentiment accuracy? Accuracy can hide a class whose actual examples are frequently assigned to other labels.
    7. Why can weighted F1 conceal a rare-class failure? Frequent classes contribute more to the weighted average; macro F1 gives classes equal weight.
    8. Why do the two recorded accuracies not compare architectures? They measure different tasks on different held-out examples under different experimental conditions.
    9. Where do cross-attention queries, keys, and values come from? Queries come from the decoder; keys and values come from encoder outputs.
    10. What would make a fine-tuning improvement convincing? A stronger result against suitable baselines on an untouched relevant test set, with controlled preprocessing, documented selection, and error analysis.

    The experiment to carry forward

    A fine-tuned model is a complete procedure: a task definition, a data split, a tokenizer, a trainable parameter policy, a loss, a selection rule, and an inference contract. The weight file is one component of that procedure.

    For a first serious experiment, choose one task, establish a simple baseline, inspect a small batch down to its labels and shapes, run the complete training path, select with validation data, and evaluate the selected model on the reserved test data. Read the mistakes before increasing the model size or switching to a more elaborate adaptation method.

    The sentiment experiment demonstrates why per-class recall matters. The news experiment demonstrates why strong benchmark performance still needs an inference audit and representative domain evaluation. The encoder–decoder extension shows why changing the output from a label to a sequence changes the objective and evaluation. Those foundations remain necessary when later replacing full updates with LoRA, QLoRA, or other adaptation methods.

    Sources

    1. Hugging Face: text classification
    2. Hugging Face: Trainer and TrainingArguments
    3. Hugging Face: GPT-2 model documentation
    4. Tweet sentiment multilingual dataset
    5. AG News dataset
    6. PyTorch: CrossEntropyLoss
    7. scikit-learn: precision, recall, F-score, and support
    8. Hugging Face: summarization
    9. LoRA: Low-Rank Adaptation of Large Language Models
    10. QLoRA: Efficient Finetuning of Quantized LLMs