Autoregressive models, step by step

What is worth noticing

An autoregressive model assigns a sequence probability by predicting each token from the tokens before it:

$$p(x_1,\ldots,x_T)=\prod_{t=1}^{T}p(x_t\mid x_{<t})$$

Training can evaluate all next-token positions in parallel when a causal mask blocks future information. Generation remains sequential: choose one new token, append it to the prefix, and run the next step.

Training one position

For the sequence “attention moves information,” the training pairs are conceptually:

• prefix “attention” → target “moves”
• prefix “attention moves” → target “information”

The model produces logits over its vocabulary. Softmax converts logits into a probability distribution, and the loss rewards probability assigned to the observed next token. A causal mask ensures the representation at “attention” cannot inspect “moves” while making that prediction.

Teacher forcing supplies the real earlier tokens during training. During generation, the model receives its own selected tokens. This mismatch is one reason an early generation error can alter later context.

Executable miniature

The following bigram model is deliberately transparent. It learns counts rather than neural weights, but it exposes the same autoregressive control flow: read a prefix, score next-token candidates, choose one, append, and repeat.

from collections import Counter, defaultdict

tokens = "attention moves information attention selects context".split()
transitions = defaultdict(Counter)

for current, following in zip(tokens, tokens[1:]):
    transitions[current][following] += 1

generated = ["attention"]
for _ in range(4):
    candidates = transitions.get(generated[-1])
    if not candidates:
        break
    next_token = min(candidates, key=lambda token: (-candidates[token], token))
    generated.append(next_token)

print(" ".join(generated))

Output:

attention moves information attention moves

Line by line:

1. tokens supplies the tiny training sequence.
2. transitions maps each token to counts of observed followers.
3. zip(tokens, tokens[1:]) creates adjacent input-target pairs.
4. generated starts with a one-token prompt.
5. Each loop reads candidates after the latest token.
6. The min key selects the highest count; alphabetical order makes ties deterministic.
7. Appending the token changes the prefix for the next iteration.

The repeated phrase is expected: this corpus contains only two continuations after “attention,” and the deterministic tie break chooses “moves.” A neural language model replaces counts with context-dependent logits, but still performs a decoding choice at each step.

Decoding changes the output

Greedy decoding selects the highest-probability token. Sampling draws from the distribution, often after applying temperature or top-k/top-p constraints. Beam search keeps several candidate sequences. These methods do not retrain the model; they change how its predicted distribution is explored.

Limitations

Next-token likelihood does not guarantee factuality, planning, or calibrated confidence. Autoregressive generation can repeat, drift, or commit to an early mistake. Longer output also requires more sequential decoding steps, although key-value caching can avoid recomputing every earlier attention projection from scratch.

The miniature code is not a Transformer and should not be used to infer model quality or runtime. It demonstrates the generation loop and deterministic output only.

Connection to the Notes

The attention mechanism Note explains how a causal mask blocks future keys. The encoder-decoder Note places masked self-attention in the decoder and distinguishes that design from encoder-only models.