Direct answer
A Transformer combines token representations with positional information, then repeatedly applies attention and a position-wise feed-forward network through residual paths and normalization. In an encoder-decoder model, the encoder builds contextual source representations; the decoder reads earlier target tokens and attends to the encoder while producing the next-token distribution.
Why position must be added
Self-attention compares content at all positions but, by itself, does not encode whether a token came first, last, or three steps away. Position information breaks that symmetry.
The original Transformer adds fixed sine and cosine signals to token embeddings:
Other valid designs include learned absolute embeddings, relative-position biases, and rotary position embeddings. They are not interchangeable implementation details: each changes how position enters attention and may behave differently beyond trained sequence lengths.
Encoder block
The original encoder repeats two sublayers:
- Multi-head self-attention, where every non-padding source position can read other source positions.
- A feed-forward network applied independently at each position.
Each sublayer sits on a residual path and is paired with normalization. The feed-forward network changes features within each token; attention moves information between tokens.
After the final encoder block, the output acts as memory for decoder cross-attention. It contains contextual source representations rather than a single compressed sentence vector.
Decoder block
The original decoder repeats three sublayers:
- Masked self-attention over the target prefix.
- Cross-attention whose queries come from the decoder and whose keys and values come from encoder output.
- A position-wise feed-forward network.
The causal mask is essential during autoregressive training. Position (t) may use target tokens up to (t), but not later ground-truth tokens. Without that mask, training would leak the answer the model is supposed to predict.
End-to-end information flow
| Stage | Reads from | Produces |
|---|---|---|
| Source embedding + position | Source token IDs | Position-aware source vectors |
| Encoder stack | Whole unmasked source | Contextual encoder memory |
| Target embedding + position | Known target prefix | Position-aware target vectors |
| Masked decoder attention | Earlier target positions | Prefix context |
| Cross-attention | Encoder memory | Source-conditioned decoder state |
| Output projection | Final decoder state | Next-token logits |
Encoder-only, decoder-only, and encoder-decoder models keep different subsets of this blueprint. A decoder-only language model has causal self-attention but no separate encoder or cross-attention. Calling every Transformer “encoder-decoder” hides that distinction.
Limitations and design choices
Dense attention still has a quadratic score matrix in sequence length. Fixed absolute positions can extrapolate poorly in some settings; newer methods address position differently but introduce their own assumptions. Residual and normalization placement also varies across model families, so diagrams should name the architecture they describe rather than presenting one order as universal.
Interview takeaway
Position makes order available; encoder self-attention builds source memory; masked decoder self-attention preserves causality; and cross-attention connects generated tokens to the encoded source.