Attention mechanism in detail
A step-by-step account of queries, keys, values, masking, scaling, softmax, and the final weighted sum.
3 min readTopic curriculum
Transformers
A structured path through attention, multi-head composition, positional information, and encoder-decoder architecture.
Prerequisites
No prior topic is required.
Learning outcomes
- Trace how queries, keys, and values produce a contextual token representation.
- Explain what multiple attention heads add and what they do not guarantee.
- Locate positional information, residual paths, and masking in an encoder-decoder Transformer.
Curated sequence
Curriculum
Attention
How tokens select and combine information from a sequence.
This section is mapped; its first Note is still being reviewed.
Attention mechanism
Queries, keys, values, masks, scaling, and weighted sums.
Multi-head attention
Parallel learned projections and their combined representation.
Multi-head attention
How parallel attention projections create complementary views and recombine them into one representation.
3 min read
Architecture
How positional information and encoder-decoder blocks form a complete Transformer.
Positional encoding and the encoder-decoder
Where position, masks, residual paths, encoder blocks, and decoder blocks fit in a complete Transformer.
3 min read