Transformers
Field note · 17 Sept 2026Transformers
FP64, FP32, and INT8 in Transformers
How floating-point precision and integer quantization change storage, arithmetic, accuracy, and deployment choices.
Read the DiscoverySai AI Knowledge Hub
Build durable intuition for modern AI.
Visual explanations for the concepts that last, plus concise field notes on what is changing now.
- Topics
- 1
- Durable Notes
- 3
- Discoveries
- 3
Latest signal
Today’s Discovery
Dated observations stay brief and current; durable explanations live in Notes.
Durable understanding
Explore Notes
Follow intentionally ordered paths instead of piecing together isolated posts.
foundation
Transformers
A structured path through attention, multi-head composition, positional information, and encoder-decoder architecture.
A clear first path
Start Here
Begin with the map, then add the mechanism. Each step is ordered so the next idea has somewhere to land.
Attention mechanism in detail
A step-by-step account of queries, keys, values, masking, scaling, softmax, and the final weighted sum.
Multi-head attention
How parallel attention projections create complementary views and recombine them into one representation.
Positional encoding and the encoder-decoder
Where position, masks, residual paths, encoder blocks, and decoder blocks fit in a complete Transformer.
From the field
Recent Discovery
A small, chronological feed of ideas worth carrying into deeper study.
Looped Transformers versus traditional depth
How repeatedly applying a shared Transformer block differs from stacking separately parameterized layers.
3 min readAutoregressive models, step by step
How next-token factorization, causal masking, training, and decoding turn a prefix into a generated sequence.
2 min read