Tokenization: From First Principles to Production-Grade Domain-Specific Tokenizers
Trace BPE merges, UTF-8 bytes, WordPiece and Unigram, then train and evaluate domain tokenizers with measured vocabulary trade-offs.
48 min readTopic curriculum
From text and bytes to subword algorithms, domain tokenizer training, held-out evaluation, and model compatibility.
No prior topic is required.
Curated sequence
One continuous research note with worked arithmetic, code, and handwritten teaching diagrams.
Trace BPE merges, UTF-8 bytes, WordPiece and Unigram, then train and evaluate domain tokenizers with measured vocabulary trade-offs.
48 min read