Find a concept

Search Notes and Discovery

Enter a term to search published Notes and Discovery.

    Topic curriculum

    Tokenization

    From text and bytes to subword algorithms, domain tokenizer training, held-out evaluation, and model compatibility.

    foundation
    Notes
    1
    Reading time
    48 min
    Sections
    1

    Prerequisites

    No prior topic is required.

    Learning outcomes

    • Count frequency-weighted pairs and trace BPE training and inference.
    • Distinguish Unicode, UTF-8 bytes, token IDs, and tokenizer families.
    • Train and evaluate a domain tokenizer without confusing compression with model quality.
    • Choose vocabulary size and preserve the tokenizer-model contract.

    Curated sequence

    Curriculum

    From first principles to production

    One continuous research note with worked arithmetic, code, and handwritten teaching diagrams.