Tokenizer From Scratch: BPE as Learned Compression
A toy-example-first introduction to tokenization and byte-level BPE, followed by a small runnable tokenizer and optional deeper artifact exploration.
Bpe
A toy-example-first introduction to tokenization and byte-level BPE, followed by a small runnable tokenizer and optional deeper artifact exploration.