Tokens

> A fixed word-level vocabulary maps `walks`, `talker`, and `stalking` to the same unknown token in this toy corpus. Eight learned BPE merges preserve more lexical detail, while also exposing a misleading `s` + `talk` + `ing` segmentation that frequency alone cannot judge semantically.