Tokenization

Tokenization converts raw text into the units a model can process. Most systems use subword algorithms such as byte-pair encoding, which balance vocabulary size against the ability to represent rare words. Good tokenization keeps common sequences intact while still handling unusual spellings and code. It is the first step in every NLP pipeline.