Every time you send a message to an AI language model, something happens before the model processes a single word. Your text gets broken apart — not into letters, not neatly into words, but into chunks called tokens. This step, called tokenization, is easy to overlook, but it shapes almost everything about how a model understands you, how much it can hold in memory at once, and even how much running it costs.
Tokenization is the process of converting raw text into a sequence of numerical IDs that a model can actually work with. Neural networks cannot process language directly — they operate on numbers. So a tokenizer acts as the translator between human-readable text and the mathematical space the model lives in.
The most widely used approach today is called Byte Pair Encoding, or BPE. It works by starting with a base vocabulary of individual characters or bytes, then repeatedly merging the most frequently appearing adjacent pairs into a single token. Run this process over a large body of text long enough and you end up with a vocabulary of tens of thousands of tokens — a mixture of whole common words, frequent word fragments, punctuation marks, and individual characters for rare cases.
What this means in practice is that common words like "the" or "run" usually become a single token, while longer or less frequent words get split. The word "tokenization" itself might be represented as two or three tokens. Numbers, code, and words from languages that were underrepresented in training data often fragment into far more tokens than you would expect.
This has real consequences. A model with a context window measured in tokens — say, a hundred thousand tokens — cannot simply hold a hundred thousand words. Depending on the content, the actual word count might be considerably lower. Code and certain non-English languages are especially token-hungry, meaning you effectively get less model capacity for the same input.
Tokenization also explains some of the stranger failure modes you may have noticed. Asking a model to count the letters in a word, reverse a string, or spot unusual spellings can trip it up because the model never sees individual characters during inference — it sees tokens. The underlying units of its perception are not what you might assume.
Different model families use different tokenizers trained on different data, which means the same sentence can produce a different number of tokens depending on which system you use. It is one of those low-level infrastructure choices that ripples all the way up to cost, capability, and user experience — which makes it worth understanding even if you are using AI tools rather than building them.
