Type "strawberry" into a language model and ask it to count the letters, and there's a decent chance it gets the answer wrong. Not because the model is bad at spelling, but because it never actually sees the word "strawberry" as a sequence of eleven letters. It sees a couple of numeric codes that stand in for chunks of that word. Everything an LLM does — every prediction, every dollar it costs to run, every sentence it can or can't fit into memory — starts with this invisible step: turning your text into tokens.
Tokenization is the layer between human language and the math underneath it. It's rarely discussed outside of AI engineering circles, but it quietly determines how much you pay for API calls, how much text a model can "remember" in one conversation, why models struggle with certain puzzles, and why the same prompt can behave differently across languages. Understanding it isn't optional trivia — it's one of the few pieces of LLM internals that directly affects product decisions.
What Tokenization Actually Is
A language model doesn't operate on letters or words. It operates on vectors of numbers. Before any text reaches the neural network, it has to be converted into a sequence of integers, and after the network produces its output, those integers have to be converted back into text. The component that does this conversion — text to integers, integers to text — is the tokenizer.
The integers themselves are indices into a fixed vocabulary, typically ranging from around 32,000 to over 200,000 entries depending on the model family. Each entry in that vocabulary is a "token," and a token can be a whole word, part of a word, a single character, a punctuation mark, or even a fragment of a multi-byte emoji. The word "unbelievable" might be split into "un," "believ," and "able." The word "the" is almost certainly its own single token because it's so common. A rare technical term or a word in a low-resource language might get chopped into five or six small pieces.
This matters because the model's entire worldview is built at the token level. It doesn't learn grammar rules about English; it learns statistical relationships between token sequences. When people talk about a model "understanding" language, what's actually happening underneath is a prediction task over a stream of these tokens — given the tokens so far, what's the most probable next token.
Why Not Just Use Characters or Whole Words?
It's worth pausing on why tokenization exists at all, since the two obvious alternatives — feeding in raw characters or feeding in whole words — both have serious drawbacks.
- Character-level tokenization keeps the vocabulary tiny (just the alphabet, digits, and symbols), but it makes every sequence extremely long. A 500-word paragraph might become 2,500+ characters, and transformers scale expensively with sequence length, so this is computationally painful. It also makes it harder for the model to learn meaning, since it has to reconstruct "word-level" concepts from scratch every time.
- Word-level tokenization keeps sequences short and semantically meaningful, but the vocabulary becomes enormous once you account for every inflection, typo, compound word, and proper noun in every language the model needs to support. Worse, any word not seen during training becomes an "unknown" token, and the model has no way to reason about it at all.
Subword tokenization is the compromise that won out, and it's what every major LLM uses today.
How Subword Tokenization Works
The dominant approach is a family of algorithms broadly called subword tokenization, with Byte Pair Encoding (BPE) and its variants being the most widely used. The core idea is simple: start with individual characters (or bytes), then repeatedly merge the most frequently occurring adjacent pairs into new tokens, building up a vocabulary of common chunks.
Here's the process in outline:
- Start with a large corpus of training text, broken into individual characters or bytes.
- Count every pair of adjacent symbols and find the most frequent pair.
- Merge that pair into a single new symbol and add it to the vocabulary.
- Repeat the counting-and-merging process thousands of times, until the vocabulary reaches a target size (commonly tens of thousands of entries).
- Once trained, this fixed set of merge rules is applied to any new text to split it into tokens — no further learning happens at inference time.
The effect is that extremely common sequences — whole common words, frequent suffixes like "-ing" or "-tion," common punctuation patterns — end up as single tokens, while rare or novel sequences get broken into smaller, more generic pieces. This gives the tokenizer graceful degradation: it can represent literally any input, including made-up words, typos, code, or foreign scripts, by falling back to smaller and smaller chunks, down to individual bytes if necessary.
Most modern LLMs use a byte-level variant of BPE, which operates on raw UTF-8 bytes rather than characters. This guarantees that any input — any language, any emoji, any binary-adjacent string — can be tokenized without ever hitting an "unknown" symbol, since every possible byte value is already in the base vocabulary.
A Concrete Example
Consider the sentence "Tokenization isn't always intuitive." A typical modern tokenizer might split it something like this:
| Text fragment | Likely token(s) |
|---|---|
| "Tokenization" | "Token" + "ization" |
| " isn't" | " isn" + "'t" |
| " always" | " always" (single token, very common word) |
| " intuitive" | " intuitive" or " int" + "uitive" |
| "." | "." |
Notice that the leading space is often bundled into the token itself — that's a common convention (used by GPT-family tokenizers, among others) that lets the tokenizer distinguish "the" at the start of a sentence from " the" appearing mid-sentence, without needing separate rules for whitespace.
The exact split depends entirely on which tokenizer and vocabulary a given model uses. There is no universal token boundary — it's an artifact of the training corpus and algorithm choices made by whoever built that particular model.
Why Tokenization Matters Right Now
Tokenization isn't a historical footnote from early NLP research — it's an active design constraint that every team building on LLMs runs into immediately, usually the first time they get an API bill or hit a context-length error. A few reasons it deserves more attention than it typically gets:
- Pricing is per-token, not per-word or per-character. Every major LLM API charges based on input and output token counts. A prompt that "looks short" in English can be deceptively expensive if it contains a lot of code, rare jargon, or non-English text, all of which tend to tokenize less efficiently than plain English prose.
- Context windows are measured in tokens. When a provider advertises a "200K context window," that's 200,000 tokens, not 200,000 words — and depending on the tokenizer, that could be anywhere from roughly 130,000 to 160,000 English words, or dramatically fewer if the content is code, poetry with unusual formatting, or a language the tokenizer handles inefficiently.
- Token efficiency varies by language. Languages that were underrepresented in a tokenizer's training data — many non-Latin-script languages in particular — often require more tokens to express the same idea than English does. This means users writing in those languages can pay more and fit less into the same context window for equivalent content, a real fairness and cost issue for global products.
- Tokenization explains specific model failure modes. The classic "count the letters in a word" failure, some arithmetic mistakes, and certain quirks in how models handle unusual spacing or formatting all trace back to the fact that the model never sees raw characters — it sees whatever chunks the tokenizer happened to produce.
None of this is a new development tied to a specific release — it's a structural property of how every current-generation LLM works, from the smallest open-weight models to the largest frontier systems. What has changed over time is that vocabularies have grown (from roughly 50,000 tokens in earlier GPT-family models to well over 100,000 in many current ones) and byte-level, language-agnostic tokenization has become close to universal, partly to reduce the cross-language efficiency gap described above.
Practical Implications for Builders
If you're building products on top of LLMs, tokenization shows up in very concrete ways, well before it becomes an abstract concern.
Cost Estimation
Because API pricing is token-based, any serious cost modeling needs to work in tokens, not characters or words. Most providers publish tokenizer libraries (or compatible open-source equivalents) so you can count tokens locally before sending a request — this is worth doing for any high-volume application, since a rough word-count estimate can be off by 30% or more depending on content type.
Prompt and Context Design
Since the context window is a hard token budget shared between the system prompt, conversation history, retrieved documents, and the model's own output, token efficiency becomes a real engineering constraint in retrieval-augmented generation (RAG) pipelines, long-running agents, and chat applications with long histories. Teams often need to:
- Truncate or summarize older conversation turns once a token budget is approached.
- Choose more token-efficient formats for structured data (compact JSON or delimited text often tokenizes more efficiently than verbose prose describing the same data).
- Be cautious with deeply nested or heavily indented code and markup, which can tokenize less efficiently than expected.
Working Around Character-Level Weaknesses
If a task genuinely requires character-level reasoning — counting letters, reversing a string, checking exact character positions — it often helps to explicitly reformat the input so the tokenizer is more likely to produce single-character tokens, for example by inserting spaces or hyphens between letters ("s-t-r-a-w-b-e-r-r-y"). This doesn't fix the underlying limitation, but it can measurably improve accuracy on this narrow class of problems.
Multilingual Products
Teams building for non-English-first markets should benchmark actual token counts for representative content in their target languages rather than assuming parity with English. This affects both cost projections and effective context window size, and it can influence which model or provider is the better fit for a given market.
Limitations and Open Questions
Tokenization is a solved-enough engineering problem that it rarely gets redesigned from scratch, but it's not without real downsides, and some of them don't have clean fixes.
| Limitation | Why it's hard to fix |
|---|---|
| Poor arithmetic and character-level reasoning | Numbers and letters get chunked inconsistently, so the model isn't reliably seeing individual digits or characters as such |
| Tokenizer bias toward high-resource languages | Fixing this requires retraining the tokenizer on more balanced data, which is a foundational change most providers make rarely, not per-release |
| Wasted tokens on repetitive structure (whitespace, markup) | Some gains are possible with smarter tokenizers, but very verbose formats will always cost more tokens |
| Vocabulary is frozen after training | New slang, brand names, or technical terms coined after training either get split awkwardly or treated as out-of-vocabulary combinations, with no way to add single tokens without retraining |
| Different models use different tokenizers | Token counts aren't portable across model families, which complicates cost comparisons and makes "just switch providers" harder than it sounds for token-sensitive applications |
There's also a genuinely open research question about whether subword tokenization is the right long-term approach at all. Some newer architectures and research efforts have explored tokenizer-free or byte-level models that skip the discrete tokenization step entirely, feeding raw bytes directly into the network. These approaches remove the tokenizer's biases and quirks but tend to require significantly more compute for the same sequence length, since the model has to work with much longer raw sequences. As of now, subword tokenization remains the practical default across essentially every deployed frontier model, but it's not treated as a permanently settled question inside AI research.
What to Watch Next
A few threads worth tracking if you're following this space:
- Larger and more balanced vocabularies. Expect continued growth in vocabulary size and more deliberate multilingual balancing, narrowing (though probably not eliminating) the cost and context-window gap between English and other languages.
- Tokenizer-free architectures maturing. If byte-level or patch-based models close the efficiency gap with subword tokenization, it could remove an entire category of model quirks — but this would require compute efficiency gains that haven't yet been demonstrated at frontier scale.
- Domain-specific tokenizers. Some providers and open-source projects build specialized tokenizers for code, chemistry notation, or other structured domains, since a general-purpose vocabulary is often inefficient for these use cases. More of this specialization is likely as vertical AI applications grow.
- Standardized token counting across tools. As more products build on multiple LLM providers, expect more tooling aimed at normalizing cost and context estimates across tokenizers that don't otherwise agree on how to count anything.
Tokenization won't show up in a product demo, and most end users will never need to know it exists. But for anyone building with LLMs — estimating costs, designing prompts, debugging odd model behavior, or choosing between providers — it's one of the few pieces of "under the hood" knowledge that pays off almost immediately.
FAQ
What is tokenization in the context of LLMs?
Tokenization is the process of converting text into a sequence of discrete units called tokens — which can be whole words, subword fragments, or individual characters — so that a language model can process it as numbers. It's the first step in any interaction with an LLM and the last step when converting the model's output back into readable text.
How many words is one token, roughly?
As a rough rule of thumb for English text, one token is about three-quarters of a word, or roughly four characters. This ratio varies significantly by content type — code, non-English languages, and text with unusual formatting typically use more tokens per word.
Why do LLMs struggle with counting letters or doing simple arithmetic?
Because the model doesn't see individual letters or digits directly — it sees tokens, which often bundle multiple characters together in ways that don't align with the boundaries a counting or math task needs. This mismatch between token boundaries and the structure of the problem is the main reason these specific tasks are unreliable even for otherwise capable models.
Is tokenization the same across all AI models?
No. Each model family typically uses its own tokenizer, trained on its own corpus with its own vocabulary size, so the same sentence can be split into a different number of tokens by different models. This is why token counts and context window comparisons aren't directly portable between providers.
Does tokenization affect API costs?
Yes, directly. Most LLM APIs charge per input and output token, so understanding how your specific content tokenizes is essential for accurate cost estimation, especially for code, structured data, or non-English text, which often tokenize less efficiently than plain English prose.
What is Byte Pair Encoding (BPE)?
BPE is an algorithm that builds a subword vocabulary by starting with individual characters or bytes and iteratively merging the most frequently co-occurring pairs into new tokens. It's the basis for the tokenizers used by most current large language models, often in a byte-level variant that guarantees any input can be represented.
Can I reduce the number of tokens my prompts use?
Yes, to a degree. Removing redundant instructions, using more compact data formats, avoiding unnecessary verbose formatting, and trimming conversation history are all practical ways to reduce token usage without changing the underlying request. Providers also publish tokenizer libraries you can use to measure token counts before sending a request.
Teams that need help designing token-efficient prompts, RAG pipelines, or multilingual LLM products can find hands-on support from Woyce Technologies.