Tokenisation Explained: How LLMs Read, Split, and Count Text

Type "strawberry" into a language model and ask it to count the letters, and there's a decent chance it gets the answer wrong. Not because the model is bad at spelling, but because it never actually sees the word "strawberry" as a sequence of eleven letters. It sees a couple of numeric codes that stand in for chunks of that word. Everything an LLM does — every prediction, every dollar it costs to run, every sentence it can or can't fit into memory — starts with this invisible step: turning your text into tokens.

Tokenization is the layer between human language and the math underneath it. It's rarely discussed outside of AI engineering circles, but it quietly determines how much you pay for API calls, how much text a model can "remember" in one conversation, why models struggle with certain puzzles, and why the same prompt can behave differently across languages. Understanding it isn't optional trivia — it's one of the few pieces of LLM internals that directly affects product decisions.

This explainer covers what LLM tokenization is and why models use subwords rather than characters or whole words, how subword algorithms such as byte pair encoding work with a concrete example, why tokenization matters now, the practical implications for cost, context design, and multilingual products, and the open questions about where tokenization goes next.

What LLM Tokenization Actually Is

A large language model doesn't operate on letters or words. It operates on vectors of numbers. Before any text reaches the neural network, it has to be converted into a sequence of integers, and after the network produces its output, those integers have to be converted back into text. The component that does this conversion — text to integers, integers to text — is the tokenizer.

The integers themselves are indices into a fixed vocabulary, typically ranging from around 32,000 to over 200,000 entries depending on the model family. Each entry in that vocabulary is a "token," and a token can be a whole word, part of a word, a single character, a punctuation mark, or even a fragment of a multi-byte emoji. The word "unbelievable" might be split into "un," "believ," and "able." The word "the" is almost certainly its own single token because it's so common. A rare technical term or a word in a low-resource language might get chopped into five or six small pieces.

This matters because the model's entire worldview is built at the token level. It doesn't learn grammar rules about English; it learns statistical relationships between token sequences, the same pattern-matching process that transformer architectures use to predict what comes next. When people talk about a model "understanding" language, what's actually happening underneath is a prediction task over a stream of these tokens — given the tokens so far, what's the most probable next token.

Why Not Just Use Characters or Whole Words?

It's worth pausing on why tokenization exists at all, since the two obvious alternatives — feeding in raw characters or feeding in whole words — both have serious drawbacks.

• Character-level tokenization keeps the vocabulary tiny (just the alphabet, digits, and symbols), but it makes every sequence extremely long. A 500-word paragraph might become 2,500+ characters, and transformers scale expensively with sequence length, so this is computationally painful. It also makes it harder for the model to learn meaning, since it has to reconstruct "word-level" concepts from scratch every time.
• Word-level tokenization keeps sequences short and semantically meaningful, but the vocabulary becomes enormous once you account for every inflection, typo, compound word, and proper noun in every language the model needs to support. Worse, any word not seen during training becomes an "unknown" token, and the model has no way to reason about it at all.

Subword tokenization is the compromise that won out, and it's what every major LLM uses today.

How Subword Tokenization Works

The dominant approach is a family of algorithms broadly called subword tokenization, with Byte Pair Encoding (BPE) and its variants being the most widely used. The core idea is simple: start with individual characters (or bytes), then repeatedly merge the most frequently occurring adjacent pairs into new tokens, building up a vocabulary of common chunks.

Here's the process in outline:

1. Start with a large corpus of training text, broken into individual characters or bytes.
2. Count every pair of adjacent symbols and find the most frequent pair.
3. Merge that pair into a single new symbol and add it to the vocabulary.
4. Repeat the counting-and-merging process thousands of times, until the vocabulary reaches a target size (commonly tens of thousands of entries).
5. Once trained, this fixed set of merge rules is applied to any new text to split it into tokens — no further learning happens at inference time.

The effect is that extremely common sequences — whole common words, frequent suffixes like "-ing" or "-tion," common punctuation patterns — end up as single tokens, while rare or novel sequences get broken into smaller, more generic pieces. This gives the tokenizer graceful degradation: it can represent literally any input, including made-up words, typos, code, or foreign scripts, by falling back to smaller and smaller chunks, down to individual bytes if necessary.

Most modern LLMs use a byte-level variant of BPE, which operates on raw UTF-8 bytes rather than characters. This guarantees that any input — any language, any emoji, any binary-adjacent string — can be tokenized without ever hitting an "unknown" symbol, since every possible byte value is already in the base vocabulary.

A Concrete Example

Consider the sentence "Tokenization isn't always intuitive." A typical modern tokenizer might split it something like this:

Notice that the leading space is often bundled into the token itself — that's a common convention (used by GPT-family tokenizers, among others) that lets the tokenizer distinguish "the" at the start of a sentence from " the" appearing mid-sentence, without needing separate rules for whitespace.

The exact split depends entirely on which tokenizer and vocabulary a given model uses. There is no universal token boundary — it's an artifact of the training corpus and algorithm choices made by whoever built that particular model.

Benefits of Subword Tokenization

Subword tokenization became the default because it solves several problems at once. These are the properties that make it work in practice.

Any input can be represented

Byte-level BPE can encode any text, including typos, invented words, emoji, code, and every script, by falling back to smaller pieces when a whole word isn't in the vocabulary. There is no "unknown word" that the model simply can't see. For products that handle user-generated text, that robustness matters: unexpected input degrades gracefully instead of breaking. A misspelled product name or a pasted log line still reaches the model in a form it can work with.

Sequences stay manageable

Common words and fragments become single tokens, so typical text turns into far fewer tokens than characters. Because transformer compute and memory grow with sequence length, that compression directly reduces cost and latency and lets more meaningful content fit into the same context window. Character-level models pay a heavy price here; subword models avoid most of it.

Word structure is partly shared

Splitting words into frequent pieces such as prefixes, stems, and suffixes means related words share tokens. "Tokenize," "tokenization," and "tokenizer" overlap in their pieces, which helps the model generalise across forms it has seen less often. That sharing is especially useful for long or compound words and for technical vocabulary built from common roots.

Vocabulary size stays bounded

A fixed vocabulary of tens or hundreds of thousands of tokens keeps the model's embedding and output layers a reasonable size. Word-level vocabularies would explode with every inflection, name, and typo across languages. Subword vocabularies give designers a single dial to trade sequence length against vocabulary size, which is why newer models have been able to grow their vocabularies to improve efficiency for more languages.

Tokenizers are fast and deterministic

Once trained, a tokenizer applies fixed merge rules. The same input always produces the same tokens, and tokenization is quick enough to run locally before any API call. That predictability is what makes accurate cost estimation and context budgeting possible for the teams building on these models. It also means token-related bugs can be reproduced exactly, which simplifies debugging.

Tokenization Use Cases for Builders

Understanding tokens is not just theory. These are the everyday situations where it changes engineering and product decisions.

Forecasting API costs

Before launching a feature, teams count tokens on a sample of real prompts and responses using the model's own tokenizer. Multiplying by expected volume gives a far more accurate cost forecast than word counts, especially for content with code, structured data, or non-English text. The same counts make it possible to compare providers fairly, since their tokenizers split identical text differently. Repeating the measurement after major prompt changes keeps the forecast honest.

Budgeting context in RAG and agent systems

Retrieval pipelines and long-running agents share a fixed token budget between instructions, history, retrieved documents, and output. Builders measure each component in tokens and decide how many passages to retrieve, when to summarise old turns, and how much room to reserve for the answer. Getting this wrong shows up as truncated context, cut-off responses, or answers that ignore material that was silently dropped to fit the budget.

Pricing and planning multilingual products

Products serving several languages measure token counts per language on representative content. That reveals which markets cost more to serve and how much effective context each language gets. The results feed into pricing, model choice per market, and decisions about whether a model with a more balanced vocabulary is worth adopting.

Choosing data formats

When passing structured data to a model, the format affects token count. Teams compare compact JSON, delimited text, and verbose prose for the same data and pick the most efficient format that the model still reads reliably. On high-volume endpoints, those savings add up to a meaningful share of the bill, and shorter inputs also respond faster.

Debugging odd model behaviour

When a model miscounts letters, mangles numbers, or reacts strangely to unusual spacing, inspecting how the input was tokenized often explains it. Builders use that insight to reformat inputs, such as spacing out characters or normalising whitespace, or to route character-precise tasks to deterministic code instead of the model. A quick look at the token split often saves hours of prompt rewriting aimed at the wrong cause.

Why Tokenization Matters Right Now

Tokenization isn't a historical footnote from early NLP research — it's an active design constraint that every team building on LLMs runs into immediately, usually the first time they get an API bill or hit a context-length error. A few reasons it deserves more attention than it typically gets:

• Pricing is per-token, not per-word or per-character. Every major LLM API charges based on input and output token counts. A prompt that "looks short" in English can be deceptively expensive if it contains a lot of code, rare jargon, or non-English text, all of which tend to tokenize less efficiently than plain English prose.
• Context windows are measured in tokens. When a provider advertises a "200K context window," that's 200,000 tokens, not 200,000 words — and depending on the tokenizer, that could be anywhere from roughly 130,000 to 160,000 English words, or dramatically fewer if the content is code, poetry with unusual formatting, or a language the tokenizer handles inefficiently.
• Token efficiency varies by language. Languages that were underrepresented in a tokenizer's training data — many non-Latin-script languages in particular — often require more tokens to express the same idea than English does. This means users writing in those languages can pay more and fit less into the same context window for equivalent content, a real fairness and cost issue for global products.
• Tokenization explains specific model failure modes. The classic "count the letters in a word" failure, some arithmetic mistakes, and certain quirks in how models handle unusual spacing or formatting all trace back to the fact that the model never sees raw characters — it sees whatever chunks the tokenizer happened to produce.

None of this is a new development tied to a specific release — it's a structural property of how every current-generation LLM works, from the smallest open-weight models to the largest frontier systems. What has changed over time is that vocabularies have grown (from roughly 50,000 tokens in earlier GPT-family models to well over 100,000 in many current ones) and byte-level, language-agnostic tokenization has become close to universal, partly to reduce the cross-language efficiency gap described above.

Tokenization Best Practices for Builders

If you're building products on top of LLMs — or working with a partner on LLM integration — tokenization shows up in very concrete ways, well before it becomes an abstract concern.

Cost Estimation

Because API pricing is token-based, any serious cost modeling needs to work in tokens, not characters or words — see the economics of LLM inference for how that per-token price actually gets set. Most providers publish tokenizer libraries (or compatible open-source equivalents) so you can count tokens locally before sending a request — this is worth doing for any high-volume application, since a rough word-count estimate can be off by 30% or more depending on content type.

Prompt and Context Design

Since the context window is a hard token budget shared between the system prompt, conversation history, retrieved documents, and the model's own output, token efficiency becomes a real engineering constraint in retrieval-augmented generation (RAG) pipelines, long-running agents, and chat applications with long histories. Teams often need to:

• Truncate or summarize older conversation turns once a token budget is approached.
• Choose more token-efficient formats for structured data (compact JSON or delimited text often tokenizes more efficiently than verbose prose describing the same data).
• Be cautious with deeply nested or heavily indented code and markup, which can tokenize less efficiently than expected.

Working Around Character-Level Weaknesses

If a task genuinely requires character-level reasoning — counting letters, reversing a string, checking exact character positions — it often helps to explicitly reformat the input so the tokenizer is more likely to produce single-character tokens, for example by inserting spaces or hyphens between letters ("s-t-r-a-w-b-e-r-r-y"). This doesn't fix the underlying limitation, but it can measurably improve accuracy on this narrow class of problems.

Multilingual Products

Teams building for non-English-first markets should benchmark actual token counts for representative content in their target languages rather than assuming parity with English — a gap covered in more depth in the state of multilingual AI. This affects both cost projections and effective context window size, and it can influence which model or provider is the better fit for a given market.

Logging Token Counts in Production

Token usage drifts as prompts grow, conversation histories lengthen, and retrieved content changes. Record input and output token counts for every request, broken down by feature or endpoint, and watch the trend. A sudden rise usually points to a prompt change, a retrieval setting that returns too much, or a history that is never trimmed. Catching it early keeps both cost and latency under control, and the logs make it easy to test whether an optimisation actually saved anything.

Common Tokenization Mistakes

Teams new to LLMs tend to run into the same token-related surprises. Most are easy to avoid once you know to look for them.

Estimating cost from word counts

Word counts are a rough proxy at best. Code, numbers, rare jargon, markup, and non-English text all tokenize less efficiently than plain English prose. Budgets built on words can be off by a wide margin. Count tokens on real samples with the target model's tokenizer before committing to a cost estimate.

Assuming every model counts the same way

Different model families use different tokenizers and vocabularies, so the same prompt can produce noticeably different token counts. Comparing providers on price per token without accounting for that difference gives a misleading picture. Measure the same content with each candidate's tokenizer.

Treating the context window as words

A context window advertised in tokens holds fewer words than the number suggests, and far fewer for some languages and content types. Teams that design around word counts hit context limits earlier than expected, often in production with long conversations or large retrieved documents. Plan budgets in tokens and leave headroom for output.

Asking the model to do character-precise work

Counting letters, reversing strings, and checking exact positions fight against how the model sees text. Relying on the model for these tasks produces intermittent errors that are hard to trace. Use ordinary code for character-level operations, or at least reformat the input so characters are tokenized individually.

Never revisiting token usage

Prompts and pipelines grow over time. A system prompt that was compact at launch can double as features are added, and retrieval settings can creep upward. Without monitoring, cost and latency rise quietly. Review token usage per feature on a regular schedule, and treat large increases as something to explain rather than accept.

Limitations and Open Questions

Tokenization is a solved-enough engineering problem that it rarely gets redesigned from scratch, but it's not without real downsides, and some of them don't have clean fixes.

There's also a genuinely open research question about whether subword tokenization is the right long-term approach at all. Some newer architectures and research efforts have explored tokenizer-free or byte-level models that skip the discrete tokenization step entirely, feeding raw bytes directly into the network. These approaches remove the tokenizer's biases and quirks but tend to require significantly more compute for the same sequence length, since the model has to work with much longer raw sequences. As of now, subword tokenization remains the practical default across essentially every deployed frontier model, but it's not treated as a permanently settled question inside AI research.

What to Watch Next

A few threads worth tracking if you're following this space:

1. Larger and more balanced vocabularies. Expect continued growth in vocabulary size and more deliberate multilingual balancing, narrowing (though probably not eliminating) the cost and context-window gap between English and other languages.
2. Tokenizer-free architectures maturing. If byte-level or patch-based models close the efficiency gap with subword tokenization, it could remove an entire category of model quirks — but this would require compute efficiency gains that haven't yet been demonstrated at frontier scale.
3. Domain-specific tokenizers. Some providers and open-source projects build specialized tokenizers for code, chemistry notation, or other structured domains, since a general-purpose vocabulary is often inefficient for these use cases. More of this specialization is likely as vertical AI applications grow.
4. Standardized token counting across tools. As more products build on multiple LLM providers, expect more tooling aimed at normalizing cost and context estimates across tokenizers that don't otherwise agree on how to count anything — a task made easier as providers like Anthropic publish their own token-counting documentation and libraries.

Tokenization won't show up in a product demo, and most end users will never need to know it exists. But for anyone building with LLMs — estimating costs, designing prompts, debugging odd model behavior, or choosing between providers — it's one of the few pieces of "under the hood" knowledge that pays off almost immediately.

FAQ

What is tokenization in the context of LLMs?

Tokenization is the process of converting text into a sequence of discrete units called tokens — which can be whole words, subword fragments, or individual characters — so that a language model can process it as numbers. It's the first step in any interaction with an LLM and the last step when converting the model's output back into readable text.

How many words is one token, roughly?

As a rough rule of thumb for English text, one token is about three-quarters of a word, or roughly four characters. This ratio varies significantly by content type — code, non-English languages, and text with unusual formatting typically use more tokens per word. For budgeting, the safest approach is to run a sample of your real content through the tokenizer for the model you plan to use, since rules of thumb can be off by a wide margin for anything other than plain English prose.

Why do LLMs struggle with counting letters or doing simple arithmetic?

Because the model doesn't see individual letters or digits directly — it sees tokens, which often bundle multiple characters together in ways that don't align with the boundaries a counting or math task needs. This mismatch between token boundaries and the structure of the problem is the main reason these specific tasks are unreliable even for otherwise capable models.

Is tokenization the same across all AI models?

No. Each model family typically uses its own tokenizer, trained on its own corpus with its own vocabulary size, so the same sentence can be split into a different number of tokens by different models. This is why token counts and context window comparisons aren't directly portable between providers. A context window of a given size holds a different amount of text depending on the tokenizer, and the same document can cost a different amount to process on two models with identical per-token prices. Compare models using your own content, not advertised numbers alone.

Does tokenization affect API costs?

Yes, directly. Most LLM APIs charge per input and output token, so understanding how your specific content tokenizes is essential for accurate cost estimation, especially for code, structured data, or non-English text, which often tokenize less efficiently than plain English prose. Output tokens are usually priced higher than input tokens, so verbose responses can cost more than long prompts. Logging token counts per request in production makes it easy to spot which features or users drive cost, and to see the effect of prompt changes.

What is Byte Pair Encoding (BPE)?

BPE is an algorithm that builds a subword vocabulary by starting with individual characters or bytes and iteratively merging the most frequently co-occurring pairs into new tokens. It's the basis for the tokenizers used by most current large language models, often in a byte-level variant that guarantees any input can be represented.

Can I reduce the number of tokens my prompts use?

Yes, to a degree. Removing redundant instructions, using more compact data formats, avoiding unnecessary verbose formatting, and trimming conversation history are all practical ways to reduce token usage without changing the underlying request. Providers also publish tokenizer libraries you can use to measure token counts before sending a request. Measure before and after each change.

Why does the same text cost more in some languages than others?

Tokenizers learn their vocabulary from training data, and most training corpora are dominated by English and a few other high-resource languages. Common English words often become a single token, while words in less-represented languages or scripts get split into many small pieces. The same sentence can therefore need several times more tokens in, for example, some Indic or African languages than in English. That raises API costs and uses up more of the context window for the same content, which matters when you plan multilingual products.

Conclusion

Tokenization is easy to ignore because it happens before the interesting part of a language model begins. Yet it decides what the model actually sees, how much each request costs, how much text fits in a context window, and why models still stumble on tasks like counting letters or handling some languages well.

The core ideas are straightforward once laid out. Models work on subword tokens learned from training data, usually through byte pair encoding or a close variant, as a compromise between unwieldy character sequences and an impossibly large word vocabulary. Each model family has its own tokenizer, so token counts, context sizes, and costs do not transfer neatly between providers. Code, structured data, and lower-resource languages usually tokenize less efficiently than English prose.

The open questions, such as tokenizer-free models and fairer multilingual vocabularies, may change this layer over time, but they do not change today's practical advice. Measure your real content with the tokenizer of the model you use, log token counts in production, design prompts and retrieval with token budgets in mind, and test character-level tasks rather than assuming they work. Teams that need help designing token-efficient prompts, RAG pipelines, or multilingual LLM products can find hands-on support from Woyce Technologies. For a production system, start with our LLM integration team.