Ask a large language model a question in Hindi, Swahili, or Vietnamese, and you'll often get an answer that's grammatically stiff, culturally tone-deaf, or quietly translated through English in the model's "head" before it responds. That gap isn't a rounding error. It's a structural feature of how most frontier models were built, and it's becoming one of the more consequential fault lines in AI development.
For years, progress in AI was measured almost entirely in English-language benchmarks — MMLU scores, HumanEval pass rates, chatbot arena rankings. Meanwhile, the majority of the world's roughly 7,000 living languages, and the billions of people who speak them as a first language, were treated as an afterthought: something to bolt on after the "real" model was trained. That era is ending, not because of altruism, but because language coverage has become a genuine competitive and geopolitical variable. Governments are funding sovereign models. Enterprises are discovering that English-first AI fails their actual customers. And researchers are realizing that "multilingual" often meant "works okay in a dozen European languages" rather than serving the languages spoken by most of humanity.
What "Multilingual AI" Actually Means
The term gets used loosely, so it's worth separating three distinct things that often get bundled together.
Multilingual capability refers to a model's ability to understand and generate text in multiple languages, typically achieved by training on a mixed-language corpus. Nearly every frontier model today claims some degree of this — GPT, Gemini, Claude, and Llama all handle dozens of languages to varying degrees of competence.
Local-language models are a narrower and more deliberate category: models trained or fine-tuned specifically for one language or a small language family, often incorporating regional dialects, scripts, and cultural context that a general multilingual model would flatten or miss. Examples include models built specifically for Bahasa Indonesia, Yoruba, or Thai, where the goal isn't broad coverage but depth in one linguistic and cultural context.
Sovereign AI is a related but distinct policy concept — the idea that a nation or region should have AI infrastructure (compute, data, and models) that it controls, independent of foreign providers. Language is often the entry point for sovereign AI initiatives, but the motivation is as much about data governance and strategic independence as it is about linguistic accuracy.
These three ideas overlap but aren't identical. A multilingual model can still be dominated by English "thinking" under the hood. A local-language model might be excellent linguistically but built on foreign infrastructure. Sovereign AI initiatives sometimes prioritize control over quality. Understanding which problem a given project is actually solving matters for evaluating whether it will work.
The Tokenization Problem Nobody Talks About Enough
A large part of why non-English languages perform worse isn't really about training data volume — it's about tokenization, the process of breaking text into the chunks a model actually processes. Tokenizers are typically built by finding the most statistically efficient way to represent a training corpus, and since that corpus is usually dominated by English and other Latin-script, high-resource languages, the resulting vocabulary is optimized for those languages.
The practical effect: a sentence that takes 10 tokens in English might take 25-40 tokens in Hindi, Amharic, or Burmese to express the same meaning, because the tokenizer has to fall back on splitting words into small, inefficient fragments — sometimes individual bytes — rather than clean sub-word units. This has real consequences beyond quality:
- Cost: If you're billed per token, generating the same content in a low-resource language can cost multiple times more than in English.
- Latency: More tokens means slower generation for the same semantic content.
- Context window efficiency: A model's effective context shrinks dramatically in languages that tokenize poorly, since more tokens are consumed saying less.
- Quality ceiling: Poor tokenization makes it structurally harder for a model to learn clean patterns in that language, compounding the disadvantage.
This is a big part of why simply adding more non-English text to a training run doesn't fully solve the problem — the tokenizer itself needs to be built with those languages in mind from the start, which most weren't.
Why This Matters Right Now
The clearest signal that multilingual AI has moved from a research curiosity to a funded national priority is IndiaAI, the Indian government's mission dedicated to building domestic AI capacity. As part of that effort, IndiaAI is funding the development of 12 large language models (LLMs) and 8 small language models (SLMs) trained specifically on India-specific data — covering the country's linguistic diversity, which includes dozens of official languages and hundreds of additional ones, most of them severely underrepresented in existing frontier model training sets.
This is a meaningfully different approach than what came before. Rather than waiting for global AI labs to improve their Hindi, Tamil, Bengali, or Telugu support as an afterthought, India is directly funding models built around its own linguistic reality from the ground up. It reflects a broader pattern playing out in multiple countries simultaneously: governments concluding that if they wait for foreign, English-centric labs to prioritize their languages, they'll wait indefinitely — so they're building the capability themselves.
The IndiaAI approach is also notable for splitting investment between large models and small ones. SLMs are cheaper to train and run, easier to deploy on modest infrastructure, and often more practical for specific tasks like customer service or government service delivery in a specific regional language — which matters enormously in a country where compute infrastructure and internet reliability vary widely by region.
This kind of state-backed investment changes the economics for everyone building in this space. It signals to startups, universities, and open-source communities that there's now institutional funding — and likely institutional demand — for local-language capability, not just a market gap that individual companies have to fill on their own dime.
Why Local-Language Models Matter for Businesses
For most companies outside the handful of countries where English AI works reasonably well, the gap between "AI is impressive" and "AI is useful for us" runs directly through language quality. A few concrete implications:
Customer-Facing AI Fails Quietly
A chatbot or voice assistant that mistranslates idioms, misreads honorifics, or defaults to formal register when a casual one is expected doesn't crash — it just produces mediocre, slightly-off interactions that erode trust. In markets where a company's actual customers speak a language the underlying model handles poorly, this shows up as lower engagement, higher escalation-to-human rates, and quiet reputational damage that's hard to trace back to the model.
Local Context Isn't Just Vocabulary
Language models trained mostly on English data with translated non-English text bolted on often miss cultural and contextual nuance: regional idioms, local regulatory terminology, culturally specific examples, or even which topics are sensitive in a given market. A financial services chatbot that handles compliance language correctly in English but garbles it in Portuguese isn't just a translation problem — it's a business risk.
Cost Structures Differ by Language
Given the tokenization issue described above, deploying AI at scale in a low-resource language can be meaningfully more expensive per interaction than the same deployment in English, even holding volume constant. Businesses budgeting for multilingual AI rollouts need to account for this rather than assuming uniform per-token costs across languages.
Data Availability Shapes What's Possible
Building or fine-tuning a model for a specific language requires a reasonable volume of quality text and, ideally, human feedback data in that language. For high-resource languages (English, Mandarin, Spanish), this is abundant. For most of the world's languages, it's scarce — which is why techniques like synthetic data generation, cross-lingual transfer learning, and community-sourced datasets have become active areas of work rather than niche research topics.
| Approach | What it does | Strengths | Limitations |
|---|---|---|---|
| General multilingual LLM (fine-tuned) | Adapts an existing large model with additional local-language data | Fast to deploy, leverages existing model reasoning | Tokenization and cultural bias baked in from base model |
| Purpose-built local-language model | Trained from scratch or near-scratch on a target language/region | Better tokenization efficiency, stronger cultural fit | Expensive, needs substantial local data and compute |
| Small language model (SLM) for local use | Compact model optimized for one language/domain | Cheap to run, deployable on limited infrastructure | Narrower capability, less general reasoning |
| Retrieval-augmented translation layer | Uses translation + retrieval on top of an English-first model | Quick to stand up, works with any base model | Adds latency, can lose nuance in translation |
How Businesses Should Approach This
There's no single right answer here — the right approach depends on which languages matter to a given business, how much volume they're dealing with, and how much they can invest in language-specific tooling. A practical way to think through it:
- Audit which languages your actual users need, not which languages you assume matter. Support ticket data, customer surveys, and regional usage stats are more reliable than guesswork.
- Test frontier models directly in those languages before assuming they'll work — benchmark quality, not just "does it respond."
- Measure token cost per language, not just per request, since costs can vary substantially by language even for equivalent content.
- Decide build vs. buy vs. fine-tune based on volume and criticality — a low-volume, low-stakes use case may tolerate a general multilingual model; a high-volume customer-facing product in an underserved language may justify fine-tuning or a purpose-built model.
- Plan for evaluation, not just deployment — quality in underserved languages degrades unpredictably, so ongoing monitoring matters more than it does for well-supported languages.
Limitations and Open Questions
None of this is close to solved, and it's worth being direct about where the real difficulties remain.
Evaluation is still weak. Most benchmarks used to measure model quality — MMLU, HellaSwag, and similar suites — were built for English and translated (often imperfectly) into other languages. A model can score well on a translated benchmark while still producing text that sounds unnatural or culturally off to a native speaker, because the benchmark measures the wrong thing. Building genuinely native-language evaluation suites, with native speakers assessing fluency and cultural fit rather than just factual accuracy, is expensive and unevenly funded across languages.
Low-resource languages face a data chicken-and-egg problem. Languages with little digitized text produce models that perform poorly, which reduces incentive to build products in that language, which means less digital content gets created, which keeps the language low-resource. Breaking this cycle usually requires deliberate investment — government funding, community data collection efforts, or synthetic data generation — rather than waiting for market forces to fix it organically.
"Local-language model" doesn't guarantee quality. Training a model on a target language doesn't automatically make it good — it still needs sufficient data volume, careful tokenizer design, and quality human feedback for fine-tuning. Some local-language efforts underperform general multilingual models simply because they lack the data or compute scale to compete, which is a real risk for smaller national or regional initiatives.
Dialectal and script diversity compounds the challenge. Many languages funded under initiatives like IndiaAI aren't monolithic — they have regional dialects, multiple scripts, or significant code-switching with other languages in everyday use (for instance, mixing Hindi and English mid-sentence, a common pattern known as Hinglish). Models built for a "clean" version of a language can still struggle with how people actually write and speak.
Sovereign AI and open access can pull in different directions. Government-funded models are sometimes released openly, which benefits the broader ecosystem — but they can also be built with restricted access, proprietary licensing, or infrastructure lock-in tied to specific domestic compute providers, which limits how much the wider developer community can build on top of them.
What to Watch Next
A few developments will indicate whether local-language AI is becoming genuinely competitive with English-first models, or remaining a secondary tier:
- Whether tokenizer redesigns become standard practice. Labs that rebuild tokenizers with balanced multilingual vocabularies — rather than retrofitting English-optimized tokenizers — are likely to produce meaningfully better non-English performance. Watch for this becoming a stated design choice rather than an afterthought.
- How IndiaAI's funded models perform once released, and whether other countries follow a similar public-funding model for their own underserved languages.
- Whether small language models close the gap for practical, narrow use cases faster than large general models close it for broad ones. SLMs may end up being the more scalable path to genuine multilingual coverage, given their lower compute requirements.
- Growth in native-language evaluation benchmarks, as opposed to translated versions of English benchmarks — this is a leading indicator of whether the field is taking language quality seriously rather than treating it as a checkbox.
- Enterprise adoption patterns — whether businesses in multilingual markets start treating language-specific model performance as a procurement criterion, the way they already treat data residency or compliance certifications.
FAQ
What is a multilingual AI model?
A multilingual AI model is a language model trained or adapted to understand and generate text across multiple languages, rather than being limited to one. Quality varies significantly by language depending on how much training data and tokenizer optimization went into each one, so "multilingual" doesn't guarantee equal performance across all supported languages.
Why do AI models perform worse in some languages than others?
Performance gaps mostly stem from training data imbalance and tokenization: most frontier models are trained predominantly on English and other high-resource languages, and their tokenizers are optimized for those languages, making other languages more expensive and harder for the model to process efficiently. Cultural and contextual nuance in underrepresented languages also tends to get lost when training data is thin.
What is a local-language model, and how is it different from a multilingual model?
A local-language model is built or fine-tuned specifically for one language or region, aiming for depth and cultural accuracy rather than broad coverage. A general multilingual model spreads its training across many languages and often handles each one less precisely than a model purpose-built for that language.
What is sovereign AI?
Sovereign AI refers to a country or region developing and controlling its own AI infrastructure — compute, models, and data — rather than relying entirely on foreign providers. Language localization is often a starting point for sovereign AI initiatives, but the underlying motivation includes data governance, strategic independence, and national security considerations.
Why is India investing in local-language AI models?
India's linguistic diversity — dozens of official languages and hundreds of additional ones — is poorly served by existing frontier models trained mainly on English and other high-resource languages. Government-backed programs, such as IndiaAI's funding of LLMs and SLMs trained on India-specific data, aim to close that gap by building models designed around the country's actual linguistic landscape rather than waiting for foreign labs to prioritize it.
How does tokenization affect the cost of using AI in different languages?
Tokenizers are typically optimized for high-resource languages like English, meaning the same sentence can require far more tokens to represent in a low-resource language. Since usage costs and context limits are usually measured in tokens, this can make deploying AI in some languages several times more expensive and less context-efficient than in English, for equivalent content.
Should my business build a custom local-language model or fine-tune an existing one?
It depends on volume, criticality, and available data: fine-tuning an existing multilingual model is usually faster and cheaper, and suffices for moderate-volume or lower-stakes use cases, while building a purpose-built model makes more sense for high-volume, customer-facing applications where language quality directly affects trust and revenue. Testing frontier models directly in your target language, and measuring both quality and token cost, should come before committing to either path.
Teams evaluating how multilingual or local-language AI fits their product or market can find hands-on help from Woyce Technologies.