Nineteen open-weight language models were asked the same 3,135 questions in English, in Modern Standard Arabic, and in five Arabic dialects. Not translated approximations of each other — manually translated and adapted parallel items, same content, same options, same order.
Average accuracy came back at 62.8 in English, 51.9 in Modern Standard Arabic and 47.7 in dialect. Random chance on the same items is 25%.
That is a 10.9-point drop for writing in the formal register your students read in school, and a 15.1-point drop for writing the way they actually speak. It is the closest thing the field has to a clean measurement of what an Arabic-speaking student receives when they use the same product as an English-speaking one.
The measurement
The benchmark is DialectalArabicMMLU, built by researchers at IBM Research AI, NYU Abu Dhabi and MBZUAI. It is a preprint and should be read as one. The construction matters more than the provenance: MMLU-Redux items were manually translated and adapted into Syrian, Egyptian, Emirati, Saudi and Moroccan Arabic, producing 15,675 dialectal items and 21,945 in total across 32 academic and professional domains. Machine translation would have contaminated the result with the translation system's own errors. They did it by hand.
Nineteen models from 1B to 13B parameters were evaluated, each averaged over 5 runs. The per-model table is where the argument lives:
| Model | Dialect avg | MSA | English | English − dialect |
|---|---|---|---|---|
| Falcon-H1-7B-Instruct | 57.2 | 62.4 | 76.5 | 19.3 |
| Nile-Chat-12B | 59.9 | 63.8 | 72.8 | 12.9 |
| gemma-3-12b-it | 58.7 | 62.6 | 73.7 | 15.0 |
| Fanar-1-9B-Instruct | 56.2 | 61.3 | 70.4 | 14.2 |
| jais-13b-chat | 47.0 | 52.0 | 55.3 | 8.3 |
| Mistral-7B-Instruct | 34.5 | 38.0 | 62.8 | 28.3 |
Read the last column and you would conclude that jais-13b-chat is the best Arabic model in the table. It has the smallest gap. Read the first column and you find it near the bottom: 47.0 on dialect, against 57.2 for Falcon-H1-7B, whose gap is more than twice as wide.
A small English-to-Arabic gap is not a quality signal. It usually means the English is weak. This is the single most useful thing in the paper for anyone doing diligence, because the gap is exactly the statistic a vendor will put on a slide. Ask for absolute dialect accuracy. The gap is a ratio between two numbers and it can be improved by making one of them worse.
The other reading of that table is harsher. Mistral-7B-Instruct scores 62.8 in English and 34.5 in dialectal Arabic, on a benchmark where random guessing scores 25. Any product built on an unmodified general-purpose open-weight model is not serving Arabic-speaking students in any meaningful sense. The Maghreb is worst served throughout: 45.0 average across models, against 49.8 for Emirati.
The direction has been replicated. AraDiCE, peer-reviewed at COLING 2025 and built at the Qatar Computing Research Institute from roughly 45,000 machine-translated and human-post-edited samples, found the same significant drop from English to dialectal Arabic in Llama 3 and Mistral. Two research groups, two construction methods, same finding.
Two findings with immediate product consequences
The same paper ran two interventions that a founder can act on this week.
Translating dialectal input into English before answering helps, significantly. Average gain across models was statistically significant at T = −3.54, p = 0.002, driven largely by Mistral-7B at +17.9 points and Qwen3-4B at +27.4. If your pipeline currently passes dialect straight to a general model, an internal translation step is measurable improvement rather than a guess.
Translating into Modern Standard Arabic does not help. The average benefit collapses from 5.6 points to 0.3, at p = 0.24 — not significant. This is counterintuitive and worth sitting with. MSA is the register everyone reaches for when building for the Arab world, and as an intermediate representation it buys nothing.
Telling the model which dialect it is reading makes it worse. Explicit dialect labelling degraded performance significantly across all five dialects. The obvious fix is not a fix.
One caveat on all three: these were measured on multiple-choice question answering, not on open-ended tutoring or generation. They are the best available evidence about pipeline design, and they are evidence from a different task than the one you are probably shipping.
How low the ceiling is
The gap between English and Arabic is one problem. The absolute level is a different, larger one.
BALSAM is a blind, contamination-resistant benchmark of 78 Arabic tasks across 14 categories with 52,000 examples, built by a consortium including KSAA, QCRI, MBZUAI and NYUAD. On a scale of 0 to 1, the best model — SILMA-9B Instruct v1.0 — scores 0.33. Jais-family 13B-chat scores 0.30. GPT-4o scores 0.27.
Nobody is winning this leaderboard. The paper also finds that no model is uniformly competent: Jais and Fanar do well on summarisation and badly on sequence tagging, Grok-2 leads on logic, Iron Horse leads on program execution. A model that handles your Arabic content pipeline well may be poor at the assessment task sitting next to it.
ArabicMMLU is the benchmark closest to an education product, because it is literally built from school exams — 40 tasks, 14,575 multiple-choice questions in MSA, sourced from North Africa, the Levant and the Gulf. Across 35 models, BLOOMZ, mT0, LLaMA2 and Falcon all struggle to reach 50%. The best Arabic-centric model reached 62.3%.
Underneath all of it sits one structural fact, from the Fanar team at QCRI and Hamad Bin Khalifa University: Arabic is roughly 0.5% of web data, for roughly 400 million native speakers. Their own response to that scarcity is instructive. Fanar-27B is continually pre-trained from a Gemma-3-27B backbone on 120 billion curated tokens using 256 NVIDIA H100 GPUs. A national AI programme, with sovereign funding, is fine-tuning somebody else's base model. That is the honest state of the frontier in Arabic, and it is not a reason for despair — it is a description of where the scarce input is.
What this means for a founder in the region
The market you are selling into does not know these numbers. That is both the risk and the opening.
Regional adoption is running well ahead of evaluation. EdTech Hub's policy brief for the region, commissioned by FCDO advisors and published in January 2026, documents adoption concentrated among urban elites, private schools and well-resourced universities, and records that ministries in Jordan, Iraq and Egypt have not embedded AI training into national professional development. The most telling thing about that brief is what it does not contain: not one learning-outcomes evaluation of AI in education anywhere in MENA.
So the buyer cannot check you, and no independent party has checked anyone. In that setting the temptation is to ship the English demo with an Arabic toggle. The measurement above says what the student on the other side of that toggle actually receives.
What to do about it
Benchmark in the variety you sell in, and publish absolute numbers. If your product is used in Egyptian Arabic, an MSA evaluation overstates what your users get. The parallel-item method in DialectalArabicMMLU is reproducible: same questions, manual adaptation, report the score for each variety side by side.
Never quote the gap alone. Report dialect accuracy in absolute terms and let the gap fall out of it. If a competitor is quoting a narrow gap, ask for the English number and watch the claim collapse.
Test the pipeline, not just the model. Internal translation to English before answering is a measured, significant gain on the one task it has been tested on. It is also the sort of design decision that never appears in a demo and determines the whole product's accuracy.
Treat Arabic evaluation as an asset, not overhead. There is no published randomised trial of AI tutoring in Arabic, and no independent evaluation of any Arabic learning product. Whoever builds a credible Arabic evaluation set for their own domain owns something no competitor can buy, and can answer the one question a serious investor will ask.
Assume the Maghreb is the hardest case. It scores lowest across models. If you are pricing a North African expansion off Gulf performance, you are pricing the wrong number.
The 15.1-point gap is not permanent and it is not a verdict on the region. It is a measurement of a data shortage: 0.5% of the web for 400 million people. Products that treat that shortage as something to be measured and closed will outlast products that treat it as something to be papered over with a language toggle.