In this post, we share the results of the 2025 Open Data shared task at WMT, a release note for FLORES+ v4, and a couple of other updates on open datasets for massively multilingual machine translation.
As we told in the previous post, OLDI is a group of researchers maintaining datasets for machine translation research in a large number of languages: FLORES+, a benchmark for translation in over 225 language varieties, and OLDI-Seed, a training dataset for machine translation in 45 languages, mostly low-resourced ones.
This year, we organized for the second time a shared task at WMT25, a conference on machine translation. The goal of the shared task was to provide revisions or extensions to FLORES+, Seed, or other open multilingual datasets, or to create new such datasets. As with the other shared tasks, the participants had an opportunity to publish a paper about their contribution, enhancing the visibility of their language in the NLP research community, and eventually boosting the development of language technologies for it.
This year, our shared task received eight submissions (described in our paper from which this post copypastes extensively), including revisions of FLORES+:
Oktem et al revised FLORES+ and OLDI-Seed sentences in Standard Moroccan Tamazight as a part of the Awal initiative, correcting various mistakes in about 40% of the sentences. They also found that fine-tuning NLLB on the corrected version of Seed (combined with other Tamazight) datasets yields improvements in Tamazight-English translation and SOTA English-Tamazight translation.
Mæhlum et al revised the FLORES+ dataset in Norwegian Bokmål (where whopping 70% of sentences required revisions of grammatical and lexical mistakes) and created a new version of it in Radical Bokmål, an alternative written standard for the Norwegian language.
… extensions of FLORES+:
Mamasaidov et al extended FLORES+ to Southern Uzbek, a variety spoken in Afghanistan and written in Arabic script. They released lutfiy, a suite of a translation model, dataset, and post-correction scripts for Southern Uzbek translation.
Frontull et al translated FLORES+ into Val Badia and Gherdëina, two varieties of the Ladin language spoken in Northern Italy. They also released a bunch of parallel training datasets connecting these two languages and Italian, and demonstrated that this data is valuable for finetuning NLLB or retrieving few-shot examples for LLM-base translation of Ladin.
… extensions of OLDI-Seed:
Marmonier et al translated OLDI-Seed into French, with the purpose of serving as a pivot language for the under-resourced regional languages of France. The translations were created by post-editing LLM translations, with the post-edition data available here for potential preference optimization experiments.
Jumashev et al expanded OLDI-Seed to the Kyrgyz language and demonstrated that this dataset is useful for fine-tuning diverse translation models for translation between English and Kyrgyz.
… extensions of other datasets:
Vamvas et al expanded the WMT24++ evaluation benchmark with six (6) varieties of the Romansh language spoken in Switzerland (the data available here; the code is here). They found that modern LLMs already understand Romansh relatively well, but still struggle with translation into its varieties.
… and creation of entirely new multilingual datasets:
Caswell et al created the SMOL dataset: a multiway parallel training dataset with high lexical coverage. It consists of two parts: SmolSent is based on 863 English sentences with diverse words selected from the web, and SmolDoc is based
on 584 English LLM-generated documents with with diverse topics and styles. The dataset has been translated by professionals and extended by community volunteers into 123 languages already, and is still gaining momentum.
Given that SMOL is much easier that Seed and its value for improving translation quality is empirically validated, we recommend it as the default choice of the first multilingual dataset to expand into a new language.
Overall, the shared task has been pretty successful, helping enhance machine translation resources and technologies for quite a few language varieties.
The shared task also highlighted certain challenges, most of all:
The translation quality of major multilingual datasets: as Tamazight and Bokmål examples demonstrated, even supposedly double-validated professional translations could be swarmed with errors.
The unreasonable difficulty of their extension and revision: the participants who translated the Seed dataset keep reporting the high complexity of its content.
In the future, we hope to address the first problem by developing human-computer interaction tools (such as grammatical error detectors) for helping validate translations in diverse languages. As for the second problem, it is already partially addressed by introducing more translator-friendly datasets, such as SMOL for training and BOUQuET for evaluation.
Based on the contributions of the shared task, we released an update to the OLDI-Seed dataset: version 3.0, including the added French and Kyrgyz translations and updated Tamazight translations.
In 2025, FLORES+ already had a major release: version 3.0, described in the previous post. Based on the shared task contributions, we have upgraded it to the current latest version, 4.2, which included the following changes:
Updates for Ladin, Norwegian Bokmål, Southern Uzbek, and Standard Moroccan Tamazight, mentioned above.
Addition of the devtest split for the Meadow Mari language, contributed without publishing a paper — but still very useful!
Addition of the Mauritian Creole language by Rajcoomar, 2025, the contribution to the CreoleMT shared task that resulted in a nice by-product of translating FLORES into this language.
Also, for the purpose of distinguishing the two varieties of Norwegian Bokmål (the moderate variety and the newly contributed radical variety), as well as any future orthographical variations in the dataset, we introduced to FLORES+ since its version 4.0 the new field: variant. It will denote the variations not covered by our existing tags for identifying the language, its writing system, and its dialect.
This year has been pretty fruitful for us (and it’s not even over yet!). And next year, we could be doing a lot of new other wonderful stuff. We haven’t made our minds yet, but there are ideas, such as:
Publishing a live leaderboard of open models’ translation quality based on FLORES+;
Doing yet another round of FLORES+ revisions, fixing the errors found in English sentences and potential pivot languages to stop their propagation to new translations, as well as hunting for errors in any other language;
Focusing on massively multilingual tools for detecting such translation errors;
A closer integration of SMOL, BOUQuET, and other datasets into OLDI;
Integration with other massively multilingual datasets, such as CommonVoice (wouldn’t it be nice to have a part of its speech recordings translated to other languages?) or maybe tatoeba.org.
If you have any other suggestions, please don’t hesitate to comment!
We are almost certain that the WMT 2026 shared task of OLDI is going to happen, and in any case, there will be news — so please subscribe to our newsletter (or join our Discord) to keep in touch.
See you later!
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.