publications

2024

1. Multitask Multilingual Model Adaptation with Featurized Low-Rank Mixtures

   Chu-Cheng Lin, Xinyi Wang, Jonathan H Clark, and 4 more authors

   arXiv preprint arXiv:2402.17934, 2024

2023

1. Gemini: a family of highly capable multimodal models

   Gemini Team, Rohan Anil, Sebastian Borgeaud, and 8 more authors

   arXiv preprint arXiv:2312.11805, 2023
2. Sira: Sparse mixture of low rank adaptation

   Yun Zhu, Nevan Wichers, Chu-Cheng Lin, and 8 more authors

   arXiv preprint arXiv:2311.09179, 2023
3. Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization

   Alexandra Chronopoulou, Jonas Pfeiffer, Joshua Maynez, and 3 more authors

   arXiv preprint arXiv:2311.09344, 2023
4. mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinations

   Jonas Pfeiffer, Francesco Piccinno, Massimo Nicosia, and 3 more authors

   In EMNLP Findings , 2023

   Abs

   Multilingual sequence-to-sequence models perform poorly with increased language coverage and fail to consistently generate text in the correct target language in few-shot settings. To address these challenges, we propose mmT5, a modular multilingual sequence-to-sequence model. mmT5 utilizes language-specific modules during pre-training, which disentangle language-specific information from language-agnostic information. We identify representation drift during fine-tuning as a key limitation of modular generative models and develop strategies that enable effective zero-shot transfer. Our model outperforms mT5 at the same parameter sizes by a large margin on representative natural language understanding and generation tasks in 40+ languages. Compared to mT5, mmT5 raises the rate of generating text in the correct language under zero-shot settings from 7% to 99%, thereby greatly alleviating the source language hallucination problem.

5. Evaluating and Modeling Attribution for Cross-Lingual Question Answering

   Benjamin Muller, John Wieting, Jonathan Clark, and 6 more authors

   In EMNLP , 2023

   Abs

   Trustworthy answer content is abundant in many high-resource languages and is instantly accessible through question answering systems — yet this content can be hard to access for those that do not speak these languages. The leap forward in cross-lingual modeling quality offered by generative language models offers much promise, yet their raw generations often fall short in factuality. To improve trustworthiness in these systems, a promising direction is to attribute the answer to a retrieved source, possibly in a content-rich language different from the query. Our work is the first to study attribution for cross-lingual question answering. First, we collect data in 5 languages to assess the attribution level of a state-of-the-art cross-lingual QA system. To our surprise, we find that a substantial portion of the answers is not attributable to any retrieved passages (up to 50% of answers exactly matching a gold reference) despite the system being able to attend directly to the retrieved text. Second, to address this poor attribution level, we experiment with a wide range of attribution detection techniques. We find that Natural Language Inference models and PaLM 2 fine-tuned on a very small amount of attribution data can accurately detect attribution. With these models, we improve the attribution level of a cross-lingual QA system. Overall, we show that current academic generative cross-lingual QA systems have substantial shortcomings in attribution and we build tooling to mitigate these issues.

6. XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages

   Sebastian Ruder, Jonathan Clark, Alexander Gutkin, and 24 more authors

   In EMNLP Findings 2023 , 2023

   Abs

   Data scarcity is a crucial issue for the development of highly multilingual NLP systems. Yet for many under-represented languages (ULs) — languages for which NLP research is particularly far behind in meeting user needs — it is feasible to annotate small amounts of data. Motivated by this, we propose XTREME-UP, a benchmark defined by: its focus on the scarce-data scenario rather than zero-shot; its focus on user-centric tasks — tasks with broad adoption by speakers of high-resource languages; and its focus on under-represented languages where this scarce-data scenario tends to be most realistic. XTREME-UP evaluates the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies including ASR, OCR, MT, and information access tasks that are of general utility. We create new datasets for OCR, autocomplete, semantic parsing, and transliteration, and build on and refine existing datasets for other tasks. XTREME-UP provides methodology for evaluating many modeling scenarios including text only, multi-modal (vision, audio, and text), supervised parameter tuning, and in-context learning. We evaluate commonly used models on the benchmark. We release all code and scripts to train and evaluate models.