Bilingual Data Mixing Beats Hyperparameter Tuning in Data-Constrained LM Training

Apple researchers compared two remedies for data-constrained language model pre-training: aggressive hyperparameter tuning, such as high weight decay, and mixing in data from a high-resource auxiliary language. The study used Arabic as the low-resource target and English as the auxiliary, running approximately 1000 pre-training runs across four model scales from 150M to 1.43B parameters.

Mixing wins, and the gap grows with scale

The paper reports that mixing yields larger improvements than hyperparameter tuning on both validation loss and downstream task accuracy, with the gap growing as model size increases. The authors quantify the benefit: mixing boosts performance by an amount equivalent to 2-3x the unique target data on validation loss and 2-13x on downstream task accuracy, with the gain scaling steeply with model size.

For inference engineers serving low-resource language models, this suggests that training data composition matters more than regularization settings. A model trained with auxiliary data mixing can reach accuracy levels that would otherwise require several times more target-language data, which is often unavailable in practice.

Validation loss underestimates mixing's value

The third finding is that target-language validation loss systematically underestimates mixing's benefit. The authors explain that mixing regularizes by diversifying the training signal and contributes knowledge the repeated target corpus cannot supply. Validation loss captures only the first effect, not the knowledge contribution.

This has practical implications for model evaluation. Teams that rely on validation loss to decide between training strategies may underweight mixing and overweight hyperparameter tuning, potentially choosing a weaker configuration.

Practical recommendations

The authors recommend mixing in a high-resource language, prioritizing the mixing ratio over hyperparameter tuning, and transferring hyperparameters from a small proxy model via μP. The μP recommendation connects to prior Apple work on hyperparameter transfer across model sizes, which enables searching optimal global hyperparameters at small scale and transferring them to larger models.

For teams building models in data-constrained languages, the actionable takeaway is to treat auxiliary data mixing as a primary lever rather than a secondary tweak. The mixing ratio becomes a key configuration parameter, and small proxy models can be used to find good settings before scaling up.

The paper is published under the title "Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings" and is listed under Apple's speech and natural language processing research area.