Apple study: LLM and human MT quality annotation both unreliable

Apple researchers evaluated how well large language models agree with human annotators on two machine translation quality schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA). The study, presented at EMNLP, used a long-context test set covering 70 language pairs plus the public WMT23 and WMT25 data. The goal was to test whether LLMs can replace human judgment for MT evaluation across languages, domains, and levels of annotation detail.

The results show LLM agreement with humans exceeds human-to-human agreement for some evaluation tasks, but both vary substantially across annotation schemes, language pairs, and domains. For most settings, neither LLM nor human annotation remains reliable. The paper identifies distinct failure modes: humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants, and error span annotation.

What this means for model serving

For teams that use LLMs to score or annotate translation output, the practical takeaway is that agreement numbers from one language pair or domain do not transfer. The authors report that reliability depends heavily on the evaluation scheme, the language pair, and the domain, so a model that tracks human annotators well on high-resource pairs can degrade sharply on low-resource ones or on fine-grained error categories.

The study also points to a possible path forward: human-LLM collaborative annotation pipelines. The authors suggest such pipelines could address the reliability issues identified, since the two sides fail in different ways. Humans miss fine-grained MQM details and low-resource language nuances; LLMs miss minor errors, wrong language variants, and precise error spans. Combining both could cover complementary blind spots.

Annotation schemes matter more than model choice

The gap between MQM and ESA annotation is central to the findings. LLM performance on one scheme does not predict performance on the other, and the same holds for human annotators. For inference engineers building evaluation pipelines, this means the choice of annotation scheme is as important as the choice of model, and validation should be done per scheme, per language pair, and per domain.

The study is positioned as a reliability assessment rather than a leaderboard comparison, and it highlights potential for improvement on both the human and LLM sides through collaborative workflows.