The automatic evaluation of instruction following typically involves using large language models (LLMs) to assess response quality. However, there is a lack of comprehensive evaluation of these LLM-based evaluators across two dimensions: the base LLMs and the evaluation protocols. Therefore, we present a thorough meta-evaluation of instruction following, including 25 base LLMs and 15 recently proposed evaluation protocols, on 4 human-annotated datasets, assessing the evaluation accuracy of the LLM-evaluators. Our evaluation allows us to identify the best-performing base LLMs and evaluation protocols with a high degree of robustness. Moreover, our evaluation reveals key findings: (1) Base LLM performance ranking remains largely consistent across evaluation protocols, with less capable LLMs showing greater improvement from protocol enhancements; (2) Robust evaluation of evaluation protocols requires many base LLMs with varying capability levels, as protocol effectiveness depends on the base LLM used; (3) Evaluation results on different datasets are not always consistent, so a rigorous evaluation requires multiple datasets with distinctive features. We release our meta-evaluation suite \ours,\footnote\ours stands for \textbf{Re-\textbfevaluation of \textbfInstruction-\textbfFollowing \textbfEvaluation: \urlhttps://github.com/yale-nlp/ReIFE.} which provides the codebase and evaluation result collection for over 500 LLM-evaluators, laying groundwork for future research in instruction-following evaluation.

@inproceedings{Yixin-naacl-25, abstract = {The automatic evaluation of instruction following typically involves using large language models (LLMs) to assess response quality. However, there is a lack of comprehensive evaluation of these LLM-based evaluators across two dimensions: the base LLMs and the evaluation protocols. Therefore, we present a thorough meta-evaluation of instruction following, including 25 base LLMs and 15 recently proposed evaluation protocols, on 4 human-annotated datasets, assessing the evaluation accuracy of the LLM-evaluators. Our evaluation allows us to identify the best-performing base LLMs and evaluation protocols with a high degree of robustness. Moreover, our evaluation reveals key findings: (1) Base LLM performance ranking remains largely consistent across evaluation protocols, with less capable LLMs showing greater improvement from protocol enhancements; (2) Robust evaluation of evaluation protocols requires many base LLMs with varying capability levels, as protocol effectiveness depends on the base LLM used; (3) Evaluation results on different datasets are not always consistent, so a rigorous evaluation requires multiple datasets with distinctive features. We release our meta-evaluation suite \ours,\footnote{\ours stands for \textbf{R}e-\textbf{e}valuation of \textbf{I}nstruction-\textbf{F}ollowing \textbf{E}valuation: \url{https://github.com/yale-nlp/ReIFE}.} which provides the codebase and evaluation result collection for over 500 LLM-evaluators, laying groundwork for future research in instruction-following evaluation.}, address = {New Mexico, USA}, author = {Yixin Liu and Kejian Shi and Alexander Fabbri and Yilun Zhao and Peifeng Wang and Chien-Sheng Wu and Shafiq Joty and Arman Cohan}, booktitle = {2025 Annual Conference of the North American Chapter of the Association for Computational Linguistics}, publisher = {ACL}, series = {NAACL-25}, title = {ReIFE: Re-evaluating Instruction-Following Evaluation}, url = {https://arxiv.org/abs/2410.07069}, year = {2025} }