GenEval · GenEval: Linking Generation and Evaluation for Reliable NLG Assessment

Large Language Models (LLMs) are increasingly leveraged as evaluators of machine-generated text, a paradigm known as ""LLMs-as-judges."" While this approach offers flexibility and typically strong performance, its reliability remains inconsistent and poorly understood. Strong generative performance does not guarantee reliable evaluation, and the mechanisms linking these two capabilities remain opaque. Without systematic validation, current evaluation practices risk being blind to misleading and factually incorrect content and misrepresenting system capabilities.GenEval addresses this challenge by investigating the fundamental relationship between generation and evaluation in LLMs, developing novel representation-based metrics, and predicting LLMs' evaluation reliability across tasks and models. We will analyze LLMs at a mechanistic level, identifying circuits and representations that underlie generative and evaluative behaviors. This understanding will enable the design of new evaluation metrics that directly exploit LLMs' internal representations, providing interpretable, efficient, and robust alternatives to existing approaches. Finally, GenEval will develop predictive tools to estimate when an LLM is likely to be a reliable evaluator even in the absence of human judgment data, supporting informed model selection and human-in-the-loop evaluation.By integrating mechanistic insights with practical evaluation methods, GenEval will deliver both theoretical advances and applied tools.This action will be possible thanks to the integration of the scientific expertise of Prof. Horacio Saggion, an internationally recognized expert in Natural Language Generation, and that of the researcher, who has a strong background in evaluation, Natural Language Processing, and Machine Learning. The action will develop impacting technology, and provide the researcher with the necessary training to become independent and strengthen her academic profile.""