Meta-evaluation of text generation metrics and LLM-judges is primarily conducted using metric-human correlations.
However, prior work has shown that correlation is not an appropriate estimate of a metric's evaluation capability and can be unstable across evaluation settings, such as different datasets or languages.
Proposed Regression-Based Meta-Evaluation
Therefore, we propose a regression-based meta-evaluation of metrics that learns on calibrated metric outputs to predict human ratings.
We ground this measurement relative to a trivial constant-mean baseline and human agreements to provide a comparable measure of metric capability across evaluation settings.
Developing a Large-Scale Benchmark
To test our meta-evaluation framework, we developed a large-scale multi-lingual style-personalized text generation (SPTG) benchmark.
Collecting multiple annotations across 10 topologically diverse languages and four evaluation dimensions, each with varying degrees of subjectivity.
Meta-Evaluation of 32 Metrics
We meta-evaluated 32 metrics from three style evaluation paradigms (string-based, embedding-based, and LLM-judges) and their ensembles over these 40 evaluation settings.
Our analysis revealed that even simple string- and embedding-based measurements can match strong LLM-judges when calibrated to human ratings.
Identifying Characteristics of Human Annotations
We also identified characteristics of human annotations that determine metric performance.
Conditions for Metric Ensemble Performance
as well as conditions under which a metric ensemble outperforms its individual constituents.