Recent advances in language models have prompted a growing body of work that questions prevailing evaluation practices.
However, many such critiques have already been discussed in natural language processing (NLP), a field with a long history of methodological reflection on evaluation.
We conduct a scoping review spanning four decades and develop a taxonomy of 22 recurring concerns.
For each concern, we examine how recently it was discussed in the reviewed literature and whether it receives explicit attention in institutional checklists, finding that some concerns have become less visible in literature and many lack coverage in these checklists.