The Benchmarkpocalypse

funnily enough, I have some faith in some of the projects that people are the most skeptical of, e.g., every time I see pgrust somewhere, there are a lot of skeptical comments. But, without having looked into the details of what he's optimizing, I would trust that they're not doing something shady with their benchmarks because Michael Malis started the project (and is still involved). I used to look at most benchmark claims that cross my radar in some detail, but there are so many of these now that I don't really have time to do that and generally assume that claims are false in spirit (even if technically correct) unless there's some reason to believe otherwise. Of course this will sometimes be wrong (e.g., if I didn't know Michael Malis, I would've guessed that pgrust is just another low-quality "have an LLM re-write this thing" project), but LLMs are such an incredible machine for DoSing human attention that I don't know what else I would do about it (I've tried having LLMs analyze performance claims and, while the result is correlated with what I'd think if I looked at something myself, the result is often quite wrong). Someone can spend seconds (or, if using the right framework, actually none of their time) generating something that takes people minutes to hours to understand. This is a topic for another post, but from talking to people about their experiences with this in the workplace, companies with poor norms for this kind of thing are really struggling with productivity today. [return] funnily enough, I have some faith in some of the projects that people are the most skeptical of, e.g., every time I see pgrust somewhere, there are a lot of skeptical comments. But, without having looked into the details of what he's optimizing, I would trust that they're not doing something shady with their benchmarks because Michael Malis started the project (and is still involved). I used to look at most benchmark claims that cross my radar in some detail, but there are so many of these now that I don't really have time to do that and generally assume that claims are false in spirit (even if technically correct) unless there's some reason to believe otherwise. Of course this will sometimes be wrong (e.g., if I didn't know Michael Malis, I would've guessed that pgrust is just another low-quality "have an LLM re-write this thing" project), but LLMs are such an incredible machine for DoSing human attention that I don't know what else I would do about it (I've tried having LLMs analyze performance claims and, while the result is correlated with what I'd think if I looked at something myself, the result is often quite wrong). Someone can spend seconds (or, if using the right framework, actually none of their time) generating something that takes people minutes to hours to understand. This is a topic for another post, but from talking to people about their experiences with this in the workplace, companies with poor norms for this kind of thing are really struggling with productivity today. This is referring to the geomean of all rebar benchmarks. This is probably not the right metric to use, in that this implicitly says that each benchmark is of the same importance, which probably isn't the case. Unlike something like SPEC CPU, the rebar benchmarks don't position themselves as something where you get a meaningful summary metric that tries to represent overall performance (the repo actually notes that it's "a biased barometer for gauging the relative speed of some regex engines on a curated set of tasks"). But, to get a number that is a useful summary metric, you'd have to know a lot about how people use regexes in practice, and I know approximately zero about that. For all I know, you should have two different numbers (like SPECfp and SPECint for SPEC CPU) or ten or a hundred because there are all sorts of different ways people apply regexes. [return] The first few regex benchmarks I looked at had already been incorporated into rebar, so they wouldn't work as a holdout. And, as previously discussed, current SOTA LLMs aren't very good at benchmarking, so I wouldn't be able to trust the LLM to come up with a holdout benchmark unless I knew enough about regex performance to judge the quality of the benchmark suite. Since I know approximately zero about string matching algorithms or regex performance, that was also off the table. It turns out that BurntSushi also maintains ripgrep and the benchmarks for ripgrep, which are big enough benchmarks that they didn't get bundled into rebar, so I tried using those benchmarks as a holdout. [return] The first few regex benchmarks I looked at had already been incorporated into rebar, so they wouldn't work as a holdout. And, as previously discussed, current SOTA LLMs aren't very good at benchmarking, so I wouldn't be able to trust the LLM to come up with a holdout benchmark unless I knew enough about regex performance to judge the quality of the benchmark suite. Since I know approximately zero about string matching algorithms or regex performance, that was also off the table. It turns out that BurntSushi also maintains ripgrep and the benchmarks for ripgrep, which are big enough benchmarks that they didn't get bundled into rebar, so I tried using those benchmarks as a holdout.