High-dimensional Bayesian optimisation (HDBO) is an increasingly crowded field with new methods proposed at every major conference, making it ever harder to answer a deceptively simple question: "Which method is currently state-of-the-art?"
- In this paper, we question the efficacy of benchmarking practices in the field, and find them to be systematically ineffective: multiple published works fail to benchmark against the strongest available baselines, or even include incorrect implementations of key methods.
- Without a shared reference implementation, such errors are difficult to detect during peer review.
- We therefore introduce BO-Arena, an actively maintained and extensible package providing canonical implementations of state-of-the-art algorithms, which allows researchers to develop new methods and benchmark them rigorously.
- Using BO-Arena, we find that what currently drives performance in HDBO is even simpler than previously thought.
- We use these insights to propose a new algorithm which is competitive with the state of the art on problems with many observations, without resorting to the complex GP approximations used in recent methods.
- Finally, we question whether there is currently sufficient evidence that any more complex method outperforms a simple Gaussian process baseline for HDBO.