There's no reason for software to be slow anymore

though, as we discussed in that post as well as before, the benchmarking and experimental design skills of SOTA models aren't good enough to do this in the general case without a human (or a skill) setting up the benchmarking environment for the agent. [return] we can see from our old benchmarks that, even with time to run the compiler, there are a lot of cases where the native code compiled version is slower than the Rust regex crate. If we look at why this is, these tend to be more complex queries where the Rust regex crate has some algorithmic optimization and the FRE native code compiler is falling back to something naive (the agent that created FRE spent much less time on the native code compiler than it did on the "normal" regex engine). [return] I have no doubt that a hand-written AI by someone who has real AI expertise, e.g., by someone who's written one of the top Go and chess engines in the world, could beat my AI on the strength of the "AI" side of things being better than what you get when someone who knows nothing about AI (me) creates an AI, but if the levels of expertise are remotely similar, the LLM-written version is going to dominate for any given amount of time spent. [return] it's arguably unfair to compare the result of an agent picking up where he left off, since his work is a starting point which might let an agent do much better than it would do on its own, so I tried giving the fresh task to an agent and it got a very similar score to what he got when an agent re-used his work (and a quick check by another agent didn't find evidence of cheating). [return] The performance probably would've been better if I had an agent just modify a ripgrep fork directly, but I was curious if this could also solve the FRE overfitting problem with respect to my queries. [return] a while back, I reduced the size of page in our signup flow from 50 MB to 5 MB and a revenue A/B test seemed to indicate that this increased revenue by about 0.5%. In general, I'm a huge fan of doing the simple and easy wins first, such as this, and there are probably a lot of higher ROI wins than we'd get out of building custom compilers or doing other highly specialized technical work here. [return]