It's long been assumed that for most programmers writing hardware-optimized code is too hard. Few know, or want to know, how to exploit memory hierarchies, NUMA nodes, vector instructions, stop registers from spilling, or exploit the full memory bandwidth of the CPU. This is tragic. In the age of LLMs and tooling, we should expect more from our software and our hardware.
The point I'm making is general, but for now I'd like to focus on the special case of specializations of std::sort.
The classic argument for std::sort is to just call the stdlib. The performance you get from that is decent. You can get, say, 80% of the performance, for basically no work. But suppose now that it needs to be a stable sort (like std::stable_sort), but you only need the biggest n elements sorted (like std::partial_sort or std::nth_element). And let's say the values aren't all stored contiguously in an array, they're in an array of arrays. We've left the realm where the stdlib gets you good performance. Luckily the stdlib is still not useless. You can write a random access iterator over the input arrays, and call sort without problems. But if you want speed, you've got to do it yourself. Good luck getting your fancy iterator to vectorize and unroll.
LLMs should be able to help you with this. Why not just generate an implementation that's fast? The problem is that LLMs just reflect the look of the code the average programmer would write. They copy a lot of surface level aesthetics, not the subtle unspoken reasons why good code is fast and correct. Or they do capture that, but not so much as the aesthetics. Realistically speaking, there's no LLM that can help you with the problem I described. I know because I tried them all. They'll hit one or two of the requirements, then get stuck. And that's really disappointing, because you'd really think they'd be good at this. It's a problem that gets significantly and predictably better with scale, but there's still not a lot of code out there that combines specific edge cases in this way.
So, you have to write your own implementation. But unfortunately, these stdlib functions are black boxes. Since the compiler won't be able to help very much, you're going to be reaching for vector intrinsics. Yet another relatively low-data code subset. Yet another reason why I wish we didn't spend so much time scaring new programmers away from understanding how computers actually work. I don't think the existing data distribution off github is actually good for training on, if the goal is to create models useful for optimizing performance.
There's the potential for reinforcement learning to help with this. But it's going to turn into a lot of very manual work. I think that if you want LLMs to get better at this sort of thing, you have to craft difficult synthetic data. It's good, though, that we now don't have to rely on filtering for high quality data and finetuning. We can do RL instead, and acquire learning signal from synthetic data. It remains an open question on how to do so in a way that feels general, tasteful, and preference-aligned. But it's coming, I think.