Prefetching Weights in llama.cpp

This post about about a PR to llama.cpp which adds support to prefetch weights, overlapping compute for the current layer with the weight loading for the next layer. This is useful for memory-constrained environments where the model weights don't fit into VRAM. This is the basic setup: Timeline ---------------------------------------> [Compute Layer 1] [Compute Layer 2] .... [Prefetch Weights-1]…