Oh no 128GB is not enaught for 7B parameters!

TL/DR If you’re working with large language models (LLMs) on systems like the DGX Spark, and encountering “out of memory” errors despite having seemingly ample RAM (e.g., 128GB for a 7B parameter model), the culprit might be your operating system’s caching mechanisms. The solution is often as simple as dropping system caches. DGX Spark uses UMA (Unified Memory Architecture): CPU and GPU share the…