Llama.cpp GPU Offloading Issue - Unexpected Switch to CPU

mike_plumbs

I'm reaching out to the community for some assistance with an issue I'm encountering in llama.cpp. Previously, the program was successfully utilizing the GPU for execution. However, recently, it seems to have switched to CPU execution. Observations: BLAS=1 is set, indicating the use of BLAS routines (likely for linear algebra operations). llm_load_print_meta: LF token = 13 '<0x0A>' (potential output related to loading metadata, but its specific...

Originally asked by Montassar Jaziri on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

Try eg the parameter -ngl 100 (for llama.cpp main) or --n_gpu_layers 100 (for llama-cpp-python) to offload to gpu.

Full answer by Chris A.: https://stackoverflow.stackexchange.com/a/78509754 (CC BY-SA 4.0).

Need it done for your business? UphillTech.