Detecting GPU availability in llama-cpp-python

reno_queen

Question How can I programmatically check if llama-cpp-python is installed with support for a CUDA-capable GPU? Context In my program, I am trying to warn the developers when they fail to configure their system in a way that allows the llama-cpp-python LLMs to leverage GPU acceleration. For example, they may have installed the library using pip install llama-cpp-python without setting appropriate environment variables for CUDA acceleration, or...

Originally asked by Programmer.zip on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

Examining the source code of the library, I found a new approach for checking GPU availability: from llama_cpp.llama_cpp import _load_shared_library def is_gpu_available_v3() -> bool: lib = _load_shared_library('llama') return bool(lib.llama_supports_gpu_offload()) This works with llama_cpp_python==0.2.64 .

Full answer by Programmer.zip: https://stackoverflow.stackexchange.com/a/78450953 (CC BY-SA 4.0).

Need it done for your business? UphillTech.