llama-cpp-python not using NVIDIA GPU CUDA

penguin_pam

I have been playing around with oobabooga text-generation-webui on my Ubuntu 20.04 with my NVIDIA GTX 1060 6GB for some weeks without problems. I have been using llama2-chat models sharing memory between my RAM and NVIDIA VRAM. I installed without much problems following the instructions on its repository. So what I want now is to use the model loader llama-cpp with its package llama-cpp-python bindings to play around with it by myself. So using...

Originally asked by imbr on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

After searching around and suffering quite for 3 weeks I found out this issue on its repository. The llama-cpp-python needs to known where is the libllama.so shared library. So exporting it before running my python interpreter, jupyter notebook etc. did the trick. For using the miniconda3 installation used by oobabooga text-generation-webui I exported it like bellow: export LLAMA_CPP_LIB=/yourminicondapath/miniconda3/lib/python3.10/site-packages/llama_cpp_cuda/libllama.so Voilà!!!! On importing from llama_cpp import Llama I get ggml_init_cublas: found 1 CUDA devices: Device 0: NVIDIA GeForce GTX 1060, compute capability 6.1 And on llm =...

Full answer by imbr: https://stackoverflow.stackexchange.com/a/76963364 (CC BY-SA 4.0).

Need it done for your business? UphillTech.