Enable GPU for Python programming with VS Code on Windows 10 (llama-cpp-python)

kernel_kate

I struggled alot while enabling GPU on my 32GB Windows 10 machine with 4GB Nvidia P100 GPU during Python programming. My LLMs did not use the GPU of my machine while inferencing. After spending few days on this I thought I will summarize my step by step approach which worked for me Install C++ distribution. I did it via Visual Studio 2022 Installer and installing packages under "Desktop Development with C++" and checking the option "Windows 10...

Originally asked by Umaima Tinwala on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

Install C++ distribution. I did it via Visual Studio 2022 Installer and installing packages under "Desktop Development with C++" and checking the option "Windows 10 SDK (10.0.20348.0) as shown in this image ( https://i.sstatic.net/vLDy7.png ). Install the packages. Download and Install Nvidia CUDA Toolkit ( https://developer.nvidia.com/cuda-downloads ) Ensure that CUDA_PATH variable is set in your environment variables In Visual Studio Code, set the following environment variables $env:CMAKE_ARGS="-DLLAMA_CUBLAS=on" $env:CUDACXX="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\bin\nvcc.exe" Finally run pip install llama-cpp-python --no-cache-dir --force-reinstall --upgrade

Full answer by MingJie-MSFT: https://stackoverflow.stackexchange.com/a/77830846 (CC BY-SA 4.0).

Need it done for your business? UphillTech.