No GPU support while running llama-cpp-python inside a docker container

bake_betty

I'm trying to run llama index with llama cpp by following the installation docs but inside a docker container. Following this repo for installation of llama_cpp_python==0.2.6. DOCKERFILE # Use the official Python image for Python 3.11 FROM python:3.11 # Set the working directory in the container WORKDIR /app # Copy the current directory contents into the container at /app COPY . /app # ARG FORCE_CMAKE=1 # ARG CMAKE_ARGS="-DLLAMA_CUBLAS=on" #...

Originally asked by Pratyush on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

On Windows I use this image: FROM nvidia/cuda:11.7.1-devel-ubuntu22.04 And this is how I set the necessary vars before install. ENV CMAKE_ARGS="-DLLAMA_CUBLAS=ON" RUN pip install llama-cpp-python Works for me. Again, on Windows with Docker Desktop!

Full answer by Amin: https://stackoverflow.stackexchange.com/a/77540805 (CC BY-SA 4.0).

Need it done for your business? UphillTech.