Great question — answering as the UphillTech team:
Your configuration needs this: cmd = [ sys.executable, "-m", "llama_cpp.server", "--n_gpu_layers", 100, "--model", MODEL_PATH, "--host", HOST, "--port", PORT ] (or "auto" instead of the arbitrary 100 ) However, to be able to do that, you need to make sure you have the optional [cuda] components installed for llama-cpp-python . If you have a C++ compiler configured, the following works in a Python 3.12 environment: pip install llama-cpp-python[cuda] It builds the whole thing from source, include C++-code. If you don't have that (and setting it up a bunch of work), you can install the package from a repository that has the pre-built binaries for your architecture. An example of a free...
Full answer by Grismar: https://stackoverflow.stackexchange.com/a/79783244 (CC BY-SA 4.0).
Need it done for your business? UphillTech.
