How to run any quantized GGUF model on CPU for local inference?

fixit_fiona

In ctransformers library, I can only load around a dozen supported models. How can I run local inference on CPU (not just on GPU) from any open-source LLM quantized in the GGUF format (e.g. Llama 3, Mistral, Zephyr, i.e. ones unsupported in ctransformers )?

Originally asked by DevEnma on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

llama-cpp-python is my personal choice, because it is easy to use and it is usually one of the first to support quantized versions of new models. To install it for CPU, just run pip install llama-cpp-python . Compiling for GPU is a little more involved, so I'll refrain from posting those instructions here since you asked specifically about CPU inference. I also recommend installing huggingface_hub ( pip install huggingface_hub ) to easily download models. Once you have both llama-cpp-python and huggingface_hub installed, you can download and use a model (e.g. mixtral-8x7b-instruct-v0.1-gguf) like so: ## Imports from huggingface_hub import hf_hub_download from llama_cpp import Llama ##...

Full answer by Ryan Stewart: https://stackoverflow.stackexchange.com/a/77734862 (CC BY-SA 4.0).

Need it done for your business? UphillTech.