AssertionError when using llama-cpp-python in Google Colab

grill_gus

I'm trying to use llama-cpp-python (a Python wrapper around llama.cpp) to do inference using the Llama LLM in Google Colab. My code looks like this: !pip install llama-cpp-python from llama_cpp import ChatCompletionMessage, Llama model = Llama( "/content/drive/MyDrive/<weights-file>.bin", ) However, when running it, I get this error: AssertionError Traceback (most recent call last) <ipython-input-13-652eb650093d> in <cell line: 9>() 7 } 8 ---->...

Originally asked by Utrax on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

From the model path - model_path="/content/drive/MyDrive/careo/Wizard-Vicuna-13B-Uncensored.ggmlv3.q4_1.bin" , I can see you are using ggmlv3 model format, as per the new commit for llama-cpp-python repo, the new model format has been changed from ggmlv3 to gguf. The author also mentioned that ggmlv3 weights are still gonna work for versions before 0.1.79(new version) so you either mention the version while installing the package pip install llama-cpp-python==0.1.78 or change model format for new version gguf - refer . If CodeLlama model weights are useful for you, then there are so many model weights published in hugging face like TheBloke/CodeLlama-13B-GGUF

Full answer by Sriram Govardhanam: https://stackoverflow.stackexchange.com/a/76987348 (CC BY-SA 4.0).

Need it done for your business? UphillTech.