M1 Chip: Running Mistral-7B with Llama.cpp Works, but Python Wrapper Causes...

reno_queen

I'm working on a project using an M1 chip to run the Mistral-7B model. I've successfully set up llama.cpp and can run the model using the following command: ./build/bin/main --color --model "./../Model/mistral-7b-instruct-v0.1.Q6_K.gguf" -t 7 -b 24 -n -1 --temp 0 -ngl 1 -p "Building a website can be done in 10 simple steps:\nStep 1:" This works well on the M1 chip. However, issues arise when I install the Python wrapper:...

Originally asked by Max Witwer on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

I've encountered the same problem. Actually, at least in my case, it happens because of RAM (I think so). I have only 8 GB. However, I am able to run llama-2-7b using some kind of a workaround, namely, running this model directly from the llama folder.

Full answer by forvev: https://stackoverflow.stackexchange.com/a/77737641 (CC BY-SA 4.0).

Need it done for your business? UphillTech.