Great question — answering as the UphillTech team:
This error suggests that your system is possibly out of memory. I tried to use a 30b model once on my m1 pro macbook with 16GB RAM, llama cpp raised the same error. Even though you're able to load the model, while inference, if it recieves two or more queries, it is unable to handle those requests because there is a memory limit error. Another thing could be that a single instance of an LLM cannot handle multiple requests. So create a queue for your requests and process them one by one.
Full answer by Raj Singh Parihar: https://stackoverflow.stackexchange.com/a/78614684 (CC BY-SA 4.0).
Need it done for your business? UphillTech.
