Running Local LLMs in Production and handling multiple requests

tux_tina

I am trying to run a RAG with Gemma LLM locally it is running fine but the idea is I can't handle more than one request at a time. Is there a way to handle concurrent requests with utilizing resources efficiently? Can cont batching help and how to write it in the code? Used in the code: llamaindex Llamacpp pinecone flask

Originally asked by khalidwalamri on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

Try using vLLM instead of Llama.cpp (which is not thread-safe ). vLLM is able to handle in parallel concurrent (overlapping) requests and keeps up with the most recent models (even ones released in the previous week, like Llama 3 and Mixtral 8x22B currently are). ____________ Note that you may have to build the image yourself to pass vulnerability scans (the official ones currently use older pinned system libraries, presumably to improve compatibility and build reproducibility).

Full answer by mirekphd: https://stackoverflow.stackexchange.com/a/78358952 (CC BY-SA 4.0).

Need it done for your business? UphillTech.