Very slow Response from LLM based Q/A query engine

drywall_dan

I built a Q/A query bot over a 4MB csv file I have in my local, I'm using chroma for vector DB creation and with embedding model being Instructor Large from hugging face, and LLM chat model being LlamaCPP=llama2-13b-chat , The Vector Database created was around 44MB (stored it on Local), and after vector DB creation, I used it to make the query Q/A bot but the response is too slow, it takes around 30-40 mins for each response to be generated, in...

Originally asked by Avish Wagde on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

Inference can be very slow on CPU. The biggest performance boost can be achieved by making sure that llama.cpp uses your GPU by installing the correct drivers and libraries such as CUDA in the supported versions and then compiling llama.cpp with the appropriate compiler flags, see https://github.com/ggerganov/llama.cpp . If you do not have a supported GPU installed, then a Q/A query bot that answers within seconds is not realistic. ChatGPT runs on many GPUs in parallel. 30-40 minutes still seems excessive, you can reduce the time somewhat by selecting a smaller model such as LLaMA 7b Chat instead of LLaMA 13b Chat and by using a quantized model. However this will also reduce the quality of...

Full answer by Konrad Höffner: https://stackoverflow.stackexchange.com/a/77022488 (CC BY-SA 4.0).

Need it done for your business? UphillTech.