llama-cpp-python on GPU: Delay between prompt submission and first token...

kernel_kate

I've been building a RAG pipeline using the llama-cpp-python OpenAI compatible server functionality and have been working my way up from running on just a laptop to running this on a dedicated workstation VM with access to an Nvidia A100. After the most recent transition to a machine with access to this A100 i was expecting (naively?) this RAG pipeline to be blazing fast, but I've been surprised to find that this is not currently the case. What...

Originally asked by jhthompson12 on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Good thread on this — worth a read before you pay anyone to build it.

Discussion: https://stackoverflow.com/questions/77718803/llama-cpp-python-on-gpu-delay-between-prompt-submission-and-first-token-generat.

Want it set up for you? UphillTech.