How to efficiently make concurrent calls to a local Llama 3.1 model using...

reno_queen

I'm using a local Llama 3.1:7b model. I am interacting with the model using Python and the ollama-python package. I need to make over 20,000 calls to the model, but I'm currently invoking it sequentially, which is inefficient. I'm looking for ways to improve efficiency. Specifically, I'm trying to find out if there's a way to: Batch inputs to send multiple requests at once. Use threading, multiprocessing, or any form of concurrent processing to...

Originally asked by Brandon Walton on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

I recommend using asyncio to do the cocurrent stuff and use asyncio.Semaphore to limit the number of concurrency so you don't blow up your GPU, but you can surely do multithreading. You might need to dig a bit further over Python asynchrounous programming, but Ollama python package does have support and documentation over its async client ( https://github.com/ollama/ollama-python?tab=readme-ov-file#async-client ). Though, you probably have something working since it's like four months ago.

Full answer by Tim Chiu: https://stackoverflow.stackexchange.com/a/79675320 (CC BY-SA 4.0).

Need it done for your business? UphillTech.