Ollama model keep in memory and prevent unloading between requests (keep_alive?)

spark_sara

No matter what I do from the terminal or in code, the agent requests to Ollama models take 15–25 seconds each time on my local M2 MacBook Pro. I am pretty sure this is not a hardware issue because the model is lightning fast when used from the terminal. I am using Python's module to load gemma3.4b via Ollama. Tried different models – similar result. Tried calling via http – not much improvement. So, my conclusion is that the model keeps...

Originally asked by Lex Podgorny on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

I'm not sure about the implementation of the Ollama Python Library because I use OpenAI SDK with Ollama, but in my project ( Open-LLM-VTuber ) I have done something similar by sending some http requests to keep the model in memory forever when my server is still alive. This keeps the model in memory forever. import requests requests.post( base_url.replace("/v1", "") + "/api/chat", json={ "model": model, "keep_alive": -1, # -1 means forever. Other number means seconds before unload }, ) And this tells Ollama to unload the model. requests.post( self.base_url.replace("/v1", "") + "/api/chat", json={ "model": self.model, "keep_alive": 0, }, ) Here's the complete implementation of it in my...

Full answer by Tim Chiu: https://stackoverflow.stackexchange.com/a/79675315 (CC BY-SA 4.0).

Need it done for your business? UphillTech.