llama_index Ollama misloading model issue

taco_truck_tom

I'm working with LlamaIndex and trying to run LLM in local using Ollama. I pulled phi3:mini model successfully and tested it in terminal. but if I load it with llama_index.llms.ollama, it keeps following erros "ollama._types.ResponseError: model requires more system memory (45.3 GiB) than is available (16.8 GiB) (status code: 500)" meaning trying to load phi3 rather than phi3:mini. anyone knows why? I appreciate your reply. from...

Originally asked by Kemp on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

This seems like a LlamaIndex–Ollama integration bug, that's why you don't get the error when you run it directly from the terminal using ollama command. But, when using the Ollama from the llamaIndex integration, this error occurs perhaps because LlamaIndex tries loading the base "phi3" model during the metadata call (even though phi3:mini is mentioned). And this issue occurs for other models as well (for example, llama3.2) A simple fix is to explicitly add a context_window and you'll no longer receieve the ResponseError. from llama_index.llms.ollama import Ollama llm = Ollama( model = "phi3:mini", temperature = 0.7, max_tokens = 100, context_window=8000, ) response = llm.complete("Hello,...

Full answer by Ajeet Verma: https://stackoverflow.stackexchange.com/a/79839411 (CC BY-SA 4.0).

Need it done for your business? UphillTech.