Connect Langroid to my own LLM hosted in the cloud

reno_rita

I feel like this should be a very common pattern, but I am trying to use Langroid to build an AI agent that connects to an LLM agent I am hosting myself on azure. I cannot find anywhere in the Langroid docs that states how to point the chat agent to my cloud LLM. There is a lot of documentation around using it with a locally hosted LLM, but nothing cloud based. Is this something that Langroid allows? For context, I am hosting ollama using...

Originally asked by kpozzi90 on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

finally got it working. the config looks like below: from langroid.language_models import OpenAIGPTConfig ollama_config = OpenAIGPTConfig( chat_model="ollama/mixtral", chat_context_length=16_000, api_base="http://<your.ip>/v1" #the /v1 was necessary for my ollama llm )

Full answer by kpozzi90: https://stackoverflow.stackexchange.com/a/79660812 (CC BY-SA 4.0).

Need it done for your business? UphillTech.