Using LLM models downloaded from huggingface with langgraph

chef_carol

I downloaded the weights of a llama model from huggingface. It works for simple tasks, but I don't know how to use it with langgraph to create agents or how to bind tools. Here is how I downloaded it: model_id = "meta-llama/Llama-3.2-3B-Instruct" pipeline = transformers.pipeline( "text-generation", model=model_id, token="hf_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", model_kwargs={"dtype": torch.bfloat16}, device_map="auto", ) The code I try to use...

Originally asked by jottbe on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

There are several options to use the model for text generation and tool calling locally. I would recommend you to deploy it with vllm and call the API of it. At first you need to install the following packages: pip install langchain_openai vllm Afterwards, you deploy your model locally in a terminal via: vllm serve Qwen/Qwen3-4B \ --enable-auto-tool-choice \ --tool-call-parser hermes \ --reasoning-parser deepseek_r1 \ --gpu-memory-utilization 0.9 \ --max_model_len 4096 Please note that I use Qwen/Qwen3-4B instead of meta-llama/Llama-3.2-3B-Instruct . The value for tool-call-parser probably needs to be adjusted. reasoning-parser should be skipped completely because the llama model is not a...

Full answer by cronoik: https://stackoverflow.stackexchange.com/a/79911997 (CC BY-SA 4.0).

Need it done for your business? UphillTech.