How to use CodeLlama with Langchain

chef_carol

I am trying to write a simple program using codeLlama and LangChain. But it does not produce satisfactory output. And everytime we run this program it produces some different output. model used :- https://huggingface.co/TheBloke/CodeLlama-7B-Python-GGUF/blob/main/codellama-7b-python.Q4_0.gguf LangChain documentation :- https://python.langchain.com/docs/integrations/llms/llamacpp program :- from langchain.llms import LlamaCpp from...

Originally asked by Yogesh Tembe on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

not sure about your specific issue, seems like the model fails to return an answer, assuming that it is not Langchain problem for continuous/multiple queries. anyway following the doc , managed to make it work: from langchain.callbacks.manager import CallbackManager from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler from langchain_community.llms import LlamaCpp callback_manager = CallbackManager([StreamingStdOutCallbackHandler()]) llm = LlamaCpp( model_path="models\codellama-7b.Q4_K_M.gguf", n_ctx=5000, n_gpu_layers=1, n_batch=512, f16_kv=True, # MUST set to True, otherwise you will run into problem after a couple of calls callback_manager=callback_manager,...

Full answer by Oz Levi: https://stackoverflow.stackexchange.com/a/77978660 (CC BY-SA 4.0).

Need it done for your business? UphillTech.