Running LLM from local disk

plumb_pete

I am trying to load LLM from the local disk of my laptop which is not working. when i try to load with the following approach its working as expected and i am getting response to my query. def load_llm(): # Load the locally downloaded model here llm = CTransformers( model = "TheBloke/Llama-2-7B-Chat-GGML", model_type="llama", config={'max_new_tokens': 3000, 'temperature': 0.01, 'context_length': 3000} ) return llm If i change the above method as...

Originally asked by mohan on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

I don't usually use CTransformers but I know the latest format is GGUF and GGML was discontinued, so if you are in the lastest versions of CTransformers is likely it will not run GGML anymore. Don't know if it helps you but if you don't mind changing your approach I always used llama-cpp-python bindings and it always worked for me to run models locally. To do this you download the GGUF version of the model you want for TheBloke. Then run pip install llama-cpp-python (is possible the will ask for pytorch to be already installed). After is installed you can run any GGUF model using: from llama_cpp import Llama llm = Llama(model_path="./models/7B/llama-model.gguf", ntcx=3000) output = llm("Q:...

Full answer by Danielle França: https://stackoverflow.stackexchange.com/a/78210909 (CC BY-SA 4.0).

Need it done for your business? UphillTech.