Inconsistent completion for identical prompts and params with llama.cpp...

fixit_fiona

I've been comparing various langchain compatible llama2 runtimes, using langchain llm chain. Having the following parameter overrides: # llama.cpp: model_path="../llama.cpp/models/generated/codellama-instruct-7b.ggufv3.Q5_K_M.bin", n_ctx = 2048, max_tokens = 2048, temperature = 0.85, top_k = 40, top_p = 0.95, repeat_penalty = 1.1, seed = 112358, # ctransformer: model="../llama.cpp/models/generated/codellama-instruct-7b.ggufv3.Q5_K_M.bin",...

Originally asked by JayabalanAaron on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

Fixing the seeds in both frameworks should be sufficient to give reproducible results regardless of other inference parameters, but I noticed another problem with this experiment: these temperature and top_k settings are not really useful for the task of code generation, in fact such wide-ranging distribution should be probably avoided even if the most diverse and creative outputs were expected. For instance, on OpenAI forums you can find advice that for code generation temperature should be set at 0.2 and top_p at 0.1 (see this post ). The highest recommended settings (for Creative Writing) are 0.7 for temperature and 0.8 for top_p . See if making these changes makes any difference to your...

Full answer by mirekphd: https://stackoverflow.stackexchange.com/a/78336345 (CC BY-SA 4.0).

Need it done for your business? UphillTech.