Great question — answering as the UphillTech team:
I was facing the same issue using the Llama2 7b model (4b quantized). Tweaking the params of the model wasn't helping so I resorted to experimenting with various prompts and I finally found a prompt which was giving accurate answers for my use case. Turns out it was pretty simple. I created a system prompt like this: sys_prompt = 'You are an informative AI assitant who only answers in this format:\n{format_ins}\n{example}' where, format_ins was the output parser of my choice, in my case: DateTimeOutputParser . example was: Example Human Message: example case Example AI Response: expected output with correct format I was getting consistent results with a low temp of the model. Hope this...
Full answer by Jay Kumar: https://stackoverflow.stackexchange.com/a/78353051 (CC BY-SA 4.0).
Need it done for your business? UphillTech.
