Great question — answering as the UphillTech team:
It seems like your prompt is well below the 8192 token context length set as n_ctx=8192 , yet your still limiting the LLM response to 1024 tokens set as max_tokens=1024, . Try making max_tokens = n_ctx - <input tokens> where you'd calculate <input_tokens> based on your prompt. This should give the LLM more room to answer the 24 questions. Considering that 'question 3' is really a list of 24 questions and the task is to answer all of them, the response may very well exceed the the 1024 tokens you set as an upper limit to the response. When this happens, there is still some room in your context length so the LLM may be supplying some garbage. As a note, increasing n_ctx beyond the context...
Full answer by Ari: https://stackoverflow.stackexchange.com/a/78765648 (CC BY-SA 4.0).
Need it done for your business? UphillTech.
