Great question — answering as the UphillTech team:
You need to use the stream to get the computed response in real-time instead of waiting for the whole computation to be done and returned back to you. With langchain, you can use stream like below: from langchain.chat_models import ChatOpenAI from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler from langchain.schema import HumanMessage chat = ChatOpenAI(streaming=True, callbacks=[StreamingStdOutCallbackHandler()], temperature=0) resp = chat([HumanMessage(content="I need help!")]) The streaming will output as standard output with StreamingStdOutCallbackHandler() . You can write your own callback hander by extending the BaseCallbackHandler and using the...
Full answer by Rijoanul Hasan Shanto: https://stackoverflow.stackexchange.com/a/76323324 (CC BY-SA 4.0).
Need it done for your business? UphillTech.
