How to stop Ollama model streaming

vegan_vicky

So I have this class that streams the response form a model: from langchain_community.llms.ollama import Ollama from app.config import ( LLM_MODEL_NAME, MAX_LLM_INPUT_LENGTH, MAX_LLM_INPUT_SENTENCES, LLAMA_BASE_URL, ) from app.utils.text_processing import TextProcessing from app.utils.persona_mapper import PersonaPromptMapper from app.enums import Personas class LLMAssistant: def __init__( self, persona: Personas = Personas.PROFESSOR,...

Originally asked by KZiovas on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

class LLMAssistant: def __init__(self, persona: Personas = Personas.PROFESSOR, model_name: str = LLM_MODEL_NAME): self.persona = persona self.persona_prompt = PersonaPromptMapper.get_persona_prompt(persona) self.model = Ollama(name=model_name, base_url=LLAMA_BASE_URL) def process(self, question: str): processed_question = TextProcessing.truncate_text( question, max_words=MAX_LLM_INPUT_LENGTH, max_sentences=MAX_LLM_INPUT_SENTENCES, ) return processed_question Answer : Set Limit for max_words and max_sentences like, max_words = 500 # its enough no need for sentences Remove this part and then try it (optional ): # After finishing all chunks, yield any remaining accumulated text yield...

Full answer by Yogesh: https://stackoverflow.stackexchange.com/a/79786286 (CC BY-SA 4.0).

Need it done for your business? UphillTech.