RAG using Langchain / Chroma - Unable to save more than 99 Records to Database

bash_bob

I'm using the following code to load the content of markdown files (only one file, in my case), split it into chunks and then embed and store the chunks one by one. My file is split into 801 chunks. However, this code is unable to save the embeddings to disk in the vector db. def load_documents(): loader = DirectoryLoader(DATA_PATH, glob="*.md") documents = loader.load() return documents def split_text(documents: list[Document]): text_splitter =...

Originally asked by hassaanq on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

My problem was solved when I re-installed python on my pc due to some other problem. Now, when I run the code, it works like magic! All the chunks are being saved now regardless of whether I save them all in one go or in batches. EDIT: The problem was actually due to a conflict in the installed libraries. It gets resolved when you use a virtual environment, so no need to re-install python.

Full answer by hassaanq: https://stackoverflow.stackexchange.com/a/78774435 (CC BY-SA 4.0).

Need it done for your business? UphillTech.