Langchain / ChromaDB: Why does VectorStore return so many duplicates?

cafe_carl

import os from langchain.llms import OpenAI import bs4 import langchain from langchain import hub from langchain.document_loaders import UnstructuredFileLoader from langchain.embeddings import OpenAIEmbeddings from langchain.text_splitter import RecursiveCharacterTextSplitter from langchain.vectorstores import Chroma os.environ["OPENAI_API_KEY"] = "KEY" loader = UnstructuredFileLoader( 'path_to_file' ) docs = loader.load() text_splitter =...

Originally asked by narcissa on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

the issue is here: Chroma.from_documents(documents=all_splits, embedding=OpenAIEmbeddings()) everytime you execute the file, you are inserting the same documents into the database. you could comment out that part of code if you are inserting from same file. or you could detect the similar vectors using EmbeddingsRedundantFilter Filter that drops redundant documents by comparing their embeddings.

Full answer by Yilmaz: https://stackoverflow.stackexchange.com/a/77568404 (CC BY-SA 4.0).

Need it done for your business? UphillTech.