how to check for duplicate documents in vectorstore efficiently?

pasta_paul

How can i check for duplicate documents in my vectorstore, when adding documents? Currently I am doing something like: vectorstore = Chroma( persist_directory=persist_dir, embedding_function=embeddings ) documents = vectorstore.get()['documents'] final_docs = list(filter(lambda x: x not in documents, final_docs)) vectorstore.add_documents(documents=final_docs, embedding=embeddings) However, I am wondering about the performance in large datasets....

Originally asked by information_interchange on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

You can try compute hash for each document and then add it in python Set() which will ensure there are no duplicate documents import hashlib # Set to store hashes of documents document_hashes = set() def add_document(doc): # Compute SHA256 hash of the document doc_hash = hashlib.sha256(doc.encode()).hexdigest() # Check if hash is in the set if doc_hash in document_hashes: print("Duplicate document detected!") return False else: document_hashes.add(doc_hash) #Your Code for adding document in vector stores print("Document added successfully!") return True #Sample documents doc1 = "This is document 1" doc2 = "This is document 2" doc3 = "This is document 1" add_document(doc1) add_document(doc2)...

Full answer by ZKS: https://stackoverflow.stackexchange.com/a/76963929 (CC BY-SA 4.0).

Need it done for your business? UphillTech.