How to create vector embeddings using SentenceTransformers?

spark_sara

I found this code: https://github.com/pixegami/langchain-rag-tutorial/blob/main/create_database.py It takes the document, splits it into chunks, creates vector embeddings for each chunk, and saves those into Chroma Database. However, the source code uses OpenAI key to create embeddings. Since I don't have access to OpenAI I tried to use free alternative, - SentenceTransformers. But I wasn't able to rewrite the code properly. Here is my curent...

Originally asked by Rig Raizon on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

Define your embedding model, with HuggingFaceEmbeddings from langchain_community.embeddings import HuggingFaceEmbeddings embedder = HuggingFaceEmbeddings( model_name = "sentence-transformers/all-MiniLM-L6-v2" ) Then embed the chunks into vectorDB db = Chroma.from_documents( documents=chunks, embedding=embedder, persist_directory=CHROMA_PATH ) db.persist()

Full answer by j3ffyang: https://stackoverflow.stackexchange.com/a/78250313 (CC BY-SA 4.0).

Need it done for your business? UphillTech.