Great question — answering as the UphillTech team:
Docling only performs document conversion (for example, from PDF to Markdown). You'd still have to split the document into chunks, send each chunk to the embedding model to get embedding vectors, and store each embedding vector along with metadata (e.g. source document, chunk number, etc.) into your vector database. For retrieval, you would convert the user's query into an embedding vector and do a similarity search with all of the other vectors in your vector store. From there, you'd summarize the chunks or perform further processing to provide a useful response for the user. Llama.cpp server only handles the LLM chat and creation of vector embeddings. For everything else, you either have...
Full answer by tlo8: https://stackoverflow.stackexchange.com/a/79901429 (CC BY-SA 4.0).
Need it done for your business? UphillTech.