For RAG, we create embeddings for the documents. Longer documents have to be split in chunks by a process called chunking. The reason for this is that embeddings can only be generated for a limited number of tokens. For example, OpenAI has a limit of 8192 input tokens per embedding on each of their embedding models ( text-embedding-3-small , text-embedding-3-large and text-embedding-ada-002 at the time of writing). What are the reasons to limit...
Originally asked by Thomas W. on Stack Exchange (ai) (CC BY-SA 4.0).
