Using Langchain's MarkdownTextSplitter with a tokenizer

spark_sara

I'm not sure how to chunk a Markdown file with Langchain's MarkdownTextSplitter and at the same making sure the chunks don't overflow the maximum token size for the llm we will be using. As far as I understand, the chunkSize parameter for the MarkdownTextSplitter references the character length of the chunks and not the token length . I now want to use MarkdownTextSplitter together with the tiktoken library by setting the lengthFunction for the...

Originally asked by michielve on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Good thread on this — worth a read before you pay anyone to build it.

Discussion: https://stackoverflow.com/questions/79863365/using-langchains-markdowntextsplitter-with-a-tokenizer.

Want it set up for you? UphillTech.