"unstructured" and langchain's...

vegan_vicky

I want to read a webpage and split it into chunks to feed a vector database in a RAG pipeline. This webpage has python code examples on it, but I cannot create chunks with that code text, it is ignored by the splitters. I tried both unstructured python package, and HTMLHeaderTextSplitter class (from langchain_text_splitter package) with the same result. The HTML code I want to parse looks like this: ... <h2 id="examples_1">Examples</h2> <h3...

Originally asked by Abraham Martín Expósito on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

with langchain Version: 1.2.10, there is Language enum used with RecursiveCharacterTextSplitter from langchain_text_splitters import Language,RecursiveCharacterTextSplitter HTML_CODE=""" <h2 id="examples_1">Examples</h2> <h3 id="create-camera">Create camera</h3> <p>Create a camera. Setting invalid_entity_id as the parent entity will make the camera to be created under the Ego entity, as it must be</p> <pre><code class="language-python">camera_id = workspace.create_entity( anyverse_platform.WorkspaceEntityType.Camera, "New Camera", anyverse_platform.invalid_entity_id ) </code></pre> <hr> <h3 id="add-resource-to-workspace">Add resource to workspace</h3> """...

Full answer by Yilmaz: https://stackoverflow.stackexchange.com/a/79903429 (CC BY-SA 4.0).

Need it done for your business? UphillTech.