AsyncHtmlLoader not fetching the Wikipedia page

soup_sam

I am learning rag and below code snippet mentioned in the book. It giving error so Iadded agent thing but still no luck. from langchain_community.document_loaders import AsyncHtmlLoader #This is the URL of the Wikipedia page on the 2023 Cricket World Cup url="https://en.wikipedia.org/wiki/2023_Cricket_World_Cup" loader = AsyncHtmlLoader(url) html_data = loader.load() print(html_data) Output: Fetching pages:...

Originally asked by Muhammad Faizan Khan on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

When I run your script then (before Fetching pages ) I also see USER_AGENT environment variable not set, consider setting it to identify your requests. Error shows link https://w.wiki/4wJS which describes how to respect robot policy for Wikipedia. Point 2 is about header USER-AGENT and there is link to another page which shows example in Python with requests import requests url = 'https://example/...' headers = {'User-Agent': 'CoolBot/0.0 (https://example.org/coolbot/; coolbot@example.org)'} response = requests.get(url, headers=headers) When I use it with your URL then I get expected data. And AsyncHtmlLoader also works for me when I set the same User-Agent as environment variable or run...

Full answer by furas: https://stackoverflow.stackexchange.com/a/79903356 (CC BY-SA 4.0).

Need it done for your business? UphillTech.