Why does do_robots() Allow: /wp-admin/admin-ajax.php by default?

cafe_carl

This is happening with a fresh, new WordPress install. But all you need to do is look at the crazy do_robots() function, which outputs the following... User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://knowingart.com/wp-sitemap.xml Sending robots to "wp-admin" (by default) makes no sense: Why would I send random robots into my admin directory?! The admin directory is for me, and me alone. Robots/crawlers should...

Originally asked by Jay Brunet on Stack Exchange (wordpress) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

First: have a read-up on robots.txt , if you haven't. Another good reference is this one from Yoast . The sample you posted: User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://knowingart.com/wp-sitemap.xml ...tells all user-agents (ie, all crawlers) that they are not allowed to crawl anything in your wp-admin directory except /wp-admin/admin-ajax.php , which is required by WordPress for properly-functioning AJAX . It's not telling robots to go to your /wp-admin/ directory; it's telling them they're unwelcome there (except for the AJAX requirement). As for your question #4, I'm afraid I don't have an answer to that one.

Full answer by Pat J: https://wordpress.stackexchange.com/a/403758 (CC BY-SA 4.0).

Need it done for your business? UphillTech.