Does blocking api URL hurt rendering by googlebot

spark_sara

In my GSC crawl stats most of the 404s and URLs using page resource load are: API URLs for a JavaScript part of our content Cloudflare's managed challenge If I add an entry to robots.txt for 1, will it break how Google renders portions of our site? And for two, will it break anything if I stop Googlebot crawl requests?

Originally asked by ooxbal on Stack Exchange (webmasters) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

Robots.txt file will not stop a page from being indexed (or attempted to be indexed) if You have internal links to those pages There are external links to those pages. Warning: Don't use a robots.txt file as a means to hide your web pages from Google search results. If other pages point to your page with descriptive text, Google could still index the URL without visiting the page. If you want to block your page from search results, use another method such as password protection or noindex. From Introduction to robots.txt Google is already getting a 404, so putting it in robots.txt is not going to make it worse. Equally, it probably will not stop the errors. I would suggest that the fix is...

Full answer by Rohit Gupta: https://webmasters.stackexchange.com/a/141904 (CC BY-SA 4.0).

Need it done for your business? UphillTech.