.Google's John Mueller responded to an inquiry concerning why Google.com indexes web pages that are actually disallowed from crawling by robots.txt and also why the it's risk-free to disregard the associated Look Console records regarding those creeps.Crawler Website Traffic To Inquiry Guideline URLs.The person inquiring the question recorded that bots were actually producing web links to non-existent query criterion Links (? q= xyz) to web pages along with noindex meta tags that are additionally blocked in robots.txt. What triggered the concern is actually that Google.com is actually crawling the web links to those webpages, receiving obstructed by robots.txt (without seeing a noindex robotics meta tag) after that getting turned up in Google Explore Console as "Indexed, though blocked out through robots.txt.".The person talked to the observing inquiry:." However listed here is actually the big concern: why would certainly Google.com index web pages when they can not even see the web content? What is actually the advantage during that?".Google.com's John Mueller affirmed that if they can't crawl the web page they can not observe the noindex meta tag. He also creates an intriguing mention of the website: hunt driver, suggesting to ignore the results due to the fact that the "normal" customers won't see those end results.He wrote:." Yes, you are actually right: if our team can't creep the webpage, our company can't observe the noindex. That pointed out, if our team can't crawl the pages, then there's certainly not a lot for our team to mark. Thus while you could find some of those pages along with a targeted website:- concern, the normal user won't view all of them, so I definitely would not fuss over it. Noindex is actually also fine (without robots.txt disallow), it only means the Links will definitely wind up being actually crawled (and wind up in the Look Console report for crawled/not listed-- neither of these conditions lead to issues to the remainder of the site). The vital part is that you don't produce all of them crawlable + indexable.".Takeaways:.1. Mueller's answer affirms the restrictions in using the Internet site: hunt progressed search driver for diagnostic causes. Among those causes is because it's not hooked up to the normal hunt index, it is actually a separate point entirely.Google.com's John Mueller talked about the website hunt driver in 2021:." The quick solution is actually that a web site: concern is not meant to become complete, neither used for diagnostics reasons.A web site concern is actually a certain sort of hunt that confines the end results to a certain website. It's essentially simply the word internet site, a digestive tract, and after that the web site's domain name.This query restricts the results to a details site. It is actually certainly not implied to become a thorough assortment of all the pages coming from that web site.".2. Noindex tag without utilizing a robots.txt is great for these type of conditions where a crawler is actually connecting to non-existent web pages that are acquiring found by Googlebot.3. URLs along with the noindex tag will produce a "crawled/not indexed" entry in Explore Console and also those won't have a damaging effect on the remainder of the web site.Read the inquiry and also respond to on LinkedIn:.Why would certainly Google index web pages when they can not also find the content?Included Photo by Shutterstock/Krakenimages. com.