.Google's John Mueller addressed a question about why Google.com marks pages that are actually disallowed coming from crawling by robots.txt and also why the it's safe to dismiss the similar Explore Console documents concerning those crawls.Bot Website Traffic To Question Criterion URLs.The individual inquiring the concern documented that crawlers were producing hyperlinks to non-existent inquiry criterion Links (? q= xyz) to pages along with noindex meta tags that are likewise obstructed in robots.txt. What cued the inquiry is actually that Google.com is actually creeping the hyperlinks to those pages, getting obstructed by robots.txt (without noticing a noindex robots meta tag) at that point acquiring reported in Google Search Console as "Indexed, though shut out by robots.txt.".The person inquired the observing inquiry:." Yet right here's the big question: why would certainly Google.com index pages when they can not even see the web content? What is actually the perk because?".Google.com's John Mueller verified that if they can not crawl the page they can't view the noindex meta tag. He also produces a fascinating reference of the website: search operator, urging to ignore the outcomes considering that the "ordinary" individuals won't view those outcomes.He wrote:." Yes, you're appropriate: if our team can not crawl the page, our team can't view the noindex. That said, if we can not crawl the web pages, after that there's not a whole lot for us to mark. Thus while you may see a number of those webpages with a targeted internet site:- question, the common consumer will not see them, so I wouldn't bother it. Noindex is additionally fine (without robots.txt disallow), it merely means the Links are going to find yourself being actually crept (and also wind up in the Look Console document for crawled/not indexed-- neither of these conditions trigger concerns to the remainder of the website). The fundamental part is that you don't produce all of them crawlable + indexable.".Takeaways:.1. Mueller's response validates the limitations in using the Site: hunt advanced hunt operator for analysis factors. Some of those reasons is actually due to the fact that it's not hooked up to the frequent hunt index, it is actually a separate factor completely.Google's John Mueller commented on the website hunt operator in 2021:." The short response is that a website: inquiry is actually certainly not meant to be comprehensive, nor utilized for diagnostics purposes.A web site inquiry is a specific sort of hunt that restricts the end results to a particular internet site. It is actually essentially merely words site, a bowel, and after that the site's domain name.This query confines the results to a certain site. It's certainly not implied to become a complete collection of all the webpages from that web site.".2. Noindex tag without using a robots.txt is alright for these kinds of conditions where a crawler is connecting to non-existent webpages that are obtaining discovered through Googlebot.3. Links along with the noindex tag will certainly generate a "crawled/not catalogued" entry in Browse Console and that those won't have an adverse result on the rest of the web site.Check out the inquiry and also answer on LinkedIn:.Why would certainly Google mark pages when they can not also view the material?Featured Image through Shutterstock/Krakenimages. com.