.Google's John Mueller addressed an inquiry concerning why Google marks pages that are forbidden coming from creeping through robots.txt and why the it's secure to neglect the related Browse Console reports regarding those crawls.Bot Traffic To Inquiry Parameter URLs.The individual asking the concern documented that robots were actually developing links to non-existent concern specification URLs (? q= xyz) to webpages along with noindex meta tags that are actually likewise blocked out in robots.txt. What cued the question is that Google.com is creeping the web links to those pages, getting shut out by robots.txt (without seeing a noindex robotics meta tag) then receiving turned up in Google Browse Console as "Indexed, though obstructed by robots.txt.".The person talked to the following inquiry:." But below's the large concern: why would Google index web pages when they can't even observe the web content? What's the perk during that?".Google.com's John Mueller verified that if they can not crawl the webpage they can't view the noindex meta tag. He also makes an intriguing reference of the site: search driver, recommending to overlook the end results because the "average" customers won't observe those results.He created:." Yes, you're proper: if our team can't creep the page, our team can't observe the noindex. That stated, if our team can not creep the pages, after that there is actually not a great deal for us to mark. Therefore while you may observe a few of those web pages with a targeted website:- inquiry, the average customer won't find all of them, so I definitely would not bother it. Noindex is actually likewise alright (without robots.txt disallow), it just suggests the Links will definitely wind up being actually crawled (and find yourself in the Browse Console report for crawled/not listed-- neither of these statuses trigger issues to the rest of the web site). The fundamental part is actually that you do not create them crawlable + indexable.".Takeaways:.1. Mueller's response validates the limitations being used the Internet site: search evolved hunt operator for analysis reasons. Some of those reasons is actually due to the fact that it is actually certainly not hooked up to the frequent hunt index, it is actually a distinct trait altogether.Google.com's John Mueller commented on the web site hunt operator in 2021:." The brief response is that an internet site: query is actually not indicated to become total, nor made use of for diagnostics reasons.A site inquiry is a details kind of hunt that limits the results to a particular web site. It's primarily simply the word site, a colon, and after that the internet site's domain name.This question confines the end results to a certain web site. It's not implied to be a comprehensive assortment of all the pages from that internet site.".2. Noindex tag without utilizing a robots.txt is fine for these sort of conditions where a crawler is linking to non-existent webpages that are acquiring found by Googlebot.3. Links with the noindex tag are going to generate a "crawled/not indexed" entry in Browse Console and that those will not have an adverse effect on the rest of the site.Go through the inquiry and also address on LinkedIn:.Why will Google.com mark web pages when they can not also find the content?Included Image by Shutterstock/Krakenimages. com.