Isn’t it crazy when pages that should be blocked by robots.txt end up in search results anyway? During this week’s Wednesday Workshop, we break down just how and why this happens.
Welcome back to another Wednesday Workshop from DealerOn.
Last month, during our video on Technical SEO, we discussed that Google will no longer support robots.txt files that contain noindex directives starting September 1st. Rather than using robots.txt to block pages from being crawled, implementing noindex in robots meta tags is a better option.
But when you’re using robots.txt to hide a page from search engines, have you ever noticed that sometimes it gets indexed and ranked anyway? Today, we are going to investigate why this happens.
Google has said that it cannot index content that is blocked by robots.txt, and will typically index more accessible pages first; but it does its best to determine how relevant the content is to a search query through a couple of methods.
First, Google will compare a page’s URL with other URLs to find similarities and consistencies that may be descriptive of page content. However, this can still be difficult when a page is set to noindex.
Then, Google will evaluate the sources and types of links coming to the robotted page. In doing this, they are trying to establish relevance as it pertains to the search query.
If Google determines through backlinks to the blocked page that the content could be relevant, it may then display the page in search results; and if people are linking to the robotted content, Google feels it may be worthwhile for other users.
So what does this mean for you?
You need to limit the type of content that you block with your robots.txt file, as Google may choose to guess what it is and rank it anyway. Guessing probably won’t work out to your benefit! If it is something you truly need to have omitted from search, implementing noindex in robots meta tags is one of the best ways to prevent indexation.
That’s all the time we have left for today’s workshop. We covered a very challenging topic today; so feel free to leave your questions and comments down below, and we’ll get back to you shortly. Thanks for watching. We’ll see you next week with another Wednesday Workshop from DealerOn.