John Mueller revealed that URLs are crawled at different rates. That’s somewhat well understood. What was of more interest was that he said some URLs can be crawled as little as once every six months.
The publisher stated:
“We’re seeing stuff that’s from a long time ago, where we’ve changed the noindex nofollow but we’re still seeing it in the index. And this is several months after we’ve changed this.”
John Mueller answered:
“I think the hard part here is that we don’t crawl URLs with the same frequency all the time. So some URLs we will crawl daily. Some URLs maybe weekly. Other URLs every couple of months, maybe even every once half year or so.
So this is something that we try to find the right balance for, so that we don’t overload your server.
And if you made significant changes on your website across the board then probably a lot of those changes are picked up fairly quickly but there will be some leftover ones.
So in particular if you do things like site queries then there’s a chance that you’ll see those URLs that get crawled like once every half year. They’ll still be there after a couple of months.
And that’s kind of… the normal time for us to kind of reprocess/re-crawl things. So it’s not necessarily a sign that something is technically completely broken.
But it does mean that if you think that these URLs should really not be indexed at all, then maybe you can kind of back that up and say well here’s a sitemap file with the last modification date so that Google goes off and tries to double-check these a little bit faster than otherwise.”
Use The Site Map to Trigger Updated Crawling
John Mueller suggested updating the site map and letting Googlebot discover the last modified date and using that as a hint for it to go out and crawl the old web pages.
Google URL Inspection Tool
Something John Mueller didn’t mention is using Google’s URL Inspection tool. According to Google’s Webmaster Help page on re-indexing, a submission can take up to a week or two.
The URL Inspection tool is useful if you have a few individual URLs that need re-crawling. If you have a large amount of web pages, Google recommends submitting a site map instead.
More information on how to ask Google to re-crawl URLs here
Watch the Webmaster Hangout here
https://youtu.be/gC7aVygbMIk?t=2594[/vc_column_text][/vc_column][/vc_row][vc_section][vc_row][vc_column][vc_message message_box_color=”orange” icon_fontawesome=”fa fa-external-link”]This article was originally posted at Search Engine Journal by Roger Montti on October 16, 2018.[/vc_message][/vc_column][/vc_row][/vc_section]