Googlebot goes from a crawl to a sprint — for some sites

Back in 2000, Google's Googlebot — the crawler that updates the Google index with your site's latest changes — used to be a bit of a slowpoke. Often, the index wasn't updated for months, creating an ironic and unwanted glitch in the near-instantaneous nature of the Web. In fact, for news sites and other time-sensitive content, web searches in those distant days were more of a trawl through an archive, a saunter through recent history, than a scan through real-time changes on the web.

Changes at the Googleplex over recent years have started to transform the instantaneity and currency of web searches. By mid-2000, Google was sending its Googlebot out more or less once a month, and by summer 2003 many webmasters could reasonably expect to see the 'bot in their site logs nearly every day — albeit as part of an incremental index update rather than a full top-to-bottom scan.

Today, Google's Matt Cutts is reporting that for some sites, new documents can show up in the Google index in a matter of minutes. A logical extension of this would be truly real-time index updates for everyone — perhaps by change notices going directly to Google from suitably vetted sites. But that's got to be a little way off…

Inevitable question: how do I get my site to refresh in Google's listings with that kind of immediacy? A lot of web experts and SEOs, Qwerius included, have already started to try to find out, so stay tuned. Meanwhile, it's probably a good idea to get your Natural Search team to look at this site-visibility issue, starting with the settings in the Google Webmaster Console.