Google boosts Robot Exclusion Protocol with 'unavailable-after' meta tag

We've all been there a million times: you do a Google search, click a likely-looking result and end up with an Error 404 or (if you're lucky) a "Page Moved" message on the landing site. Either way, it's frustrating for search users and webmasters alike: a lost opportunity for everyone, and the kind of statistic we don't like to see in the web analytics.

To tackle this issue, Google announced 27-July-2007 that it has formally extended its REP (Robots Exclusion Protocol) to enable webmasters to opt to make pages auto-expire in the SERPs (Search Engine Results Pages). The extended protocol admits a new and self-explanatory meta tag:

<META NAME="GOOGLEBOT" CONTENT="unavailable_after: 25-Aug-2008 15:00:00 EST">

The above tag would instruct Google to remove the HTML page containing the tag from the SERPs after 3pm Eastern Standard Time on 25th August 2008. Note that this doesn't remove the affected page entirely from the Google memory machine: it simply prevents it from appearing in the results listings. Google's URL removal tool is still required, therefore, if you need to excise any remaining vestige of a particular URI from Google.

Impact of new 'unavailable-after' tag

The obvious appeal is to websites with time-sensitive pages such as special offers, date-related relevance and the like. But it's a little early to gauge likely take-up of the new tag. Some webmasters might take the view that any page appearing high in the listings, even an obsolete one, is better than none: better to get potential customers to the site — to a suitably crafted landing page, for example — than to see a hard-won high-ranking page simply disappear. (For example, customers could be presented with alternative special offers, or could click through to other relevant content.) Other web teams may put usability and customer-friendliness issues first, especially if other pages from their site are well represented in the SERPs.

Another thought (and this is pure speculation at this point) is whether, after a suitable honeymoon period, Google might choose to progressively enforce — through ranking penalties, for example — the use of the auto-expiry tag on clearly time-sensitive pages. This, it could be argued, is simply an extension of Google's landing-page quality and relevance metrics as seen in action billions of times a day in its organic listings as well as in its AdWords programme. Conversely, Google could choose to use the meta tag to push time-relevant pages higher up in the rankings too.