The X-Robots-Tag Header

The idea behind X-Robots-Tag is that it allows robots directives normally found in a meta element to be sent as part of the server’s HTTP response headers. In other words the instructions are sent with the file rather than within the file, with the main advantage that this can be used with any type of content. So an HTTP response might look like this (using the Live HTTP Headers plugin for Firefox):

HTTP/1.x 200 OK
Date: Thu, 27 Aug 2009 09:21:23 GMT
Server: Apache/2.0.52 (Red Hat)
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html
X-Robots-Tag: noindex


A ‘noindex’ for Images

As an example consider the case of a webmaster wishing to prevent images being indexed and appearing in search results. One approach might be to add the entire ‘images’ directory to the robots.txt file (i.e. Disallow: /images). The problem with this is that the robots.txt file provides crawling directives rather than indexing directives – that is, the search engine is instructed not to visit the ‘images’ directory when crawling your site, but could still end up at one of your images if it is embedded in someone else’s site. Furthermore, what if we only want to block crawlers from certain images – the robots.txt file would quickly become large and unmanageable.

X-Robots-Tag in Apache (htaccess)

A combination of ‘Header set’ and the FilesMatch directive allows us to add the robots tag in Apache. Here’s a couple of examples which could be in Apache’s httpd.conf or .htaccess files.

Add ‘noindex’ header to all image files:

<FilesMatch "\.(gif|jpe?g|png)$">
Header set X-Robots-Tag "noindex"
</FilesMatch>

Add noindex to image files matching a particular pattern – in this case those with ‘thumbnail’ in the filename (i.e. /images/product41-thumbnail.jpg would be served with a noindex header while /images/product41-large.jpg would not):

<FilesMatch "images/.+-thumbnail\.jpg$">
Header set X-Robots-Tag "noindex"
</FilesMatch>

X-Robots-Tag in PHP

PHP’s header function allows us to send any HTTP header, as follows:

header('X-Robots-Tag: noindex,nofollow');

This supports more complex scenarios than using Apache alone. Rather than applying the same rule to all images, we could use some custom logic such as a database lookup to determine whether to add the X-Robots-Tag header. The first step would be to route requests for images to a php script using Apache’s .htaccess file (e.g. requests for jpeg files within the images directory will be handled by ‘image-handler.php’).

RewriteEngine On
RewriteRule ^images/.*.jpg$ image-handler.php

Then in image-handler.php, we can perform our custom logic (defined in the allowImageIndexing() function) and set the appropriate header:

$filename = basename($_SERVER['REQUEST_URI']); // extract image filename 
header('Content-Type: image/jpg');             // set content type (otherwise it 
                                               // will be the dafault text/html)
if (!allowImageIndexing($filename)) {          // perform lookup
    header('X-Robots-Tag: noindex');           // set the x-robots-tag accordingly
}
readfile('images/' . $filename);               // stream image file in response

X-Robots-Tag Search Engine Support

Fortunately the three major search engines all now support the X-Robots-Tag

• Google – supports any value which can be used in the robots <meta> tag
• Yahoo – supports noindex, noarchive, nofollow, nosnippet
• Bing – as Yahoo plus noodp