Sitemap hint in robots.txt

June 13, 2007

Just a quick tip for those of you that are building XML sitemaps for your web sites. You can now add a line to your robots.txt file to include a pointer to your sitemap file, it would look like this:


This will allow your sitemap to be picked up by several search engines automatically. I first noticed this about a month ago, not sure how long this feature has been around.

Related Entries

17 people found this page useful, what do you think?



What about compatibility? IIRC, unless the RFC has changed, only User-Agent: and Disallow: are expected. And although Allow: is supposedly supported by Googlebot, even Google's robots.txt validator tells me it not to put it (?!). Where should Sitemap: be? At the beginning? End? Anywhere? I figure it's best to put it in a meta tag of the root index page, honestly.
Keilaron, the robots.txt RFC allows for "extensions": extension = token : *space value [comment] CRLF
Indeed, I stand corrected - and in fact, I see that Allow is even in the RFC as well. How odd - Just about every reference I've seen out there only mentions Disallow. Thanks for the info!
Your Welcome, I had to look it up myself so I learned something too!
Some bots don't recognize this yet so its safer to put at the bottom of your robots.txt file like @
It's been around for many years now... know more at

Recent Entries