Sitemap hint in robots.txt

June 13, 2007
web

Just a quick tip for those of you that are building XML sitemaps for your web sites. You can now add a line to your robots.txt file to include a pointer to your sitemap file, it would look like this:

Sitemap: http://www.example.com/sitemap.xml

This will allow your sitemap to be picked up by several search engines automatically. I first noticed this about a month ago, not sure how long this feature has been around.


Like this? Follow me ↯


You might also like:

17 people found this page useful, what do you think?

Comments

What about compatibility? IIRC, unless the RFC has changed, only User-Agent: and Disallow: are expected. And although Allow: is supposedly supported by Googlebot, even Google's robots.txt validator tells me it not to put it (?!). Where should Sitemap: be? At the beginning? End? Anywhere? I figure it's best to put it in a meta tag of the root index page, honestly.
Keilaron, the robots.txt RFC allows for "extensions": extension = token : *space value [comment] CRLF
Indeed, I stand corrected - and in fact, I see that Allow is even in the RFC as well. How odd - Just about every reference I've seen out there only mentions Disallow. Thanks for the info!
Your Welcome, I had to look it up myself so I learned something too!
Some bots don't recognize this yet so its safer to put at the bottom of your robots.txt file like @ http://www.askapache.com/seo/updated-robotstxt-for-wordpress.html
It's been around for many years now... know more at gianiji.com.


Foundeo Inc.