pf » Sitemap hint in robots.txt
June 13, 2007
Sitemap hint in robots.txt
Just a quick tip for those of you that are building XML sitemaps for your web sites. You can now add a line to your robots.txt file to include a pointer to your sitemap file, it would look like this:
Sitemap: http://www.example.com/sitemap.xml
This will allow your sitemap to be picked up by several search engines automatically. I first noticed this about a month ago, not sure how long this feature has been around.
add to del.icio.usRelated Entries
- Pinging Search Engines when Sitemaps Change - July 18, 2007
- Google Sitemaps Accepts RSS and Atom Feeds - September 12, 2005
- Google Site Verification - September 12, 2005
- Loss of traffic due to Google sitemaps - July 24, 2005
- Google Sitemaps Protocol - June 6, 2005
Trackback Address: 636/AC2A18A0F0BB16F50B4285B93D9E75B5
- SEO list: Where to add your Google Sitemaps. Jakob Montrasio's Net.
- SEO list: Where to add your Google Sitemaps. Jakob Montrasio's Net.
Comments
On 06/21/2007 at 12:30:16 PM MDT Keilaron wrote:
1
What about compatibility? IIRC, unless the RFC has changed, only User-Agent: and Disallow: are expected. And although Allow: is supposedly supported by Googlebot, even Google's robots.txt validator tells me it not to put it (?!). Where should Sitemap: be? At the beginning? End? Anywhere? I figure it's best to put it in a meta tag of the root index page, honestly.
On 06/21/2007 at 2:30:19 PM MDT Pete Freitag wrote:
2
Keilaron, the robots.txt RFC allows for "extensions":
extension = token : *space value [comment] CRLF
On 06/21/2007 at 3:36:39 PM MDT Keilaron wrote:
3
Indeed, I stand corrected - and in fact, I see that Allow is even in the RFC as well. How odd - Just about every reference I've seen out there only mentions Disallow. Thanks for the info!
On 06/21/2007 at 5:51:27 PM MDT Pete Freitag wrote:
4
Your Welcome, I had to look it up myself so I learned something too!
On 08/10/2007 at 10:25:09 AM MDT Mr. Apache wrote:
5
Some bots don't recognize this yet so its safer to put at the bottom of your robots.txt file like @ http://www.askapache.com/seo/updated-robotstxt-for-wordpress.html
On 06/09/2008 at 11:28:01 PM MDT T Singh wrote:
6
It's been around for many years now... know more at gianiji.com.
- ColdFusion 8 FCKeditor Vulnerability
- Ajax Same Origin Policy No More with Firefox 3.5
- Firefox 3.5 Introduces Origin Header, Security Features
- Tips for Secure File Uploads with ColdFusion
- 7 Years And Blog Entry Number 700
- CFCatch Java Exceptions
- Cheat Sheet for SQL Server
- CFML on Google App Engine for Java
Subscribe to my RSS Feed: 
RSS
RSS
Pete Freitag is a software engineer, and web developer located in central new york.
Pete's company Foundeo Inc. builds products for web developers and provides consulting services.
In 2003 Pete published the ColdFusion MX Developers Cookbook with SAMs Publishing.
More about Pete.
Use SoloSub to subscribe to my RSS feed using Bloglines, MyYahoo, NewsGator, and more.