Sitemap hint in robots.txt
June 13, 2007
Just a quick tip for those of you that are building XML sitemaps for your web sites. You can now add a line to your robots.txt file to include a pointer to your sitemap file, it would look like this:
Sitemap: http://www.example.com/sitemap.xml
This will allow your sitemap to be picked up by several search engines automatically. I first noticed this about a month ago, not sure how long this feature has been around.
Tweet
add to del.icio.usRelated Entries
- Pinging Search Engines when Sitemaps Change - July 18, 2007
- Google Sitemaps Accepts RSS and Atom Feeds - September 12, 2005
- Google Site Verification - September 12, 2005
- Loss of traffic due to Google sitemaps - July 24, 2005
- Google Sitemaps Protocol - June 6, 2005
Trackbacks
Trackback Address: 636/AC2A18A0F0BB16F50B4285B93D9E75B5
- SEO list: Where to add your Google Sitemaps. Jakob Montrasio's Net.
- SEO list: Where to add your Google Sitemaps. Jakob Montrasio's Net.
Comments
On 06/21/2007 at 2:30:16 PM EDT Keilaron wrote:
1
What about compatibility? IIRC, unless the RFC has changed, only User-Agent: and Disallow: are expected. And although Allow: is supposedly supported by Googlebot, even Google's robots.txt validator tells me it not to put it (?!). Where should Sitemap: be? At the beginning? End? Anywhere? I figure it's best to put it in a meta tag of the root index page, honestly.
On 06/21/2007 at 4:30:19 PM EDT Pete Freitag wrote:
2
Keilaron, the robots.txt RFC allows for "extensions":
extension = token : *space value [comment] CRLF
On 06/21/2007 at 5:36:39 PM EDT Keilaron wrote:
3
Indeed, I stand corrected - and in fact, I see that Allow is even in the RFC as well. How odd - Just about every reference I've seen out there only mentions Disallow. Thanks for the info!
On 06/21/2007 at 7:51:27 PM EDT Pete Freitag wrote:
4
Your Welcome, I had to look it up myself so I learned something too!
On 08/10/2007 at 12:25:09 PM EDT Mr. Apache wrote:
5
Some bots don't recognize this yet so its safer to put at the bottom of your robots.txt file like @ http://www.askapache.com/seo/updated-robotstxt-for-wordpress.html
On 06/10/2008 at 1:28:01 AM EDT T Singh wrote:
6
It's been around for many years now... know more at gianiji.com.
Post a Comment
Recent Entries
- Writing Secure CFML cfObjective 2013 Slides
- Upgrading to Java 7 on Linux
- J2EE Sessions in CF10 Uses Secure Cookies
- Learn about ColdFusion Security at cfObjective 2013
- Session Loss and Session Fixation in ColdFusion
- FuseGuard 2.3 Released
- CKEditor Spell Checker Plugin
- Adobe Says Go Ahead and Upgrade your ColdFusion JVM
