XML Sitemaps

XML Sitemaps

Google, Yahoo!, and Microsoft all support a protocol known as XML Sitemaps. Google first

announced it in 2005, and then Yahoo! and Microsoft agreed to support the protocol in 2006.

Using the Sitemaps protocol you can supply the search engines with a list of all the URLs you

would like them to crawl and index.

Adding a URL to a Sitemap file does not guarantee that a URL will be crawled or indexed.

However, it can result in pages that are not otherwise discovered or indexed by the search

engine getting crawled and indexed. In addition, Sitemaps appear to help pages that have been

relegated to Google’s supplemental index make their way into the main index.

This program is a complement to, not a replacement for, the search engines’ normal, link-based

crawl. The benefits of Sitemaps include the following:

• For the pages the search engines already know about through their regular spidering, they

use the metadata you supply, such as the last date the content was modified (lastmod

date) and the frequency at which the page is changed (changefreq), to improve how they

crawl your site.

• For the pages they don’t know about, they use the additional URLs you supply to increase

their crawl coverage.

• For URLs that may have duplicates, the engines can use the XML Sitemaps data to help

choose a canonical version.

• Verification/registration of XML Sitemaps may indicate positive trust/authority signals.

• The crawling/inclusion benefits of Sitemaps may have second-order positive effects, such

as improved rankings or greater internal link popularity.

The Google engineer who in online forums goes by GoogleGuy (a.k.a. Matt Cutts, the head of

Google’s webspam team) has explained Google Sitemaps in the following way:

Imagine if you have pages A, B, and C on your site. We find pages A and B through our normal

web crawl of your links. Then you build a Sitemap and list the pages B and C. Now there’s a

chance (but not a promise) that we’ll crawl page C. We won’t drop page A just because you

didn’t list it in your Sitemap. And just because you listed a page that we didn’t know about

doesn’t guarantee that we’ll crawl it. But if for some reason we didn’t see any links to C, or

maybe we knew about page C but the URL was rejected for having too many parameters or

some other reason, now there’s a chance that we’ll crawl that page C.

Sitemaps use a simple XML format that you can learn about at http://www.sitemaps.org. XML

Sitemaps are a useful and in some cases essential tool for your website. In particular, if you

have reason to believe that the site is not fully indexed, an XML Sitemap can help you increase

the number of indexed pages. As sites grow in size, the value of XML Sitemap files tends to

increase dramatically, as additional traffic flows to the newly included URLs.

Layout of an XML Sitemap

The first step in the process of creating an XML Sitemap is to create an .xml Sitemap file in a

suitable format. Since creating an XML Sitemap requires a certain level of technical know-how,

it would be wise to involve your development team in the XML Sitemap generator process

from the beginning. Figure 6-2 shows an example of some code from a Sitemap.

FIGURE 6-2. Sample XML Sitemap from Google.com

To create your XML Sitemap, you can use the following:

An XML Sitemap generator

This is a simple script that you can configure to automatically create Sitemaps, and

sometimes submit them as well. Sitemap generators can create these Sitemaps from a URL

list, access logs, or a directory path hosting static files corresponding to URLs. Here are

some examples of XML Sitemap generators:

• SourceForge.net’s google-sitemap_gen

• ROR Sitemap Generator

• XML-Sitemaps.com Sitemap Generator

• Sitemaps Pal

• XML Echo

Simple text

You can provide Google with a simple text file that contains one URL per line. However,

Google recommends that once you have a text Sitemap file for your site, you use the

Sitemap Generator to create a Sitemap from this text file using the Sitemaps protocol.

Syndication feed

Google accepts Really Simple Syndication (RSS) 2.0 and Atom 1.0 feeds. Note that the

feed may provide information on recent URLs only.