Currently it is an on going struggle to have all your website?s pages indexed by the major search engines. This problem is especially prevalent if you have a very large site that is a number of layers deep. Typically the spiders will search down a couple of layers and then given up and move on to a different site, leaving many of your sites pages undiscovered. Recently the most effective remedy to this issue has been the use of a sitemap. This is a simple text based page that contains links to many deeper pages that would not otherwise be indexed by the search engines. By being text based the sitemap is easily spidered and is discovered by search engines through a link from the home page when the site was crawled.

While helping to alleviate the common problem of not having all of your sites pages indexed, the sitemap is not without flaws and limitations. Firstly the sitemap has to be limited in length to 101K and if more size is needed then multiple sitemaps have to be used. A more significant problem is that the site map is only followed by search engines when the site is crawled and for newer or less established websites in can be weeks or months between crawlings.

The ultimate solution to all these problems is the Google Sitemap Protocol. This is how Google describes the new Google Sitemap:

Google Sitemaps is an easy way for you to help improve your coverage in the Google index. It's a collaborative crawling system that enables you to communicate directly with Google to keep us informed of all your web pages, and when you make changes to these pages. The Google Sitemap Protocol will allow you to inform search engines about the new and existing URL?s on your site that are available for crawling. The Sitemap Protocol is essentially an XML file that lists the URL?s for a site. This list can accommodate sites of virtually any size (<50,000URL?s) and is submitted directly to Google. It even allows you to ?hint? to Google how often the site should be crawled and what URL?s are the most important. While Google states that using a sitemap feed doesn?t automatically mean better listings in the search results it will most likely help as more pages are indexed and thus more links recognized. One consideration when submitting your site is that now Google with have a more complete view of you site, so if you are using any search engine spam such as duplicate pages the probability of getting caught will increase significantly. From personal experience once you have submitted an xml sitemap to Google for a particular site it can take as long as two months for Google to begin indexing your pages. I have had a situation where the number of indexed pages jumped from 17 to approximately 1500 once all the errors on the xml sitemap had been fixed. This is clear evidence that your website can benefit from submitting to Google sitemaps, you just have to be patient with the error correction and amount of time before your site is indexed.