Accurate, Focused Research on Law, Technology and Knowledge Discovery Since 2002

Google’s Beta “Collaborative Crawling System”

“Google Sitemaps is an experiment in web crawling. Using Sitemaps to inform and direct our crawlers, we hope to expand our coverage of the web and improve the time to inclusion in our index. By placing a Sitemap-formatted file on your webserver, you enable our crawlers to find out what pages are present and which have recently changed, and to crawl your site accordingly.” [Link]

  • Google Sitemaps (BETA)
  • Sorry, comments are closed for this post.