Common Crawl – free, open repository of web crawl data that can be used by anyone

Common Crawl is a 501(c)(3) non–profit founded in 2007. We make wholesale extraction, transformation and analysis of open web data accessible to researchers.

Overview

  • Over 300 billion pages spanning 19 years.
  • Free and open corpus since 2007.
  • Cited in over 10,000 research papers.
  • 3–5 billion new pages added each month.

Posted in: AI, Education, Internet, Knowledge Management, Legal Research, Search Engines