Accurate, Focused Research on Law, Technology and Knowledge Discovery Since 2002

With Google dominating search, the internet needs crawl neutrality

Fast Company: “Today, one company—Google—controls nearly all of the world’s access to information on the internet. Their monopoly in search means for billions of people, their gateway to knowledge, to products, and their exploration of the web is in the hands of one company. Most agree, this lack of competition in search is bad for individuals, communities and democracy. Unbeknownst to many, one of the biggest obstacles to competing in search is a lack of crawl neutrality. The only way to build an independent search engine and the chance to fairly compete against Big Tech is to first efficiently and effectively crawl the Internet. However, the web is an actively hostile environment for upstart search engine crawlers, with most websites only allowing Google’s crawler and discriminating against other search engine crawlers like Neeva’s. This critically important, yet often overlooked, issue has an enormous impact on preventing upstart search engines like Neeva from providing users with real alternatives, further reducing competition in search. Similar to net neutrality, today we need an approach to crawl neutrality. Without a change in policy and behavior, competitors in search will remain fighting with one hand tied behind our backs…”

Sorry, comments are closed for this post.