Palo Alto: A History of California, Capitalism, and the World Highlight

In their 1998 coauthored academic paper reflecting on Google’s first years, Brin and Page acknowledge that web crawlers are hectic programs by nature. “Because of the immense variation in web pages and servers, it is virtually impossible to test a crawler without running it on [a] large part of the internet,” they write. “Invariably, there are hundreds of obscure problems which may only occur on one page out of the whole web and cause the crawler to crash, or worse, cause unpredictable or incorrect behavior.”21 Like automata from a cautionary fable, crawlers are much easier to create than they are to manage, and this early document has an ominous Sorcerer’s Apprentice energy.