Mark Gilbert’s Blog

Google’s URL indel has now hit one trillion for the first time. 26 million was the size of their first index in 1998, with 1billion being reached in 2000. The Google Blog has a nice write up of how they manage to achieve this feat. Makes for an interesting read.

Back then [In 1998], we did everything in batches: one workstation could compute the PageRank graph on 26 million pages in a couple of hours, and that set of pages would be used as Google’s index for a fixed period of time. Today, Google downloads the web continuously, collecting updated page information and re-processing the entire web-link graph several times per day

Cheers

This entry was posted on Saturday, July 26th, 2008 at 12:07 and is filed under Technology. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.