Also, some sites have a LOT of generated pages for a tiny section. For instance,...

		TazeTSchnitzel on Aug 11, 2012 \| parent \| context \| favorite \| on: How to crawl a quarter billion webpages in 40 hour... Also, some sites have a LOT of generated pages for a tiny section. For instance, crawling my GitHub account with my homemade crawler was impossible. There is an amazing level of depth there.