commoncrawl

• looking for a domain:
• http://index.commoncrawl.org/
• enter:   http://www.cnn.com/*
• download from:
• https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-36/segments/1471982292181.27/warc/CC-MAIN-20160823195812-00279-ip-10-153-172-175.ec2.internal.warc.gz
• list of segments
• https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-40/warc.paths.gz