3.5B Web Pages from CommonCrawl 2012

Overview

Large-scale web crawl dataset containing approximately 3.5 billion web pages collected by CommonCrawl in 2012. Intended for research and experimentation in areas such as web mining, search, and network analysis. Listed within an “awesome-style” collection of computer networks datasets.

• Category: Themed directories
• Type: Dataset (web crawl / big data)
• Year of crawl: 2012
• Source: CommonCrawl (referenced via BigDataNews)
• URL: http://www.bigdatanews.com/profiles/blogs/big-data-set-3-5-billion-web-pages-made-available-for-all-of-us
• Tags: datasets, web, big-data

Features

• Contains about 3.5 billion web pages from a large-scale crawl.
• Based on the CommonCrawl 2012 corpus.
• Suitable for web mining research, including:
  • Content analysis at large scale
  • Topic modeling and classification
  • Language and text mining experiments
• Applicable to search and information retrieval research, such as:
  • Indexing and ranking experiments
  • Query log–independent search evaluation scenarios
• Supports network and graph analysis, including:
  • Web graph construction
  • Link structure and connectivity studies
  • Page-level and domain-level graph metrics
• Appropriate for big data processing frameworks (e.g., Hadoop/Spark-style workflows), given its scale.
• Included in an awesome-style curated list of computer networks datasets, indicating use as a reference dataset for networking and web research communities.

Use Cases

• Academic and industrial web-scale research projects.
• Benchmarking big data processing pipelines and distributed systems.
• Building and testing experimental search engines.
• Studying web structure, connectivity, and evolution around 2012.

Pricing

• Not specified in the provided content. (CommonCrawl datasets are typically freely available, but the exact access terms should be confirmed on the linked source.)