Three papers published by Google that ignited the era of big data

1. GFS paper - published in 2003

  In 2003, Google released the Google File System paper, which is a scalable distributed file system for large, distributed applications that access large amounts of data. It runs on cheap, ordinary hardware and provides fault tolerance. Basically: files are divided into many blocks and stored in a redundant manner on a cluster of commercial machines.

2. MapReduce paper - published in 2004

  Following this was the MapReduce paper published in 2004, which described a distributed computing method for big data. The main idea is to decompose the task and process it simultaneously on multiple computing nodes with weaker processing power, and then merge the results to complete the big data processing.

3. BigTable Paper - Published in 2006

  Bigtable was released in 2006 and inspired countless NoSQL databases, such as Cassandra, HBase, etc. Half of Cassandra's architecture imitates Bigtable, including data model, SSTables, and write-ahead logs (the other half imitates Amazon's Dynamo database, using a peer-to-peer cluster model).