In 2018, a group of organizations from all of the world will begin construction of the largest radio telescope ever built, the Square Kilometre Array (SKA). With one million square meters of collecting area and enough optical fiber to wrap around the Earth twice, this marvel of modern engineering will be sensitive enough to detect airport radar on a planet 50 light years away.
SKA will also generate 700 terabytes of data every second, equivalent to roughly 35 times the data stored in the Library of Congress. At full capacity, the SKA’s aperture arrays are expected to produce 100 times more data than the entire Internet. It doesn’t take a rocket scientist to realize that such a deluge of information creates a big data problem, perhaps the biggest we have ever encountered.