Slaying the white whale on a stormy sea of big data, for most companies is like the fictional story of Captain Ahab’s Moby Dick. The term Big Data has suddenly become en vogue. And like its sister term “The Cloud” – these all encompasing terms are a mystery and elusive to everyone. For businessman and executives, its the holy grail. For the technical experts and data scientists its the next big project with hyped expectations. So what is big data? If you want wikipedia’s answer – click here. But here’s my simple explanation.
The old ways of dealing with data do not work. Exponential growth of applications, smart devices and just about anything with a sensor or computer chip creates “Data” — a whole ocean’s worth! While the ocean of data is getting bigger, Moby Dick by comparison is getting smaller. And that’s the real problem; a real “Big” problem. You don’t need to be a whaler to understand that means we need to get better at spotting whale sign. We need better sonar and radar (pattern recognition systems) and we need faster boats to cover more ocean (and process more data).
In a very real sense, Big Data is anything that is trying to harpoon Moby Dick.