The ingenuity problem: What big data really means for big business
Big data presents a unique problem for big business that intelligence, as we now understand it, cannot solve alone. Big data gives rise to mega-problems - problems whose complexity grows much faster than the size of the data set. Solving big data problems requires the development of entirely new methods. Below, Mihnya Molodovianu explains that what we really need is ingenuity-the ability to create new procedures to solve the Mega Problems that Big Data generates.
"Big data" is vaunted as the next "mega-problem" for business, and rightly so: business is busy making lots of decisions based on scarce, noisy information. Better decisions depend on having more accurate, relevant and timely information. Having the information we need to make decisions in the form we can best use is a particular challenge, even in an era where the capacity to store, collect and process data doubles every 18 months to 2 years. Why?
The Big Data Challenge
The "big data" megaproblem is not just a technical problem: not just a problem of optimal logical inference, or storage, or communication, or CPU design. To see for yourself, conduct a mental experiment by multiplying the storage capacity of all your computing machines, including your smartphones, and the bandwidth of all your communications media by a factor of 10. Will your big data problem disappear? No. It will get worse: you will have more data, but maybe less useful information. In fact, advances in information processing can exacerbate the problem of big data: they give us access to more and more volatile data. information, as innovations at the level of new devices and architecture provide "instant access". The challenge of big data is this: how do we efficiently, effectively and reliably transform relevant data into useful information?
For example:
- In the next decade, the developing world presents tremendous opportunities for Western companies' products and services. But to win what https://newexness.com/login/ called the trillion-dollar decathlon, companies need to predict the specific preferences, tastes and behavioral patterns of a large, diverse and entirely new group of core users. How should they do this?
- The Internet turns every connected user into a network node of links to other users, ideas, products, technologies, philosophies and lifestyles and behaviors, all of which can be tracked in real time. It is a hive of data, ready for intelligent analysis - provided we can make sense of this widely distributed and interactive information base: what should we focus our analytical tools on first?
Big data creates mega-problems: lots of variables, lots of relationships between them, all of which require lots of operations to produce a solution.
- Each of the 7 billion people can be thought of as a constant process of data generation. Their shopping choices and lifestyles, moods, physiological signs and signals all generate data that can help us understand and predict their behavior to help them with health emergencies and respond to their problems in real time. What should we measure, how should we measure it, when should we measure it, and how should we optimally aggregate, integrate, and combine these measurements into predictions of behavior?