Data-Driven Decision Making
When Hurricane Katrina was about to hit the coast of the United States a large retailer did a study to prepare themselves by asking what products they might sell out of and what they should stock up on. A roomful of intelligent and experienced executives thought through what those products might be and came up with reasonable answers such as flashlights, batteries, water, canned foods, sandbags and more, but when they ran the data and analytics the number one product was, in fact, Budweiser beer. This is the power of data to illuminate insight, to take us beyond intuition and help us make a data empowered decision and it has relevance for almost everything we do.
More and more of our actions and interactions with the world are becoming mediated by data. This alters how we interact and the choices we make. Understanding and seeing data can completely change the ranking of a set of options available to us and hence how we allocate resources, both as individuals and collectively. Almost everything can be tested, measured and improved and this is truly bringing about a quiet but fundamental cultural transformation in how we make decisions. Datafication brings about a more objective form of decision making, what is called data-driven decision making. For example, when it comes to choosing a movie we used to go to the store and pick up the movie, browse through all the titles, read the description and decide if we want to see it. Now we are confronted with algorithms that make recommendations based on the data from the past films that you have seen, as well as who your friends are, what films they have seen and liked and the aggregation of feedback from thousands or millions of other people.
Madeline McIntosh from a book publishing house talks about how the culture of publishing change with the arrival of Amazon’s data-driven approach. The traditional culture of publishing was what she called a culture of lunches, culture of conversations where people had hunches and ideas about books and they discussed them. Amazon then brought a data-driven, numbers and math-driven approach to this and was able to basically figure out much better what was working and what wasn’t working, with the result being that they basically took over the market. This transformation is happening in many areas of our economy, more traditional companies are being displaced by companies that have embraced this new technology and cultural paradigm of data.
Take wine tasting for example which is maybe the quintessential human skill, there are human experts who look at and smell the wine to tell you what it tastes like if it is of good quality. This is a highly refined skill and sensory ability but it’s also true that wine is at the end of the day just a certain molecular composition and you can analyze it with numbers. The wine analytics company Enolytics have been able to figure out that you can predict how an expert will rate it before they even taste the wine with remarkable accuracy. Of course, this applies to more and more spheres of life, Wall Street is no longer full of people on seats making trades based on intuition and hope, but up to 70% of those decisions are now made by algorithms acting on data. Likewise, decisions on healthcare diagnostics are increasingly made by analytical systems acting on data, sports decisions are based on big data extracted from cameras and sensors in the shirts of players etc. The implicit premise of big data is that decisions can be made wholly based upon data and computerized models, shifting the locus of decision making from people and intuition to data and formal models.
EMC big data guru Bill Schmarzo describes well how decisions are currently made based on gut feeling1 “One of the most critical aspects of big data is its impact on how decisions are made and who gets to make them. When data are scarce, expensive to obtain, or not available in digital form, it makes sense to let well-placed people make decisions, which they do on the basis of experience they’ve built up and patterns and relationships they’ve observed and internalized. ‘Intuition’ is the label given to this style of inference and decision-making. People state their opinions about what the future holds—what’s going to happen, how well something will work, and so on—and then plan accordingly.” The term HiPPO is an acronym now used to describe this typical corporate decision-making process, where the highest-paid person in the room gets to make the final call. Much of our approach to decision making has been a function of simply not having data and not knowing. In the past, we have had to make decisions about complex environments and complex systems without being able to see or know what they were really like, just based on some intuition. But big data analytics offers this new telescope with which to actually see these systems and the difference between having a hunch and actually seeing the data can be huge in terms of the actual decisions that get made.
Every minute, the world loses an area of forest the size of 48 football fields. And deforestation in the Amazon Basin accounts for the largest share, contributing to reduced biodiversity, habitat loss, climate change, and other devastating effects. But better data about the location of deforestation and human encroachment on forests could help governments and local stakeholders respond more quickly and effectively. A project called Planet2 is currently developing the world’s largest constellation of Earth-imaging satellites, it will soon be collecting daily imagery of the entire land surface of the earth at 3-5 meter resolution. While considerable research has been devoted to tracking changes in forests, it typically depends on coarse-resolution imagery. Furthermore, these existing methods generally cannot differentiate between human causes of forest loss and natural causes. Planet are challenging the analytics community to develop machine learning models for labeling satellite image clips with atmospheric conditions and various classes of land cover and land usage types. Resulting algorithms will help to better understand where, how, and why deforestation happens all over the world. A much clear image of this complex system would enable action-oriented decisions to be taken.
1. Harvard Business Review. (2012). Big Data: The Management Revolution. [online] Available at: https://hbr.org/2012/10/big-data-the-management-revolution [Accessed 9 Feb. 2018].
2. Kaggle.com. (2018). Planet: Understanding the Amazon from Space | Kaggle. [online] Available at: https://www.kaggle.com/c/planet-understanding-the-amazon-from-space [Accessed 9 Feb. 2018].