Analytics is all about using data to answer questions, however how we approach doing this is very much dependent upon the complexity of the system we are dealing with and the kind of questions we want to ask. Analytics has been used within organizations in a formal way for over a century now, being pioneered by Frederick Taylor in his analysis of work processes and Henry Ford’s measuring of his newly established assembly line. However, this is a relatively basic form of analytics where we are studying a very limited number of components and processes in a somewhat linear fashion. This linear approach has largely dominated business intelligence until quite recently, but things have just got a lot more complex and our approach to analytics is likewise changing to become a lot more sophisticated, and this is now what we call advanced analytics or complex analytics. Whereas with more basic approaches to analytics we are asking relatively straightforward questions about a relatively simple system that is limited in scope. With complex analytics, we can try to answer complex questions or another way of saying that is that it is analytics for complex systems.
To give this context and relevance we can note how our world has just got greatly more complex along almost every dimension over the past decades. Whether we are talking about, telecommunications, transport, international politics, supply chain management or the media landscape, the number of nodes and the degree of connectivity has greatly increased, in many cases, such as with financial markets, it is orders of magnitude greater than it was prior to the 90s. This new level of complexity that has emerged within virtually all of our systems of organization has major implications for how we should approach analytics.
Living in a small town you could simply walk around and talk to people to find out what was going on. As we formed larger organizations during the industrial age new tools of communications emerge, the telegraph, telephone, and postal system enable people to communicate effectively within large organizations. Television and newspaper informed people of what was going on in their region, city or nation. But today we find ourselves embedded within vast complex systems that often span around the planet. Corporate supply chains, huge social networks, sprawling metropolitan areas, global air transport systems, the global biosphere, financial markets these are things that shape all our lives, but somehow we don’t have the means, the vocabulary or the methods to actually grasp them in their complexity, to see them, to really know what is going on. As a consequence of not being able to see the workings of these systems – and thus in some way manage them – we get financial crises, we get environmental crises, we get violent social outburst because we can’t see the mounting tensions, large corporations drop off the S&P 500 faster and faster because they can’t see through the complexity and respond fast enough.
In these increasingly complex environments that organizations operate in today traditional conceptions of linear cause and effect break down, things become ambiguous and simply left without interpretation, we find out selves in a reactionary state, continuously surprised and shocked, volatility increases, we start to see only some of the trees and no longer the forest, we become incapable of acting decisively, knowing what to do and making important long-term decisions and investments. As the complexity proliferates we are burying the proverbial needle in the haystack. More data means more noise; we stop being able to hear the things that we need to be able to hear. This is the same problem for business, for individuals, for researchers, for policymakers for everyone living in this world of globalization and information technology.
So how do we make sense of a complex world? We need new models and new ways of looking at the world, but just as importantly we need new tools and methods for amassing and processing data and information. This is what this course is about, these new data sources and tools to support a more comprehensive understanding of the complex systems that we are now challenged to try and manage. Whereas traditional business analytics of the past has been based upon well defined and well-structured data sets that were limited in size and complexity, today we have a wealth of new data coming from a myriad of new sources and we call this big data. Whereas the data of the past was structured into specific vertical categories being used to answer specific questions. This stream of unstructured data from a multiplicity of sources enables us to create context by making connections between a diversity of data.
We no longer just have data about what products we sold and look to see how changes in price changed sales, but we now have a massive amount of data from different sources that can be used to find much more complex patterns and correlations. Data about customers, about locations, about all the other products available, specificities about the store where an item was sold or the time of day etc. and all of this can be put together in new ways to find new patterns that were previously hidden. Whereas in the past if we wanted to know where a fire might break out in a city or an accident on a highway happen, we were inclined to look at that phenomenon itself, but with complex analytics, insight may come from a completely different realm that has nothing to do with that actual activity. With analytics, we look simply at the event, with complex analytics we can now look at a network of data points to create some kind of context to the phenomenon. So that we are no longer dependent upon simplified mechanistic cause and effect descriptions but we can begin to look at things in a more realistic fashion as a network of interacting factors.
John Kelly of IBM talks about this change as such “What’s different now and has changed is it’s no longer about taking this data putting it into a computer running a calculation and getting a balance sheet answer… what’s important now is what is the context of the data, what is it connected to, what effect is it having on data around it… it’s basically a network of the data, it’s no longer sort of tabular columns, of rows, of data, it’s interconnected patterns.” Just as data is no longer a single thing, so to the means with which we process it is changing. In the past, we used mechanistic rules, like formulas in a spreadsheet that were dependent upon strict well-defined datasets as input. Today we are moving from rule-based mechanistic algorithms, where all components are prespecified and well-defined to computational graphs, which are networks of nodes, that learn through self-organization. Complex analysis gives computers the capacity to understand data in a new way like humans do, this means that computation can now act on data derived directly from unstructured contexts and real-world environments.
One of the best examples of complex data analytics is web search. Web search was one of the first widely used application of big data and advanced analytics. A search engine like Google’s looks through massive amounts of data and within seconds analyzes it in a multiplicity of ways. Web search has given us the capacity to look into big data, to look into all this information that we have and we can think about the effects of that on transport, on research, on almost every area which now depends upon this big data of the internet and complex analytics in the form of search engines.
In this respect, people often equate our newly found technological capacities of complex data analytics to that of the microscope or telescope. Jay Walker of Tedmed describes this revolution well when he says “the microscope in the 1650s and 60s opened up the invisible world and we for the first time we’re seeing cells and bacteria and creatures that we couldn’t imagine were there. It then happened again when we reveal the atomic world… but now there’s actually a super visible world coming into play, ironically big data is a microscope, we’re now collecting exabytes and petabytes of data and we’re looking through this microscope using incredibly powerful algorithms to see what we could never see.”
With these new technologies that we will be talking about in this course, it is like we are building a new kind of instrument like we are building the telescope for complex systems. For the first time, we are able to look at these complex systems that are all around us which have a structure, a pattern and even a beauty that are invisible without the right instruments. The implications of this are huge, in terms of how we understand the world and our place within it and of course how we make decisions and act.