Heuristics in Analytics

Preface

Suppose you have a magical black box. You put a huge amount of data into it, from everywhere you can find. And say that each of these inputs are in different formats and each format, in turn, represents several distinct time periods, frequencies, and seasons, as well as different characteristics. Imagine that you can then push a button and voilà! Out pops new knowledge from that hodgepodge of input. You now know the likelihood that the next event will occur, you can predict behavior; you have quantified risk, the propensity to act, and well-defined emerging patterns. And all this magically produced output would be ready to use—in the format needed to take action. This is amazing, isn’t? You just fill some box with lots of different stuff and get results that are easy to understand and that can immediately be put into use.

That black box is analytics. And it isn’t a brand-new discipline but is heavily used in most industries these days. Analytics combines mathematics, statistics, probability, and heuristics theories. So now, after this brief description, we all understand that analytics is easy, right?

Mathematical disciplines, including statistics, probability, and others, have always been assigned to practical applications in some form or another, including those of business. Sometimes the relationship between the method and the application is quite clear, and sometimes it is not. As human beings, we have used trade since we started to relate to each other. Because of a need to interact we’d exchange goods. Mathematics was, and still is, a wonderful tool to support exchanges and business purposes—used to address issues from the simple to the complex.

The word mathematics comes from the Greek word máthema, which means learning, study, science. This word has always related to describing quantities, to depicting structures of distinct types and shapes, and to portraying spaces. Pattern recognition is forever an elusive topic for mathematicians. Recognizing patterns, whatever it is they represent and from whatever source they emerge, people hope to better understand the past, and as a rule, understand the past to foresee the future.

History (the study of past social and human perspectives) is often used as a way to understand former events, learn from them to explain the present, and possibly help understand and even predict the future. We document how societies, governments, groups of people, ghettos, all clusters of some sort, behave prior to a particular event. By understanding and describing these sorts of scenarios (like documenting them), historians try to explain events that took place to provide context of what happens in the future.

Historians and mathematicians are quite similar that way. For mathematicians, past events and their associated facts are gathered together and analyzed in order to describe some set of data. If the attributes describing those past events are not too large in terms of size or scope, a mental correlation might be done. This is what social and human researchers often do. Given some attributes along with a description of a particular scenario in relation to a past event, they correlate distinct aspects of the event itself to explain what happened. And, based on the description and the strength of correlation between event attributes associated with the past event, they also try to foresee what might happen next.

Although the methods may differ, this isn’t far removed from what mathematicians do. We may use different terms (in fact, that is one of the plaguing realities of every discipline), but we analysts do the same thing. Mathematics is just another way to map what human beings do, think, understand, act, explain, and so on. What is a mathematical formula in the end? It is a way to describe the future by understanding past observations (well, at least one branch of mathematics does this). We used to do this in the past, when mathematics was used to count things for trade. Today, we examine scenarios with specific conditions and particular constraints, calculating outcomes for use in operations and decisions.

As with any other discipline, mathematics has evolved over time. In addition to counting physical objects, early uses of mathematics involved quantifying time, defining values for trade (bartering of goods lead to the use of currency), measuring land, measuring goods, and many other areas. As time passed, arithmetic, algebra, and geometry were used for taxation and financial calculations, for building and construction, and for astronomy. As uses continued to evolve, so did mathematical theory. Mathematics became a tool to support different scientific disciplines such as physics, chemistry, and biology, among others. And it is likely that mathematics will be used as a tool for disciplines that haven’t even been defined yet.

In order to play this important role and support science, mathematics must be quite rigorous. One of the rigors of mathematics is proofs. Mathematical proof is a method that turns theorems into axioms by following a particular set of laws, rules, constraints, meanings, and reasons. The level of rigor needed to prove theorems has varied over time, changed in different cultures, and certainly extended to satisfy distinct scenarios for political, economic, and social applications.

But a crucial truth of mathematical proofs is that they are often simply a heuristic process. The procedures used to prove a particular theorem are often derived by trial and error. Heuristic characteristics might be present in the entire theorem scenario, or just at the inception. But in reality, heuristics is often involved in one or more stages of proof. Heuristics can support the definition of the proof from simple observation of an event (therefore helping to recognize a pattern), or heuristics may govern the entire mathematical proof by exception or induction.

Most mathematical models have some limits in relation to the set of equations that portray a particular scenario. These limits indicate that the equations work properly given specific conditions and particular constraints. Consider standard conditions for temperature and pressure in chemistry. Some formulas work pretty well if temperature and pressure are in a specific range of values, otherwise, they don’t. Is the formula wrong? Or could it be that at other times in history, the formula was valid, but not now? The answer is no, on both counts. It just means that this particular formula works pretty well under some specific conditions, but not others. Remember that formulas are built to model a particular event, based on a set of observations and, therefore, this formula will work fine when the constraints of the model scenario are true.

This is a perfect example of why mathematical models can describe a particular scenario using one or more equations. Although these equations may properly depict a specific scenario, the methods work properly if, and usually just if, a particular set of conditions are satisfied. There are boundary conditions limiting accuracy of equations, and therefore, a model simply represents some specific scenario. These boundary conditions are sometimes just a specific range of possible values assigned to constants or variables.

Think about physics. The classical mechanical physics of Isaac Newton describe regular movements considering regular bodies (not too small and not too big), traveling at regular speeds (not too slow and not too fast). However, once we start to consider very high speeds, such as the speed of light, the classical Newtonian theories no longer describe the movements. This doesn’t mean the formulas are wrong; it just means that the particular scenario Newton wished to describe using those formulas requires a particular set of condition and constraints. To describe regular bodies at very high velocity we need to use Einstein’s formulas. Einstein’s theory of relativity (and his corresponding formulas) describes the movement of regular bodies at very high speeds. His theory doesn’t conflict with Newton’s; it simply explains a different scenario. Eventually, very small bodies at very high speeds also needed a distinct theory to describe their movement, and quantum physics was born. As we continue to learn, and delve even more specifically into areas of physics (and the authors would argue, any discipline), the need for different methods will continue—a story like this one never ends.

And so, we are not that different from historians. As mathematicians, statisticians, data miners, data scientists, analysts (whatever name we call ourselves), we too take into consideration a set of attributes used to represent/describe a particular scenario, and analyze the available data to try to describe and explain patterns or predict (which is, as a matter of fact, a particular type of pattern).

Once upon a time, a mathematician named Edward Lorenz was quite focused on predicting weather. At that time, this sort of work was largely based on educated guesses and heuristics. Is it so different today? Maybe (guesses are still fundamental and the heuristic process is undoubtedly still present). Back in Edward Lorenz’s day, weather prediction included assumptions, observations, and lots of guesswork—in spite of the scientific instruments available at the time. When computers came onto the scene, Lorenz foresaw the possibility to combine mathematics with meteorology. He started to build a computer mathematical model, using differential equations, to forecast changes in temperature and pressure. Lorenz had created a dozen differential equations and managed to run some simulations and estimate virtual weather conditions. This...