How to Predict the Future in Uncertain Times

What do we mean when we talk about prediction?

There is an ongoing discussion about the meaning we assign to the words “prediction” and “forecast” among economists and scientists. Their major concern seems to be our ability to “predict” extra-ordinary events. But what is an extra-ordinary event, and if it’s extraordinary, can’t we assume that by its very nature it cannot be predicted? So why bother?

The problem is that the very definitions of “prediction” and “predictability” have been heavily misunderstood and, as a consequence, we also misunderstand the definition of an extra-ordinary event. Referring to ‘Black Swans’ and ‘fat tails’ can be misleading.

What we can predict – process behavior

There is indeed a realm where predicting things is not only possible but crucial. Thanks to the work of Walter A. Shewhart (and indeed Dr. W. Edwards Deming), we can measure and predict processes. Why should that be of interest to anyone? Because every repeated activity in an organization is a process. If we can predict the behaviour of our processes we can predict what will happen in our company. It’s as simple as that. Conversely, if we don’t achieve profound knowledge of our processes then we don’t really know what’s going on. That is a very worrying prospect.

Predictability in management concerns the behavior of processes. Thanks to the development of Statistical Process Control, we can measure a process and how it oscillates between its own upper and lower limits.

The upper and lower limits are “intrinsic” to the process, and are calculated – NOT imposed like specifications – from the process behavior itself. It is the intrinsic (natural) oscillation of the process that determines the limits. The calculation of the limits, based on a statistical formula, is very straightforward.

The first step is to define what a process is. We define a process as a set of activities that repeat themselves over time according to a given procedure.

Here the concept of time is extremely relevant. What we are going to investigate is something that changes over time, i.e. something that is dynamic and not static. In order to understand our reality we don’t need a snapshot and we don’t need to  shoot a short “movie” of it. What we want to measure is the variation that affects our process.

How to measure the behavior of a process

Let’s assume we know the algorithm for calculating the limits of our process. The mathematics behind their calculation is 100 years old and can be found in any book about Statistical Process Control as well as Wikipedia.

We use the first 20-25 data points of a series to calculate the limits, and we set the limits accordingly. Then, we monitor the unfolding of the process and we study the trend. What do we do with the chart we obtain? How do we use it to make “predictions” about the process? The rules are simple. They are empirical rules, based on statistical and economical considerations, but NOT dictated by any statistical prejudice.

Statistical considerations means we use statistics to calculate the limits, and economical considerations means we set the rules so that we don’t waste time and money investigating something that is irrelevant. This is because we are interested in achieving knowledge that gives us a real, practical advantage.

In a subsequent post we will look in detail at the tool for measuring the behavior of a process and how this can help us with the Quality of what we do.

Dr. Giovanni Siepe