Vol. 28, Issue 2
Summer 2009

HISTORY TO DEMAND-DRIVEN FORECASTING
By Larry Lapide, Ph.D.

(This is an ongoing column in The Journal of Business Forecasting, which is intended to give a brief view on a potential topic of interest to practitioners of business forecasting. Suggestions on topics that you would like to see covered should be sent via e-mail by clicking here:IBF)

In early Spring I wrote the foreword to Charlie Chase’ forthcoming book, Demand-Driven Forecasting (John Wiley & Sons). It was an honor to be asked to do it, as I’ve known Charlie since the early 1990s and consider him not only a well-respected colleague, but also a close friend. In fact I owe him a great deal because he introduced me to the Institute of Business Forecasters organization, the pre-eminent business forecasting organization that I’ve been affiliated with ever since.

This column is partly based on what I wrote in the book’s foreword. As I delved into the writing, it drove me to the conclusion that the biggest change I’ve seen in business forecasting over the past few decades has been a movement from history-based forecasting to demand-driven forecasting. That is, a trend from forecasting largely based on analyzing historical data to forecasting that also incorporates the impact of demand-shaping activities, such as sales and marketing promotions, on future demand. Charlie’s book discusses the traditional time-series methods as well as the more recent demand-driven ones. To prepare for the foreword I needed to recollect my own forecasting experiences as well as those of others I have interacted with in the industry.

UNDERSTANDING UNCERTAINTY

In my inaugural JBF column, Developments In Demand Forecasting From Ancients Greeks To Present (Fall 1997), I discussed one of my favorite books—the late Peter L. Bernstein’s book titled Against the Gods, which chronicles the advances in the understanding and the modeling of uncertainty. Paraphrased from what I wrote then, the book’s premise is that man’s view on uncertainty (relevant to the forecasting of future events, such as customer demand) follows an evolutionary path that started from the Ancient Greek belief that one did not have to forecast because the gods (at times whimsically) controlled the future by dictating what would happen.

During the Renaissance the philosophy changed. People started to understand uncertainty and probabilistic events, driven by a belief that precise probabilistic behaviors or models were put in place by Nature. In the latter part of the 1900s, with the advent of theories such as game theory (e.g., the Nash Equilibrium), it became more apparent that people control much of what might happen in the future. As such, understanding the rational person would allow accurate predictions of the future.

Since the 1950s, demand forecasting has followed an evolutionary path similar to the one Bernstein described for understanding uncertainty.

Dr. Lapide is a lecturer in the college of management of the University of Massachusetts, Boston campus. He has extensive experience in industry, consulting, business research, and academia, and has a broad range of forecasting experiences. He was an industry forecaster for many years, has led forecasting-related consulting projects for clients across a variety of industries, and has researched as well as taught forecasting. most recently he was a researcher at MIT and before that a market analyst researching forecasting and supply chain software.

A WALK DOWN MEMORY LANE

years was largely based on the exponential smoothing forecasting methods developed by an industry practitioner, Robert G. (Bob) Brown, who published several books starting in the late 1950s. (These exponential smoothing methods live on today and are often the under-the-hood statistical forecasting ”engines” that power many software packages). Forecasting methods then evolved to include a wide variety of what is termed statistical time-series methods, many of which were discussed in several revisions of forecasting books written by two leading academics forecasters, Spyros Makridakis and Steven C. Wheelwright, starting in the late 1970s.

Then more sophisticated methods were developed in an attempt to understand seasonal and trend variations, yet were still based on the belief that there is a recurring pattern to historical demand that, once understood, could be used to forecast the future. Analogous to the evolution in the understanding of uncertainty, these methods are, in effect, attempts to describe the probabilistic demand behaviors or models that Nature has put in place, in order to forecast future demand. The methods include various versions of weighted-averaging methods—such as Winter’s—and sophisticated ones—such as the Fourier series, which accounts for seasonal and trend variations.

Thus, during the first half of my career, advanced methods focused on what might be termed history-driven forecasting, because the methods involved mostly analyzing years of historical data to project the future. Midway in my career, the focus began to shift toward demand-driven forecasting.

Until the recent economic meltdown, the past few decades has been marked as a period of increased consumerism, especially in the United States, during which marketing and sales organizations developed more sophisticated and effective ways to simulate (i.e., shape) demand for the products they were promoting. Industry forecasters, by necessity, started to experiment with and utilize methods that no longer assumed that demand just mystically happened and thus could only be estimated from understanding what happened in the past. In fact, people (namely their company’s sales and marketing managers and not just Nature) were shaping demand through marketing promotions and new product introductions. The forecasters started leveraging cause-effect methods— such as multiple regression methods and time-series methods incorporating causal factors. One type—ARIMA (Autoregressive Integrated Moving-Average) models with explanatory variables— were used, for example, to reflect the fact that promotional activities would shape and create demand, and therefore needed to be understood and incorporated into a forecast.

The rise in consumerism made the business forecaster’s job much more difficult. Demand forecasting methods and systems have had to become larger in scale to accommodate the dramatic growth in the entities that needed to be forecast in multinational organizations. Business planning has become more complex in terms of having to deal with the myriad products being sold, many with short lifecycles (e.g., known as Stock-Keeping Unit—SKU—proliferation) and further complicated by an increase in the number of countries into which they are sold, as well as the number of channels they are sold through.

Technology has been evolving to keep up with this dramatic growth in scale. In the early days, proprietary computer systems were developed around using moving-average and exponential smoothing methods to forecast a limited numbers of SKUs, because these time-series methods did not require analyzing a multitude of years of demand history. They just required using the past several periods of demand and the most recent forecast, so they did not need to use much of the scarce computer memory resident in these less-powerful proprietary systems.

In contrast, most forecasters today, after having blown out of their spreadsheets from increasing scale, can use commercial software applications that can accommodate the forecasting of hundreds of thousands of SKUs, while analyzing multiple years of history with a wide variety of forecasting methods. In addition, these high-powered systems are driven off large data bases of historical demand, downstream demand signals (e.g., POS data), and forecasts, from which managers using On-Line Analytical Processing (OLAP) “engines” can generate, review, and revise aggregated and disaggregated forecasts in order to incorporate the vast amounts of market intelligence information needed for demand-driven forecasting.

In summary, I’ve seen many changes in business forecasting during my career, and without attempting to plug Charlie’s book too much, I believe he captures a lot of what industry forecasters (like yourself) have been doing over time to improve forecasting by understanding the impact of demand-shaping activities. Much as a practitioner, Bob Brown described history-based forecasting over 50 years ago, Charlie, another industry practitioner, has done the same for today’s demand-driven forecasting.