Introduction

What You Need to Know about Forecasting

Everybody forecasts-it is an essential part of our lives. Predicting future outcomes is critical for success in everything from investing to careers to marriage. No one always makes the right choices, but we all strive to come close. This book shows you how to improve your decision-making by understanding how and why forecasters succeed-and sometimes fail-in their efforts. We're all familiar with economists' supposed ineptitude as prognosticators, but those who have been successful have lessons to teach us all.

I have been fortunate to have had a long and successful career in the field of economic forecasting, first at the Federal Reserve Bank of New York and the Bank for International Settlements, and then, for the majority of my working life, on Wall Street. Often I am asked about so-called tricks of the trade, of which there are many. People want to know my strategies and tactics for assembling effective forecasts and for convincing clients to trust me, even though no one's forecasts, including my own, are right all of the time. But most often, people ask me to tell them what they need to know in simple and accessible language. They want actionable information without having to wade through dense math, mounds of complicated data, or "inside-baseball" verbiage.

With that need in mind, Inside the Crystal Ball aims to help improve anyone's ability to forecast. It's designed to increase every reader's ability to make and communicate advice about the future to clients, bosses, colleagues, and anyone else whom we need to convince or whom we want to retain as a loyal listener. As such, this book shows you how to evaluate advice about the future more effectively. Its focus on the nonmathematical, judgmental element of forecasting is an ideal practitioners' supplement to standard statistical forecasting texts.

Forecasting in the worlds of business, marketing, and finance often hinges on assumptions about the U.S. economy and U.S. interest rates. Successful business forecasters, therefore, must have a solid understanding of the way the U.S. economy works. And as economic forecasts are a critical input for just about all others, delving deeper into this discipline can improve the quality of predictions in fields such as business planning, marketing, finance, and investments.

In U.S. universities, economics courses have long been among the most popular elective classes of study. However, there is an inevitable division of labor between academicians, who advance theoretical and empirical economic research, and practitioners.

My professional experience incorporates some of the most significant economic events of the past 40 years. I've "been there, done that" in good times and in bad, in stable environments and in volatile ones. One of the most valuable lessons I learned is that there is no substitute for real-world experience. Experience gives one the ability to address recurring forecasting problems and a history to draw on in making new predictions. And although practice does not make perfect, experienced forecasters generally have more accurate forecasting records than their less seasoned colleagues.

In my career, I have witnessed many forecasting victories and blunders, each of which had a huge impact on the U.S. economy. Every decade saw its own particular conditions-its own forecasting challenges. These events provide more than historical anecdotes: They offer fundamental lessons in forecasting.

At the start of my career as a Wall Street forecaster, I struggled, but I became much better over time. According to a study of interest rate forecasters published by the Wall Street Journal in 1993, I ranked second in accuracy among 34 bond-rate forecasters for the decade of the 1980s.1 MarketWatch, in 2004, 2006, and again in 2008 ranked me and my colleague James O'Sullivan as the most accurate forecasters of week-ahead economic data. In the autumn of 2011, Bloomberg News cited my team at UBS as the most accurate forecasters across a broad range of economic data over a two-year period.2 Earning these accolades has been a long and exciting journey.

When I first peered into the crystal ball of forecasting I found cracks. I had joined the forecasting team in the Business Conditions Division at the Federal Reserve Bank of New York in 1973-just in time to be an eyewitness to what would become, then, the worst recession since the Great Depression. As the team's rookie, I did not get to choose my assignment, and I was handed the most difficult economic variable to forecast: inventories. It was a trial by fire as I struggled to build models of the most slippery of economic statistics. But it turned out to be a truly great learning experience. Mastering the mechanics of the business cycle is one of the most important steps in forecasting it-in any economy.

A key lesson to be learned from the failures of past forecasters is to avoid being a general fighting the last war. Fed officials were so chastened by their failure to foresee the severity of the 1973-1975 recession and the associated postwar high in the unemployment rate that they determined to do whatever was necessary not to repeat that mistake. But in seeking to avoid it, they allowed real (inflation-adjusted) interest rates to stay too low for too long, thus opening the door to runaway inflation. My ringside seat to this second forecasting fiasco of the 1970s taught me that past mistakes can definitely distort one's view of the future.

By the 1980s, economists knew that the interest-rate fever in the bond market would break when rates rose enough to whack inflation. But hardly anyone knew the "magic rate" at which that would occur. With both interest rates and inflation well above past postwar experience, history was not very helpful. That is, unless the forecaster could start to understand the likely analytics of a high inflation economy-a topic to be discussed in later chapters.

The 1990s started with a credit crunch, which again caught the Fed off guard. A group of U.S. senators, who had been pestered by credit-starved constituents, were forced to pester then-Fed Chair Alan Greenspan to belatedly recognize just how restrictive credit had become.3,4 That episode taught forecasters how to evaluate the Fed's quarterly Senior Loan Officer Opinion Survey more astutely. Today the Survey remains an underappreciated leading indicator, as we discuss in Chapter 9.

The economy improved as the decade progressed. In fact, growth became so strong that many economists wanted the Fed to tighten monetary policy to head off the possibility of higher inflation in the future. In the ensuing debate about the economy's so-called speed limit, a key issue was productivity growth. Fed Chair Greenspan this time correctly foresaw that a faster pace of technological change and innovation was enhancing productivity growth, even if the government's own statisticians had difficulty capturing it in their official measurements. Out of this episode came some important lessons on what to do when the measurement of a critical causal variable is in question.

A forecasting success story for most economists was to resist becoming involved in the public's angst over Y2K: the fearful anticipation that on January 1, 2000, the world's computers, programmed with two-digit dates, would not be able to understand that we were in a new century and would no longer function. Throughout 1999, in fact, pundits issued ever more dire warnings that, because of this danger, the global economy could grind to a halt even before the New Year's bells stopped ringing. Most economic forecasters, though, better understood the adaptability of businesses to such an unusual challenge. We revisit this experience later, to draw lessons on seeing through media hype and maintaining a rational perspective on what really makes businesses adapt.

Forecasters did not do well in anticipating the mild recession that began in 2001. The tech boom, which helped fuel growth at the end of the previous decade and made Alan Greenspan appear very astute in his predictions on productivity, also set the stage for a capital expenditure (capex) recession. Most economists became so enthralled with the productivity benefits of the tech boom that they lost sight of the inevitable negative consequences of overinvestment in initially very productive fields.

Perhaps the largest of all forecasting blunders was the failure to foresee the U.S. home price collapse that began in 2007. It set into motion forces culminating in the worst recession since the Great Depression-the Great Recession. Such an error merits further consideration in Chapter 4, focusing on specific episodes in which forecasters failed.

By now, it should be clear that experience counts-both for the historical perspective it confers and for having addressed repetitive problems, successfully, over a number of decades. In reading this book, you will live my four decades of experience and learn to apply my hard-learned lessons to your own forecasting.

The book begins by assessing why some forecasters are more reliable than others. I then present my approach to both the statistical and judgmental aspects of forecasting. Subsequent chapters are focused on some long-standing forecasting challenges (e.g., reliance on government information, shifting business "animal spirits," and fickle consumers)...