1
Stationary Processes and Time Series

1.1 Introduction

Forecasting the evolution of a man-made system or a natural phenomenon is one of the most ancient problems of human kind. We develop here a prediction theory under the assumption that the variable under study can be considered as stationary process. The theory is easy to understand and simple to apply. Moreover, it lends itself to various generalizations, enabling to deal with nonstationary signals.

The organization is as follows. After an introduction to the prediction problem (Section 1.2), we concisely review the notions of random variable, random vector, and random (or stochastic) process in Sections 1.3-1.5, respectively. This leads to the definition of white process (Section 1.6), a key notion in the subsequent developments. The readers who are familiar with random concepts can skip Sections 1.3-1.5.

Then we introduce the moving average () process and the autoregressive () process (Sections 1.7 and 1.8). By combining them, we come to the family of autoregressive and moving average () processes (Section 1.10). This is the family of stationary processes we focus on in this volume.

For such processes, in Chapter 3, we develop a prediction theory, thanks to which we can easily work out the optimal forecast given the model.

In our presentation, we make use of elementary concepts of linear dynamical systems such as transfer functions, poles, and zeros; the readers who are not familiar with such topics are cordially invited to first study Appendix A.

1.2 The Prediction Problem

Consider a real variable depending on discrete time . The variable is observed over the interval . The problem is to predict the value that will take the subsequent sample .

Various prediction rules may be conceived, providing a guess for based on . A generic predictor is denoted with the symbol :

The question is how to choose function .

A possibility is to consider only a bunch of recent data, say , , , , and to construct the prediction as a linear combination of them with real coefficients , , ., :

The problem then becomes that of selecting the integer and the most appropriate values for parameter , , ., .

Suppose for a moment that and were selected. Then the prediction rule is fully specified and it can be applied to the past time points for which data are available to evaluate the prediction error:

Let's now consider this fundamental question: Which characteristics should the prediction error exhibit in order to conclude that we have constructed a "good predictor"? In principle, the best one can hope for is that the prediction error be null at any time point. However, in practice, this is Utopian. Hence, we have to investigate the properties that a non-null should exhibit in order to conclude that the prediction is fair.

For the sake of illustration, consider the case when has the time evolution shown in Figure 1.1a. As can be seen, the mean value of is nonzero. Correspondingly, the rule

would be better than the original one. Indeed, with the new rule of prediction, one can get rid of the systematic error.

Figure 1.1 Possible diagrams of the prediction error.

As a second option, consider the case when the prediction error is given by the diagram of Figure 1.1b. Then the mean value is zero. However, the sign of changes at each instant; precisely, for even and for odd. Hence, even in such a case, a better prediction rule than the initial one can be conceived. Indeed, one can formulate the new rule:

and

From these simple remarks, one can conclude that the best predictor should have the following property: besides a zero mean value, the prediction error should have no regularity, rather it should be fully unpredictable. In this way, the model captures the whole dynamic hidden in data and no useful information remains unveiled in the residual error, and no better predictor can be conceived. The intuitive concept of "unpredictable signal" has been formalized in the twentieth century, leading to the notion of white noise () or white process, a concept we precisely introduce later in this chapter. For the moment, it is important to bear in mind the following conclusion: A prediction rule is appropriate if the corresponding prediction error is a white process.

In this connection, we make the following interesting observation. Assume that is indeed a white noise, then

Rewrite this difference equation by means of the delay operator , namely the operator such that

Then

from which

or

with

By reinterpreting as the complex variable, this relationship becomes the expression of a dynamical system with transfer function (from to ) given by .

Summing up, finding a good predictor is equivalent to determining a model supplying the given sequence of data as the output of a dynamical system fed by white noise (Figure 1.2).

Figure 1.2 Interpreting a sequence of data as the output of a dynamic model fed by white noise.

This is why studying dynamical systems having a white noise at the input is a main preliminary step toward the study of prediction theory.

The road we follow toward this objective relies first on the definition of white noise, which we pursue in four stages: random variable random vector stochastic process white noise.

1.3 Random Variable

A random (or stochastic) variable is a real variable that depends upon the outcome of a random experiment. For example, the variable taking the value or depending on the result of the tossing of a coin is a random variable.

The outcome of the random experiment is denoted by ; hence, a random variable is a function of : .

For our purposes, a random variable is described by means of its mean value (or expected value) and its variance, which we will denote by and , respectively.

The mean value is the real number around which the values taken by the variable fluctuate. Note that, given two random variables, and with mean values and , the random variable

obtained as a linear combination of and via the real numbers and , has a mean value:

The variance captures the intensity of fluctuations around the mean value. To be precise, it is defined as

where denotes the mean value of . Obviously, being non-negative, the variance is a real non-negative number.

Often, the variance is denoted with symbols such as or . When one deals with various random variables, the variance of the th variable may be denoted as or .

The square root of the variance is called standard deviation, denoted by or . If the random variable has a Gaussian distribution, then the mean value and the variance define completely the probability distribution of the variable. In particular, if a random variable is Gaussian, the probability that it takes value in the interval and is about . So if is Gaussian with mean value 10 and variance 100, then, in cases, the values taken by range from to .

1.4 Random Vector

A random (or stochastic) vector is a vector whose elements are random variables. We focus for simplicity on the bi-dimensional case, namely, given two random variables and ,

is a random vector (of dimension 2). The mean value of a random vector is defined as the vector of real numbers constituted by the mean values of the elements of the vector. Thus,

where and are the mean values of and , respectively. The variance is a matrix given by

where

Here, besides variances and of the single random variables, the so-called "cross-variance" between and , , and "cross-variance" between and , , appear. Obviously, , so that is a symmetric matrix.

It is easy to verify that the variance matrix can also be written in the form

where ´ denotes transpose.

In general, for a vector of any dimension, the variance matrix is given by

where is the vector whose elements are the mean values of the random variables entering .

If is a vector with entries, is a matrix. In any case, is a symmetric matrix having the variances of the single variables composing vector along the diagonal and all cross-variances as off-diagonal terms.

A remarkable feature of a variance matrix is that it is a positive semi-definite matrix.

Remark 1.1 (Positive semi-definiteness)

The notions of positive semi-definite and positive definite matrix are explained in Appendix B. In a very concise way, given a real symmetric matrix , associate to it the scalar function defined as , where is an -dimensional real vector. For example, if

we take

Then

Hence, is quadratic in the entries of vector . Matrix is said to be

• positive semi-definite if ,
• positive definite if it is positive semi-definite and only for

We write and to denote a positive semi-definite...