1
Introduction

This book presents the theory of multidimensional (multiD) signals and systems, primarily in the context of image and video processing. MultiD signals are considered to be functions defined on some domain of dimension two or higher with values belonging to a set , the range. These values may represent the brightness or color of an image or some other type of measurement at each point in the domain. With this interpretation, a multiD signal is represented as

i.e. each element of the domain is mapped to the value belonging to the range . In conventional continuous-time one-dimensional (1D) signals and systems theory [Oppenheim and Willsky (1997)], and are both the set of real numbers , and so a real 1D signal would be written , . MultiD signals arise when the domain is a space with two or more dimensions. The domain can be continuous, as in the case of real-world still and time-varying images, or discrete, as in the case of sampled images. In addition, the range can also be a higher-dimensional space, for example the three-dimensional color space of human vision.

In this book, we are mainly concerned with examples from conventional still and time-varying images, although the theory has broader applicability. A conventional planar image is written , where lies in a planar region associated with the Euclidean space . Here, denotes the horizontal spatial position and is the vertical spatial position while denotes image brightness or color. The domain can be itself, or a discrete subset in the case of sampled images. Similarly, a conventional time-varying image is written , where lies in a subset (possibly discrete) of , which may also be written . Here, and are as above, and represents time. Higher-dimensional cases also exist, for example time-varying volumetric images with and where denotes some measurement taken at location at time . The domain can also be a more complicated manifold such as a cylinder or a sphere, as in panoramic imaging.

MultiD signal processing has been an active area of study for over fifty years. Early work was in optics and the continuous domain. Papoulis's classic text on Systems and Transforms with Applications in Optics appeared in 1968 [Papoulis (1968)]. Soon after, work on two-dimensional digital filtering started to appear, for example [Hu and Rabiner (1972)]. Over the years, there have been several books devoted to multiD digital signal processing and numerous books on image and video processing. The present book is distinguished from these works in a number of aspects. The book is mainly concerned with the theory of discrete-domain processing of real- or vector-valued multiD signals. The application examples are drawn from grayscale and color image processing and video processing. In particular, the book is not intended to present the state-of-the-art algorithms for particular image processing tasks.

Most previous books on multiD signals considered rectangularly sampled signals for the main development and presented non-rectangular sampling on a lattice as a subsidiary extension. A lattice, as in crystal lattice, is a mathematical structure from which we can construct more general sampling structures. In this book, the theory is developed on lattices from the beginning, and rectangular sampling is considered a special case. Another difference is that most books use normalized representations for non-rectangular sampling that are dependent on the lattice basis. Although this may be convenient for certain manipulations, this approach obscures much of the geometrical and scale information of the signal. We prefer to use basis-independent representations as much as possible, and introduce the basis where needed to perform calculations or implementations. Thus, we do not use such normalized representations but rather use the actual units of space-time and frequency.

Another distinguishing feature of this book is the treatment of color. Color signals are viewed as multiD signals with values in a vector space, in this case the vector space of human color vision, and color signal processing is viewed as vector-valued signal processing. Most multiD signal processing books deal mainly with scalar signals, representing a grayscale or brightness value. If color models are introduced, color signal processing generally involves separate processing of three color channels. Here we present the theory of multiD signal processing where the input and output are vector-valued signals, further developing the theory introduced in Dubois (2010).

In general, multiD signals in the real world, such as still and time-varying images, are functions of the continuous space and time variables. Consider for example a light signal falling on a camera sensor or emanating from a motion-picture screen. These multiD signals are converted to discrete-domain signals for digital processing, storage, and transmission. They may eventually be converted back to continuous-domain signals, for example for viewing on a display device. Thus, we begin with an overview of scalar-valued continuous-domain multiD signals and systems, i.e. the domain is for some integer . In particular we introduce the concepts of signal space, linear shift-invariant systems and the continuous-domain multiD Fourier transform, develop properties of the Fourier transform and present some examples. Continuous-domain signal spaces and transforms involve advanced mathematical analysis to provide a general theory for arbitrary signal spaces. We do not attempt to provide a rigorous analysis. We assume that signals belong to a suitable signal space for which transforms are well defined and the properties hold. For example, a space of tempered distributions would be satisfactory. However, we do not develop the theory of distributions and take an informal approach to the Dirac delta and related singularities. We refer the reader to references for a rigorous analysis, e.g., [Stein and Weiss (1971), Richards and Youn (1990)].

There are many possible domains for multiD signals, generally subsets of for some . These domains can be continuous or discrete, or a hybrid that is continuous in some dimensions and discrete in others, like in analog TV scanning. The domain can also correspond to one period of a periodic signal, whether continuous or discrete. Among the possible domains, certain of them allow for the possibility of linear shift-invariant filtering. These domains have the algebraic structure of a locally-compact Abelian (LCA) group. While we cannot go into the detail of such structures, their main feature is that the concept of shift is well defined and commutative. is an example, as is any lattice in . The LCA group is the classical setting for abstract harmonic analysis, e.g., as presented in Rudin (1962). An early work on signal processing in this setting is the Ph.D. thesis of Rudolph Seviora [Seviora (1971)]), which considered generalized digital filtering on LCA groups. More recently Cariolaro has developed signal processing on LCA groups in a comprehensive book [Cariolaro (2011)].

In this book, we have elected to provide a separate development for the cases of continuous-domain aperiodic signals, discrete-domain aperiodic signals, discrete-domain periodic signals, and continuous-domain periodic signals (Chapters 02-05). Each case has its own sphere of application, and while the development may be redundant from an abstract mathematical perspective, the concrete details are sufficiently different to warrant their own presentation. Each of these chapters follows a similar roadmap, presenting concepts of signal space, linear shift-invariant (LSI) systems, Fourier transforms and their properties. For discrete-domain signals, we use lattices to describe the sampling structure. For periodic signals, we use lattices to describe the periodicity. Since lattices form an underlying tool used throughout the book, we have chosen to gather all definitions and results about lattices that we need for this work in Chapter 13, which may be consulted any time as needed. We prefer not to interrupt the flow of the book at the beginning with this material, and we wish to give it a higher status than an appendix. This is why we have chosen to include it as the last chapter in the book.

In Chapter 6 we see the relationship between the four representations. Discrete-domain aperiodic and periodic signals can be obtained for the corresponding continuous-domain signals by a sampling operation. This is shown to induce a periodization in the frequency domain. In another view, discrete and continuous-domain periodic signals can be obtained by periodization of corresponding aperiodic signals, resulting in sampling in the frequency domain. These results are all explored in Chapter 06and various sampling theorems are presented. We do not explicitly explore hybrid signals, which may correspond to a different one of the above types in different dimensions. This extension is usually straightforward; many examples are given in Cariolaro ( 2011).

Having developed the theory of processing of multiD scalar signals, we address the nature of the signal range in Chapter 7 , specifically for color image signals. Here we take up the vector-space view of color spaces as presented in Dubois (2010), where colors are viewed as equivalence classes...