1
Introduction of Real-time Image Processing

1.1. General image processing presentation

The traditional view of an image derives heavily from experience in photography, television and the like. This means that an image is a two-dimensional (2D) structure, a representation and also a structure with meaning to a visual response system. This view of an image only accepts spatial variation (a static image). In parallel, a dynamic image has spatial and temporal variation. In most contexts, this is usually referred to as video. This more complex structure needs to be viewed as a sequence of images each representing a particular instance in time. On the other hand, an image can be formed by taking a sampling plane through that volume and so the variation in three dimensions is observed. This may be referred to as a volume image. An image linked to a volume that changes with time is a further possibility. This has particular significance in medical imaging applications [MYE 09].

Image processing is a method to convert an image into digital form and perform some operations on it in order to get an improved image or to extract some useful information from it. A digital image described in a 2D space is usually considered as 2D signals while applying already set signal methods to it. Image processing forms a core research area within engineering and computer science too.

Image processing is an enabling technology for a wide range of applications including remote sensing, security, image data sets, digital television, robotics and medical imaging, etc. The goal of this technology can be divided into three levels: low, medium and high. Low-level image processing techniques are mathematical or logical operators that perform simple processing tasks. This "processing" level possesses both image_in and image_out. Medium-level image processing combines the simple low-level operators to perform feature extraction and pattern recognition functions. Using an image_in, this "analysis" level produces measurements_out (parameters). High-level image processing uses combinations of medium-level functions to perform interpretation. This "understanding" level treats measurements_in for high-level description_out. Usually, apart from these three levels, it is also necessary to apply preprocessing techniques that are designed to remove distortions introduced by sensors.

Figure 1.1. Overview of the typical image acquisition process (see [MOE 12])

Before any image processing can commence, an image must be captured by a camera and converted into a manageable entity. This is the image acquisition process (see Figure 1.1), which consists of three steps; energy reflected from the object of interest, an optical system that focuses on the energy and finally a sensor that measures the amount of energy. In order to capture an image, a camera requires some sort of measurable energy. The energy of interest in this context is light or electromagnetic waves. Each wave can have different wavelengths (or different energy levels or different frequencies). The human visual spectrum is in the range of approximately 400-700 nm.

After having illuminated the object of interest, the light reflected from the object now has to be captured by the camera composed of an optical system and an image sensor. The role of the first is to focus the light reflected from the object onto the second (a material sensitive to the reflected light). An image sensor consists of a 2D array of cells. Each of these cells is denoted a pixel and is capable of measuring the amount of incident light and convert that into a voltage, which in turn is converted into a digital number. The more incident light, the higher the voltage and the higher the digital number.

In order to transform the information from the sensor into an image, each cell content is now converted into a pixel value in the range: (0, 255). Such a value is interpreted as the amount of light hitting a cell. This is denoted the intensity of a pixel. It is visualized as a shade of gray denoted gray-level value ranging from black (0) to white (255). Standardly, a monochromatic, static image corresponds to a matrix of m rows and n columns. Therefore, the camera records m × n pixels of 8 bit values.

In order to capture a color image, the color camera must record three matrices of three primary colors red, green and blue. Recently, a lot of applications are realized using multispectral image processing, since the multispectral cameras are more available with reasonable prices. According to application needs, multispectral images are captured using different wavelengths (bands). They can be considered as a cube formed by 2D gray-level images. Figure 1.2 displays two typical test images in the area of image processing research. Figure 1.3 gives two multispectral cube examples; the right image is captured for a skin lesion assessment application.

Figure 1.2. Lena (gray-level image) and landscape (color image). For a color version of the figure, see www.iste.co.uk/li/image.zip

Figure 1.3. Two multispectral image cubes. For a color version of the figure, see www.iste.co.uk/li/image.zip

Certain tools are central to the processing of digital images. These include mathematical tools, statistical descriptions and manipulative tools. We can cite some more used such as convolution, filtering in spatial and frequency domains, morphological operations and image transforms, etc. The types of basic operations that can be applied to digital images to transform an input image A(m, n) into an output image B(m, n) (or another representation) can be classified into three categories:

• - Point: the output value at a specific matrix coordinate is dependent only on the input value at that same coordinate. In this case, generic operation complexity per pixel is constant.
• - Local: the output value at a specific coordinate is dependent on the input values in the neighborhood of that same coordinate. In most cases, the type of neighborhood is rectangular with 8-connected (3 × 3 mask) or 24-connected (5 × 5 mask pixels). The complexity per pixel is proportional to the square of neighborhood size.
• - Global: the output value at a specific coordinate is dependent on all the values in the input image. The complexity per pixel is equal to N × N (image size = N).

The complexity per pixel of each operation type participates in the total complexity of an image processing chain for a target application. This total complexity per image decides the processing speed (time performance).

1.2. Real-time image processing

Real-time image processing is the subfield of image processing focused on producing and analyzing images in real time. Generally, a real-time system has the following three interpretations within different senses [KEH 06]:

• - real time in the perceptual sense, which is used to describe the interaction between a human and a computer device for a near instantaneous response of the device to an input by a human user;
• - real time in the software engineering sense, which is used to describe a concept of bounded response time in the computer device. With this constraint, the device must satisfy both the correctness of the outputs and their timeliness;
• - real time in the signal processing sense, which is used to describe the constraint within which the computer device has to complete processing in the time available between successive input samples.

Since image processing is a subfield of signal processing, in this book we base the interpretation of "real time" on the signal processing sense.

In order to satisfy the constraint of "real time", Kehtarnavaz [KEH 11] points out that the total instruction count of a real-time algorithm must be "less than the number of instructions that can be executed between two consecutive samples", and Ackenhusen et al. [ACK 99] describe the "real-time processing" as a computation of "a certain number of operations upon a required amount of input data within a specified interval of time" as well. Therefore, the key technique of real-time image processing is to ensure that the amount of time for completing all the requisite transferring and processing of image data is less than the allotted time for processing. For example, if the algorithm is aimed at an entire frame (a static image) and the frame frequency of the system is 30 frames per second (fps), the processing of a single frame should be finished during 33 ms.

Real-time image processing systems involve processing vast amounts of image data in a timely manner for the purpose of extracting useful information, which could mean anything from obtaining an enhanced image to intelligent scene analysis. Digital images and video are essentially multidimensional signals and are thus quite data intensive, requiring a significant amount of computation and memory resources for their processing. A common theme in real-time image processing systems is how to deal with their vast amount of processing and computations. The key to cope with this issue is the concept of parallel processing, a concept well known to those working in the architecture area who deal with computations on large data sets [KEH 06]. In fact,...