2
Pixel-based Classification Techniques for Satellite Image Time Series

Charlotte PELLETIER1 and Silvia VALERO2

1IRISA Laboratory UMR 6074, Université Bretagne Sud, Lorient, France

2CESBIO Laboratory UMR 5126, University of Toulouse, France

2.1. Introduction

Satellite image time series have proven to be an effective tool for monitoring vegetation dynamics, resources and the effects of climate change. These multitemporal data offer frequent and consistent information on what the Earth's land cover looks like. Recent advances in remote sensing technology have resulted in the acquisition of new satellite image time series (SITS) rendered in high spatial, spectral and temporal resolution. New Earth Observation (EO) missions, such as Landsat-8 or Sentinel-1 and -2, acquire time series that differ significantly from previous datasets. Indeed, their analysis is a Big Data challenge due to the quantity of data, for example, 12 TB per day for the European Sentinel programme.

To automatically extract information about the biophysical cover of the Earth's surface (namely land cover), machine learning and more specifically supervised learning, is the most significant approach used. When applied to multitemporal remote sensing data, its goal is to associate a target label with each temporal profile pixel. Supervised classifiers use a priori information about the data to train a model that is able to extract and recognize relevant features. The trained model is then used to label each pixel in the image scene. The classification of a remote sensing scene differs from a classical classification computer vision task, which aims to categorize all pixels in an image into a single class.

The high-dimensional spectro-spatio-temporal representation of the SITS data opens up new challenges and problems for supervised classification algorithms. Example constraints are the irregular temporal sampling because of the presence of artifacts (clouds, cloud shadows or saturated pixels) and the presence of mixed pixels that are due to the medium spatial resolution of these datasets.

Other crucial prerequisites for successfully training a classification model are the selection of a suitable classification algorithm, as well as the availability of quality and representative labeled training samples. A significant quantity of labeled samples is also a key requirement for the classification of these high-dimensional datasets.

Furthermore, applying a supervised classification algorithm requires the determination of a suitable input feature representation. Traditionally, human knowledge has been used to supervise the feature engineering step, which extracts and selects relevant features from the data. This manual feature engineering process and the construction of a large high-quality reference dataset are usually the two most time-consuming tasks for supervised classification tasks.

By contrast, in the last few years, promising classification strategies based on deep learning automatically generated important features. The use of deep learning approaches saves feature engineering costs, but in return it requires important expertise in network architecture engineering. It also needs a larger amount of labeled data to train the models, which is not always available in remote sensing classification applications.

This chapter addresses the above-mentioned key issues in the classification of SITS by presenting an overview of supervised classification algorithms proposed in the literature. It focuses on pixel-wise methods without considering the spatial context.

We begin by introducing some fundamentals about supervised classification in the context of remote sensing applications. Section 2.2 also includes a description of some key data preprocessing steps and the classical evaluation procedure based on the confusion matrix. Section 2.3 overviews three well-established classification algorithms massively used for SITS classification (Gómez et al. 2016): support vector machines, random forests and k-nearest neighbors. Section 2.4 presents several classification approaches proposing low-dimensional temporal feature representations of EO time series. We describe the phenological temporal features extracted from vegetation indices (Valero et al. 2016), bag-of-words approaches (Bailly et al. 2016) and shapelet methods. Finally, section 2.5 details the main deep learning networks recently used for the classification of EO time series, i.e., temporal convolutional neural networks (Pelletier et al. 2019; Zhong et al. 2019) and recurrent neural networks (and their variants) (Ienco et al. 2017; Rußwurm and Körner 2017).

2.2. Basic concepts in supervised remote sensing classification

The goal of supervised learning is for the algorithm to automatically learn rules to predict the labels of new samples. The set of rules is learned from reference data, which is an essential component for developing robust supervised machine learning algorithms. Reference data is described by a feature vector extracted from the satellite data and a reference label. The reference label denotes the class to be predicted.

The quality of the reference dataset is vital for supervised classifiers, and even the most powerful algorithms can be rendered useless when they use inadequate, inaccurate or irrelevant data. Accordingly, the preparation of reference data is one of the most essential steps in supervised classification. The amount and the quality of available reference data in remote sensing represent important constraints for the choice of an appropriate classification scheme (input data, classifier, nomenclature, etc.). In general, the quality of such a dataset depends on the quality of the reference labels and the quality of features describing the data.

In this section, we will overview how to prepare the satellite and reference datasets. Then, we will present some key considerations for training supervised classification algorithms. Finally, we will describe how to evaluate a classification result.

2.2.1. Preparing data before it is fed into classification algorithms

Data preparation should result in an accurate reference dataset, which is ready to be used by a machine learning algorithm to uncover insights and/or make predictions. Challenging and time-consuming, this task involves the preparation of satellite data and reference labels as detailed in the following.

2.2.1.1. Satellite data

Preparing satellite data implies costly preprocessing operations. The goal of this preparation is to correct for sensor- and platform-specific distortions of data. The set of preparation tasks is especially important for the analysis of multitemporal images, since satellite images composing SITS are acquired on different time periods. This requires special attention because each preparation procedure further alters the data from their original values and can thus increase the potential to introduce errors. Optical satellite data preprocessing involves geometric and radiometric calibration corrections, whereas radar data requires speckle filtering.

One of the most popular preprocessing steps is the imputation of missing values. SITS data indeed have irregular temporal sampling introduced by cloud cover, acquisitions from different orbit tracks and/or sensor geometry artifacts. In addition, optical images from several geographical areas are affected differently by clouds and thus present various irregular temporal samplings. Most of the classification algorithms require that all input samples have the same feature vector dimension, but they do not efficiently deal with missing values themselves. The most common way to handle missing values consists of filling-in approximations. Those methods are called "gap filling" or "imputation" methods. Although different techniques are proposed in the literature, classical weighted linear interpolation is the most used one for large-scale studies (Inglada et al. 2015). Data normalization is another important preprocessing step needed by most of the algorithms to align all features on the same scale. This scaling procedure consists of standardizing the range of each feature independently (Pelletier et al. 2019, section 3.3.3).

Once preprocessing is completed, the second most important preparation step is the transformation of the preprocessed data into a feature representation that is as suitable as possible for learning.

The SITS data offers a natural time-based representation of the data, which is used in most classification tasks. The use of all available images helps classification algorithms to handle high class intra-variability, i.e. classes with different modes, and also low class inter-variability, i.e. two classes that have strong similarity. On the contrary, the use of few satellite images might prevent the recognition of different classes that have a similar appearance at a given time. For example, using only images acquired during summer makes the recognition of winter crops impossible. Furthermore, the selection of a subset of images is not a straightforward task and it might result in a loss of valuable information.

For this reason, a suitable feature representation is the raw data. The classification algorithms are thus fed with all of the satellite images contained in the sequence. Another possible representation can be obtained by computing new features. They are used to either...