Image, Video and 3D Data Registration

Preface xi Acknowledgements xiii 1 Introduction 1 1.1 The History of Image Registration 1 1.2 Definition of Registration 2 1.3 What is Motion Estimation 3 1.4 Video Quality Assessment 5 1.5 Applications 5 1.5.1 Video Processing 5 1.5.2 Medical Applications 7 1.5.3 Security Applications 8 1.5.4 Military and Satellite Applications 10 1.5.5 Reconstruction Applications 11 1.6 Organization of the Book 12 References 13 2 Registration for Video Coding 15 2.1 Introduction 15 2.2 Motion Estimation Technique 16 2.2.1 Block-Based Motion Estimation Techniques 16 2.3 Registration and Standards for Video Coding 30 2.3.1 H.264 30 2.3.2 H.265 34 2.4 Evaluation Criteria 35 2.4.1 Dataset 35 2.4.2 Motion-Compensated Prediction Error (MCPE) in dB 38 2.4.3 Entropy in bpp 39 2.4.4 Angular Error in Degrees 40 2.5 Objective Quality Assessment 41 2.5.1 Full-Reference Quality Assessment 41 2.5.2 No-Reference and Reduced-Reference Quality Metrics 44 2.5.3 Temporal Masking in Video Quality Assessment 46 2.6 Conclusion 48 2.7 Exercises 49 References 49 3 Registration for Motion Estimation and Object Tracking 53 3.1 Introduction 53 3.1.1 Mathematical Notation 54 3.2 Optical Flow 55 3.2.1 Horn-Schunk Method 56 3.2.2 Lukas-Kanade Method 56 3.2.3 Applications of Optical Flow for Motion Estimation 57 3.3 Efficient Discriminative Features for Motion Estimation 61 3.3.1 Invariant Features 62 3.3.2 Optimization Stage 64 3.4 Object Tracking 64 3.4.1 KLT Tracking 64 3.4.2 Motion Filtering 66 3.4.3 Multiple Object Tracking 67 3.5 Evaluating Motion Estimation and Tracking 68 3.5.1 Metrics for Motion Detection 68 3.5.2 Metrics for Motion Tracking 69 3.5.3 Metrics for Efficiency 70 3.5.4 Datasets 70 3.6 Conclusion 70 3.7 Exercise 75 References 75 4 Face Alignment and Recognition Using Registration 79 4.1 Introduction 79 4.2 Unsupervised Alignment Methods 80 4.2.1 Natural Features: Gradient Features 81 4.2.2 Dense Grids: Non-rigid Non-affine Transformations 81 4.3 Supervised Alignment Methods 83 4.3.1 Generative Models 84 4.3.2 Discriminative Approaches 86 4.4 3D Alignment 88 4.4.1 Hausdorff Distance Matching 88 4.4.2 Iterative Closest Point (ICP) 89 4.4.3 Multistage Alignment 89 4.5 Metrics for Evaluation 90 4.5.1 Evaluating Face Recognition 90 4.5.2 Evaluating Face Alignment 90 4.5.3 Testing Protocols and Benchmarks 91 4.5.4 Datasets 92 4.6 Conclusion 94 4.7 Exercise 94 References 94 5 Remote Sensing Image Registration in the Frequency Domain 97 5.1 Introduction 97 5.2 Challenges in Remote Sensing Imaging 100 5.3 Satellite Image Registration in the Fourier Domain 102 5.3.1 Translation Estimation Using Correlation 102 5.4 Correlation Methods 103 5.5 Subpixel Shift Estimation in the Fourier Domain 107 5.6 FFT-Based Scale-Invariant Image Registration 111 5.7 Motion Estimation in the Frequency Domain for Remote Sensing Image Sequences 115 5.7.1 Quad-Tree Phase Correlation 116 5.7.2 Shape Adaptive Motion Estimation in the Frequency Domain 119 5.7.3 Optical Flow in the Fourier Domain 120 5.8 Evaluation Process and Related Datasets 122 5.8.1 Remote Sensing Image Datasets 123 5.9 Conclusion 123 5.10 Exercise - Practice 124 References 124 6 Structure from Motion 129 6.1 Introduction 129 6.2 Pinhole Model 131 6.3 Camera Calibration 133 6.4 Correspondence Problem 135 6.5 Epipolar Geometry 136 6.6 Projection Matrix Recovery 140 6.6.1 Triangulation 141 6.7 Feature Detection and Registration 141 6.7.1 Auto-correlation 143 6.7.2 Harris Detector 143 6.7.3 SIFT Feature Detector 146 6.8 Reconstruction of 3D Structure and Motion 148 6.8.1 Simultaneous Localization and Mapping 149 6.8.2 Registration for Panoramic View 150 6.9 Metrics and Datasets 152 6.9.1 Datasets for Performance Evaluation 154 6.10 Conclusion 155 6.11 Exercise - Practice 155 References 155 7 Medical Image Registration Measures 162 7.1 Introduction 162 7.2 Feature-Based Registration 163 7.2.1 Generalized Iterative Closest Point Algorithm 164 7.2.2 Hierarchical Maximization 165 7.3 Intensity-Based Registration 165 7.3.1 Voxels as Features 166 7.3.2 Special Case: Spatially Determined Correspondences 168 7.3.3 Intensity Difference Measures 169 7.3.4 Correlation Coefficient 170 7.3.5 Pseudo-likelihood Measures 171 7.3.6 General Implementation Using Joint Histograms 181 7.4 Transformation Spaces and Optimization 184 7.4.1 Rigid Transformations 185 7.4.2 Similarity Transformations 186 7.4.3 Affine Transformations 186 7.4.4 Projective Transformations 187 7.4.5 Polyaffine Transformations 187 7.4.6 Free-Form Transformations: 'Small Deformation' Model 188 7.4.7 Free-Form Transformations: 'Large Deformation' Models 189 7.5 Conclusion 193 7.6 Exercise 193 7.6.1 Implementation Guidelines 195 References 196 8 Video Restoration Using Motion Information 201 8.1 Introduction 201 8.2 History of Video and Film Restoration 203 8.3 Restoration of Video Noise and Grain 206 8.4 Restoration Algorithms for Video Noise 208 8.5 Instability Correction Using Registration 211 8.6 Estimating and Removing Flickering 214 8.7 Dirt Removal in Video Sequences 217 8.8 Metrics in Video Restoration 221 8.9 Conclusions 225 8.10 Exercise - Practice 225 References 225 Index 229