
Camera Image Quality Benchmarking
Description
Alles über E-Books | Antworten auf Fragen rund um E-Books, Kopierschutz und Dateiformate finden Sie in unserem Info- & Hilfebereich.
Camera Image Quality Benchmarking contains the basic information and approaches for the use of subjectively correlated image quality metrics and outlines a framework for camera benchmarking. The authors show how to quantitatively compare image quality of cameras used for consumer photography. This book helps to fill a void in the literature by detailing the types of objective and subjective metrics that are fundamental to benchmarking still and video imaging devices. Specifically, the book provides an explanation of individual image quality attributes and how they manifest themselves to camera components and explores the key photographic still and video image quality metrics. The text also includes illustrative examples of benchmarking methods so that the practitioner can design a methodology appropriate to the photographic usage in consideration.
The authors outline the various techniques used to correlate the measurement results from the objective methods with subjective results. The text also contains a detailed description on how to set up an image quality characterization lab, with examples where the methodological benchmarking approach described has been implemented successfully. This vital resource:
* Explains in detail the entire process behind performing a complete characterization and benchmarking of cameras through image quality analysis
* Provides best practice measurement protocols and methodologies, so readers can develop and define their own camera benchmarking system to industry standards
* Includes many photographic images and diagrammatical illustrations to clearly convey image quality concepts
* Champions benchmarking approaches that value the importance of perceptually correlated image quality metrics
Written for image scientists, engineers, or managers involved in image quality and evaluating camera performance, Camera Image Quality Benchmarking combines knowledge from many different engineering fields, correlating objective (perception-independent) image quality with subjective (perception-dependent) image quality metrics.
More details
Other editions
Additional editions


Persons
JONATHAN B. PHILLIPS, is a Staff Image Scientist at Google, USA. He is a United States delegate to the technical committee ISO/TC 42 Photography and a major contributor to the IEEE Camera Phone Image Quality (CPIQ) initiative.
HENRIK ELIASSON, PHD, is an image sensor and image analysis specialist at Eclipse Optics, Sweden. He is a senior member of SPIE.
Content
About the Authors xv
Series Preface xvii
Preface xix
List of Abbreviations xxiii
About the CompanionWebsite xxvii
1 Introduction 1
1.1 Image Content and Image Quality 2
1.1.1 Color 3
1.1.2 Shape 8
1.1.3 Texture 10
1.1.4 Depth 11
1.1.5 Luminance Range 12
1.1.6 Motion 15
1.2 Benchmarking 18
1.3 Book Content 22
Summary of this Chapter 24
References 25
2 Defining Image Quality 27
2.1 What is Image Quality? 27
2.2 Image Quality Attributes 29
2.3 Subjective and Objective Image Quality Assessment 31
Summary of this Chapter 32
References 33
3 Image Quality Attributes 35
3.1 Global Attributes 35
3.1.1 Exposure, Tonal Reproduction, and Flare 35
3.1.2 Color 39
3.1.3 Geometrical Artifacts 40
3.1.3.1 Perspective Distortion 40
3.1.3.2 Optical Distortion 42
3.1.3.3 Other Geometrical Artifacts 42
3.1.4 Nonuniformities 43
3.1.4.1 Luminance Shading 45
3.1.4.2 Color Shading 45
3.2 Local Attributes 45
3.2.1 Sharpness and Resolution 45
3.2.2 Noise 49
3.2.3 Texture Rendition 50
3.2.4 Color Fringing 50
3.2.5 Image Defects 51
3.2.6 Artifacts 51
3.2.6.1 Aliasing and Demosaicing Artifacts 52
3.2.6.2 Still Image Compression Artifacts 53
3.2.6.3 Flicker 53
3.2.6.4 HDR Processing Artifacts 55
3.2.6.5 Lens Ghosting 55
3.3 Video Quality Attributes 56
3.3.1 Frame Rate 56
3.3.2 Exposure and White Balance Responsiveness and Consistency 58
3.3.3 Focus Adaption 58
3.3.4 Audio-Visual Synchronization 58
3.3.5 Video Compression Artifacts 59
3.3.6 Temporal Noise 60
3.3.7 Fixed Pattern Noise 60
3.3.8 Mosquito Noise 60
Summary of this Chapter 60
References 61
4 The Camera 63
4.1 The Pinhole Camera 63
4.2 Lens 64
4.2.1 Aberrations 64
4.2.1.1 Third-Order Aberrations 65
4.2.1.2 Chromatic Aberrations 66
4.2.2 Optical Parameters 67
4.2.3 Relative Illumination 69
4.2.4 Depth of Field 70
4.2.5 Diffraction 71
4.2.6 Stray Light 73
4.2.7 Image Quality Attributes Related to the Lens 74
4.3 Image Sensor 75
4.3.1 CCD Image Sensors 75
4.3.2 CMOS Image Sensors 77
4.3.3 Color Imaging 81
4.3.4 Image Sensor Performance 82
4.3.5 CCD versus CMOS 89
4.3.6 Image Quality Attributes Related to the Image Sensor 90
4.4 Image Signal Processor 91
4.4.1 Image Processing 91
4.4.2 Image Compression 98
4.4.2.1 Chroma Subsampling 98
4.4.2.2 Transform Coding 98
4.4.2.3 Coefficient Quantization 99
4.4.2.4 Coefficient Compression 100
4.4.3 Control Algorithms 101
4.4.4 Image Quality Attributes Related to the ISP 101
4.5 Illumination 102
4.5.1 LED Flash 103
4.5.2 Xenon Flash 103
4.6 Video Processing 103
4.6.1 Video Stabilization 103
4.6.1.1 Global Motion Models 104
4.6.1.2 Global Motion Estimation 105
4.6.1.3 Global Motion Compensation 106
4.6.2 Video Compression 107
4.6.2.1 Computation of Residuals 107
4.6.2.2 Video Compression Standards and Codecs 109
4.6.2.3 Some Significant Video Compression Standards 110
4.6.2.4 A Note On Video Stream Structure 111
4.7 System Considerations 111
Summary of this Chapter 112
References 113
5 Subjective Image Quality Assessment-Theory and Practice 117
5.1 Psychophysics 118
5.2 Measurement Scales 120
5.3 PsychophysicalMethodologies 122
5.3.1 Rank Order 123
5.3.2 Category Scaling 123
5.3.3 Acceptability Scaling 124
5.3.4 Anchored Scaling 125
5.3.5 Forced-Choice Comparison 125
5.3.6 Magnitude Estimation 125
5.3.7 Methodology Comparison 126
5.4 Cross-Modal Psychophysics 126
5.4.1 Example Research 127
5.4.2 Image Quality-Related Demonstration 128
5.5 Thurstonian Scaling 129
5.6 Quality Ruler 131
5.6.1 Ruler Generation 134
5.6.2 Quality Ruler Insights 135
5.6.2.1 Lab Cross-Comparisons 135
5.6.2.2 SQS2 JND Validation 136
5.6.2.3 Quality Ruler Standard Deviation Trends 139
5.6.2.4 Observer Impact 141
5.6.3 Perspective from Academia 142
5.6.4 Practical Example 144
5.6.5 Quality Ruler Applications to Image Quality Benchmarking 147
5.7 Subjective Video Quality 148
5.7.1 Terminology 149
5.7.2 Observer Selection 149
5.7.3 Viewing Setup 150
5.7.4 Video Display and Playback 151
5.7.5 Clip Selection 152
5.7.6 Presentation Protocols 154
5.7.7 Assessment Methods 156
5.7.8 Interpreting Results 158
5.7.9 ITU Recommendations 159
5.7.9.1 The Double-Stimulus Impairment Scale Method 160
5.7.9.2 The Double-Stimulus Continuous Quality Scale Method 160
5.7.9.3 The Simultaneous Double-Stimulus for Continuous Evaluation Method 160
5.7.9.4 The Absolute Category Rating Method 161
5.7.9.5 The Single Stimulus Continuous Quality Evaluation Method 161
5.7.9.6 The Subjective Assessment of Multimedia Video Quality Method 161
5.7.9.7 ITU Methodology Comparison 162
5.7.10 Other Sources 162
Summary of this Chapter 162
References 163
6 Objective Image Quality Assessment-Theory and Practice 167
6.1 Exposure and Tone 168
6.1.1 Exposure Index and ISO Sensitivity 168
6.1.2 Optoelectronic Conversion Function 169
6.1.3 Practical Considerations 170
6.2 Dynamic Range 170
6.3 Color 171
6.3.1 Light Sources 171
6.3.2 Scene 174
6.3.3 Observer 176
6.3.4 Basic Color Metrics 178
6.3.5 RGB Color Spaces 180
6.3.6 Practical Considerations 181
6.4 Shading 181
6.4.1 Practical Considerations 182
6.5 Geometric Distortion 182
6.5.1 Practical Considerations 184
6.6 Stray Light 184
6.6.1 Practical Considerations 185
6.7 Sharpness and Resolution 185
6.7.1 The Modulation Transfer Function 186
6.7.2 The Contrast Transfer Function 191
6.7.3 Geometry in Optical Systems and the MTF 193
6.7.4 Sampling and Aliasing 194
6.7.5 System MTF 195
6.7.6 Measuring the MTF 198
6.7.7 Edge SFR 198
6.7.8 Sine Modulated Siemens Star SFR 201
6.7.9 Comparing Edge SFR and Sine Modulated Siemens SFR 203
6.7.10 Practical Considerations 204
6.8 Texture Blur 204
6.8.1 Chart Construction 206
6.8.2 Practical Considerations 206
6.8.3 AlternativeMethods 207
6.9 Noise 207
6.9.1 Noise and Color 207
6.9.2 Spatial Frequency Dependence 209
6.9.3 Signal to Noise Measurements in Nonlinear Systems and Noise Component Analysis 211
6.9.4 Practical Considerations 212
6.10 Color Fringing 213
6.11 Image Defects 214
6.12 Video Quality Metrics 214
6.12.1 Frame Rate and Frame Rate Consistency 215
6.12.2 Frame Exposure Time and Consistency 215
6.12.3 Auto White Balance Consistency 216
6.12.4 Autofocusing Time and Stability 216
6.12.5 Video Stabilization Performance 217
6.12.6 Audio-Video Synchronization 218
6.13 Related International Standards 218
Summary of this Chapter 221
References 221
7 Perceptually Correlated Image Quality Metrics 227
7.1 Aspects of Human Vision 227
7.1.1 Physiological Processes 227
7.2 HVS Modeling 232
7.3 Viewing Conditions 232
7.4 Spatial Image Quality Metrics 234
7.4.1 Sharpness 235
7.4.1.1 Edge Acutance 235
7.4.1.2 Mapping Acutance to JND Values 237
7.4.1.3 Other Perceptual Sharpness Metrics 239
7.4.2 Texture Blur 239
7.4.3 Visual Noise 240
7.5 Color 244
7.5.1 Chromatic Adaptation Transformations 244
7.5.2 Color Appearance Models 245
7.5.3 Color and Spatial Content-Image Appearance Models 247
7.5.4 Image Quality Benchmarking and Color 249
7.6 Other Metrics 251
7.7 Combination of Metrics 252
7.8 Full-Reference Digital Video Quality Metrics 252
7.8.1 PSNR 253
7.8.2 Structural Similarity (SSIM) 256
7.8.3 VQM 260
7.8.4 VDP 262
7.8.4.1 Further Considerations 263
7.8.5 Discussion 265
Summary of this Chapter 267
References 267
8 Measurement Protocols-Building Up a Lab 273
8.1 Still Objective Measurements 273
8.1.1 Lab Needs 274
8.1.1.1 Lab Space 274
8.1.1.2 Lighting 275
8.1.1.3 Light Booths 278
8.1.1.4 Transmissive Light Sources 279
8.1.1.5 Additional Lighting Options 280
8.1.1.6 Light Measurement Devices 281
8.1.2 Charts 282
8.1.2.1 Printing Technologies for Reflective Charts 282
8.1.2.2 Technologies for Transmissive Charts 286
8.1.2.3 Inhouse Printing 286
8.1.2.4 Chart Alignment and Framing 287
8.1.3 Camera Settings 289
8.1.4 Supplemental Equipment 289
8.1.4.1 RealWorld Objects 290
8.2 Video Objective Measurements 293
8.2.0.2 Visual Timer 293
8.2.0.3 Motion 294
8.3 Still Subjective Measurements 297
8.3.1 Lab Needs 297
8.3.2 Stimuli 298
8.3.2.1 Stimuli Generation 298
8.3.2.2 Stimuli Presentation 301
8.3.3 Observer Needs 302
8.3.3.1 Observer Selection and Screening 302
8.3.3.2 Experimental Design and Duration 303
8.4 Video Subjective Measurements 304
Summary of this Chapter 305
References 305
9 The Camera Benchmarking Process 309
9.1 Objective Metrics for Benchmarking 309
9.2 Subjective Methods for Benchmarking 311
9.2.1 Photospace 312
9.2.2 Use Cases 313
9.2.3 Observer Impact 314
9.3 Methods of Combining Metrics 315
9.3.1 Weighted Combinations 316
9.3.2 Minkowski Summation 316
9.4 Benchmarking Systems 317
9.4.1 GSMArena 317
9.4.2 FNAC 318
9.4.3 VCX 318
9.4.4 Skype Video Capture Specification 319
9.4.5 VIQET 320
9.4.6 DxOMark 321
9.4.7 IEEE P1858 323
9.5 Example Benchmark Results 324
9.5.1 VIQET 324
9.5.2 IEEE CPIQ 325
9.5.2.1 CPIQ Objective Metrics 327
9.5.2.2 CPIQ Quality Loss Predictions from Objective Metrics 337
9.5.3 DxOMark Mobile 338
9.5.4 Real-World Images 339
9.5.5 High-End DSLR Objective Metrics 339
9.6 Benchmarking Validation 345
Summary of this Chapter 348
References 349
10 Summary and Conclusions 353
References 357
Index 359
Preface
The seed for the content of this book started in 2011 when Nicolas Touchard of DxO Labs in France, being a participant in the Camera Phone Image Quality (CPIQ) initiative just like us, contacted us about a short course they wanted to teach on objective and subjective camera image quality benchmarking at the then SPIE/IS&T Electronic Imaging conference (now IS&T International Symposium on Electronic Imaging). Nicolas proposed to have us join some of his colleagues, Harvey (Hervé) Hornung, Frédéric Cao and Frédéric Guichard, to plan this new course. In 2012, we launched our short course on camera image quality benchmarking with a set of nearly 400 slides, with the DxO team being the major contributor. Over a period of several years, the course was supplemented and revised, with particular attention to adding video image quality to our initial focus of still imaging. Five individuals were involved with the class instruction over time: apart from Hervé and the two of us (Henrik and Jonathan), also Nicolas Touchard and Hugh Denman. When John Wiley & Sons Ltd asked Jonathan in 2014 about converting our course slides to a book, he contacted each of the course contributors with the same inquiry. Finally, the two of us decided we were up to the challenge. As we began the writing, we realized we needed to convince Hugh, who was at YouTube at the time, to join our efforts as a contributing author on the topic of video image quality.
We have been involved in image quality measurements for many years, in the mobile industry as well as more generally. Our backgrounds are slightly different: while Jonathan has mainly been focusing his efforts on the subjective and pragmatic side of image quality assessment, Henrik has been looking more at objective image quality metrics and the theory of such. Thus, the book has been naturally divided between us, with Jonathan responsible for Chapters 1, 5, 8, and 9, and Henrik for Chapters 2, 3, 4, 6, and 7. We need to mention here also the contribution from Hugh Denman, who has been responsible for the video-related content in Chapters 1 and 3 through 8.
We have met regularly, nearly every weekend via webcam, for the past several years as we have been collaborating on the book. To increase our productivity, we have even had "Writers Workshops" in our respective countries, with Henrik having spent time in San Jose with Jonathan, and Jonathan traveling to spend time in both Sweden with Henrik and Ireland with Hugh. Those workshops as well as weekly meetings have enabled us to assemble this book in a cohesive fashion, including both still and video image quality benchmarking material. Our photography in the following chapters includes images of scenes from our respective countries among others, which we have captured throughout the writing process and are excited to share with the readers of the book.
It has been very interesting to follow the evolution of the camera from a specialized piece of equipment that one carried mostly during vacations and other important events to a ubiquitous component in a mobile communications device one always has in one's pocket or carryall. 15 years ago, it would have been hard to believe that the image quality provided by an average mobile phone camera today would actually be better in most cases compared with the compact cameras then available. Still, there are areas in which the mobile phone camera is lacking, like low-light photography and zooming capabilities. At the same time, the basic principle of photography and video capture has not changed significantly: you point the camera toward the subject you wish to capture and press the shutter button. The camera then records the image or video and one can view it, for example, directly on screen or as a printout at a later time. What has changed tremendously is in the way we use the photos and videos taken, as well as the subjects captured. From being "frozen memories" helping us to recall important moments in our lives and brought out to be viewed at specific occasions, photos are now to a large extent consumed immediately and used to augment social interactions. This has led to an enormous increase in the quantity of pictures taken.
Even with this exponentially increasing volume of new photos taken every day, which inevitably leads to a shorter attention span with regard to viewing the images, the quality of the images is still important. This is not least evident from the advertising made by the large mobile phone companies, where camera image quality has a prominent place. Therefore, the quantification of image quality has not become less important, but rather more so.
One often hears the claim that image quality cannot be measured due to reasons such as its subjective nature, or the complexities involved. What we have realized over the years is that image quality measurements are indeed hard to perform, but not prohibitively so. This book will demonstrate this point through theoretical reasoning but also through examples. Furthermore, even with the development of new technologies, many of the traditional concepts and techniques remain valid today. They do need continuous development, but the effort spent on learning the important theories and methods is not wasted, and will instead help substantially in understanding many of the new technologies and how to develop metrics to cope with their particular quirks and intricacies. It is therefore our hope that this book will give the reader and benchmarking practitioner a good start in tackling the challenges that lie ahead in image quality camera benchmarking and characterization.
In order to provide even more useful and accurate content, we sought out notable authorities and peers in the field to both review and provide dialog on chapters related to their respective expertise. Of note, we appreciate and thank the following: Kjell Brunnström, Mark D. Fairchild, Harvey (Hervé) Hornung, Paul Hubel, Elaine W. Jin, Kenneth A. Parulski, Andreas von Sneidern, Nicolas Touchard, and Mats Wernersson.
We want to give a very special thanks to our contributing author, Hugh Denman, for providing invaluable knowledge and insights to the video-related sections spread throughout the book.
We have been given very valuable help by the Wiley staff, Ashmita Thomas Rajaprathapan and Teresa Netzler. We would also like to thank our commissioning editor, Alexandra Jackson, who helped us through the initial stages of the book proposal process and launch of writing.
We also want to thank DxO Labs for providing images as well as video examples for the electronic version of the book. Nicolas Touchard of DxO has been invaluable in supporting us, and also for providing so much extremely valuable feedback at the beginning of this project. We are grateful to the team at Imatest for allowing us to use images of their test charts and equipment, and to Dietmar Wüller at Image Engineering for many fruitful discussions and helpful comments as well as graciously letting us use images of their equipment.
Jonathan B. Phillips
Henrik Eliasson
Of course there are many others to thank along the way. For me, I first want to thank the outstanding colleagues from Kodak who have been my inspiration in so many ways. Starting out my professional career in Kodak's Image Science Career Development Program was a superb introduction to the world of imaging science. When contemplating this book project, I sought out the advice from instructors in that program who have been excellent advisors over the years: Brian W. Keelan and Edward J. Giorgianni. I thank my former Kodak managers, Kenneth A. Parulski and Brian E. Mittelstaedt, who provided the opportunities for me to branch into mobile imaging and the image quality standards efforts in ISO Technical Committee 42/Working Group 18 on photography and the IEEE CPIQ (Camera Phone Image Quality) initiative. I also thank NVIDIA and Google for continuing the sponsorship of my participation in ISO and CPIQ with support from Margaret Belska, Boyd A. Fowler, and Vint Cerf. In many ways, much of the content of this book is built on the dedicated expert efforts of the members and delegates of these image quality standards bodies. Additionally, fundamental to the writing of this book are the faculty and staff of the Munsell Color Science Lab and Center for Imaging Science at Rochester Institute of Technology, who were my instructors and guides as I spent over eight years there for my graduate work. Finally, I thank the many observers who have participated in countless subjective studies, without which we would not have the understanding of camera image quality benchmarking presented in this book.
Jonathan B. Phillips
San Jose, California
Writing this book would never have been possible if I hadn't started to become involved in imaging standards activities some ten years ago through the CPIQ initiative. For me, this has been a great learning experience, and I have enjoyed tremendously being a part of this community where I have met so many outstanding and great personalities over the years. I am thankful to my former employer and my managers there, Martin Ek, Pontus Nelderup, Fredrik Lönn, Per Hiselius, and Thomas Nilsson, for giving me the opportunity to participate in these activities. I would also like to thank my father, Lars Eliasson, for reviewing parts of the book and giving valuable feedback. Last, but in no way least, my wife, Monica, has played a big role in making this possible, not only by putting up with all the evenings I have spent by the computer, but also for giving...
System requirements
File format: ePUB
Copy protection: Adobe-DRM (Digital Rights Management)
System requirements:
- Computer (Windows; MacOS X; Linux): Install the free reader Adobe Digital Editions prior to download (see eBook Help).
- Tablet/smartphone (Android; iOS): Install the free app Adobe Digital Editions or the app PocketBook before downloading (see eBook Help).
- E-reader: Bookeen, Kobo, Pocketbook, Sony, Tolino and many more (not Kindle).
The file format ePub works well for novels and non-fiction books – i.e., „flowing” text without complex layout. On an e-reader or smartphone, line and page breaks automatically adjust to fit the small displays.
This eBook uses Adobe-DRM, a „hard” copy protection. If the necessary requirements are not met, unfortunately you will not be able to open the eBook. You will therefore need to prepare your reading hardware before downloading.
Please note: We strongly recommend that you authorise using your personal Adobe ID after installation of any reading software.
For more information, see our ebook Help page.