Case Study Research in Software Engineering: Guide lines and Examples

FOREWORD xiii PREFACE xv ACKNOWLEDGMENTS xvii PART I CASE STUDY METHODOLOGY 1 INTRODUCTION 3 1.1 What is a Case Study? 3 1.2 A Brief History of Case Studies in Software Engineering 5 1.3 Why a Book on Case Studies of Software Engineering? 6 1.4 Conclusion 9 2 BACKGROUND AND DEFINITION OF CONCEPTS 11 2.1 Introduction 11 2.2 Research Strategies 11 2.3 Characteristics of Research Strategies 13 2.3.1 Purpose 13 2.3.2 Control and Data 14 2.3.3 Triangulation 15 2.3.4 Replication 16 2.3.5 Inductive and Deductive Enquiries 16 2.4 What Makes a Good Case Study? 17 2.5 When is the Case Study Strategy Feasible? 19 2.6 Case Study Research Process 20 2.7 Conclusion 21 3 DESIGN OF THE CASE STUDY 23 3.1 Introduction 23 3.2 Elements of the Case Study Design 24 3.2.1 Rationale for the Study 24 3.2.2 Objective of the Study 24 3.2.3 Cases and Units of Analyses 26 3.2.4 Theoretical Framework 29 3.2.5 Research Questions 30 3.2.6 Propositions and Hypotheses 31 3.2.7 Concepts 32 3.2.8 Methods of Data Collection 32 3.2.9 Methods of Data Analysis 33 3.2.10 Case Selection 33 3.2.11 Selection of Data 35 3.2.12 Data Definition and Data Storage 36 3.2.13 Quality Control and Assurance 36 3.2.14 Maintaining the Case Study Protocol 37 3.2.15 Reporting and Disseminating the Case Study 38 3.3 Legal, Ethical, and Professional Issues 40 3.4 Conclusion 45 4 DATA COLLECTION 47 4.1 Introduction 47 4.2 Different Types of Data Source 47 4.2.1 Classification of Data Sources 47 4.2.2 Data Source Selection 49 4.3 Interviews 50 4.3.1 Planning Interviews 50 4.3.2 The Interview Session 52 4.3.3 Postinterview Activities 53 4.4 Focus groups 54 4.5 Observations 56 4.6 Archival Data 57 4.7 Metrics 58 4.8 Conclusion 60 5 DATA ANALYSIS AND INTERPRETATION 61 5.1 Introduction 61 5.2 Analysis of Data in Flexible Research 62 5.2.1 Introduction 62 5.2.2 Level of Formalism 64 5.2.3 Relation to Hypotheses 65 5.3 Process for Qualitative Data Analysis 65 5.3.1 Introduction 65 5.3.2 Steps in the Analysis 66 5.3.3 Techniques 68 5.3.4 Tool support 70 5.4 Validity 71 5.4.1 Construct Validity 71 5.4.2 Internal Validity 71 5.4.3 External Validity 71 5.4.4 Reliability 72 5.5 Improving Validity 72 5.6 Quantitative Data Analysis 74 5.7 Conclusion 76 6 REPORTING AND DISSEMINATION 77 6.1 Introduction 77 6.2 Why Report and Disseminate 78 6.3 The Audience for the Report 79 6.4 Aspects of the Case Study to Report and Disseminate 80 6.5 When to Report and Disseminate 81 6.6 Guidelines on Reporting 82 6.6.1 The Generic Content of an Academic Report 82 6.6.2 Reporting Recommendations from Evaluative Case Studies 84 6.6.3 Reporting to Stakeholders, Including Sponsor(s) 85 6.6.4 Reporting the Context of the Case Study 87 6.6.5 Reporting to Students 89 6.6.6 Ad Hoc and Impromptu Reporting 90 6.7 Formats and Structures for a Report 91 6.8 Where to Report 94 6.9 Ethics and Confidentiality 94 6.10 Conclusion 95 7 SCALING UP CASE STUDY RESEARCH TO REAL-WORLD SOFTWARE PRACTICE 97 7.1 Introduction 97 7.2 The Aims of Scaling up Case Studies 98 7.3 Dimensions of Scale 99 7.4 Longitudinal Case Studies 100 7.5 Multiple Case Studies 102 7.5.1 Multiple Cases and Replications 102 7.5.2 Selecting the Cases 104 7.6 Multiresearcher Case Studies 105 7.7 Conclusion 107 8 USING CASE STUDY RESEARCH 109 8.1 Introduction 109 8.2 Reading and Reviewing Case Studies 109 8.2.1 Development of Checklists 110 8.2.2 Checklists for Conducting Case Study Research 111 8.2.3 Checklists for Reading and Reviewing Case Studies 111 8.2.4 Development of Practice 111 8.3 Identifying and Synthesizing Use Case Research 111 8.3.1 Identifying Primary Studies 112 8.3.2 Synthesis of Evidence from Multiple Case Studies 113 8.3.3 Current State of Synthesis 117 8.4 The Economics of Case Study Research 118 8.4.1 Costs and Benefits of Evaluation Techniques 119 8.4.2 Evaluation of the DESMET Methodology 119 8.4.3 Frameworks for Organizing Methods of Evaluation 119 8.5 Specializing Case Study Research for Software Engineering 121 8.5.1 The Longitudinal Chronological Case Study Research Strategy 122 8.5.2 Controlled Case Studies 123 8.6 Case Studies and Software Process Improvement 123 8.7 Conclusion 125 PART II EXAMPLES OF CASE STUDIES 9 INTRODUCTION TO CASE STUDY EXAMPLES 129 9.1 Introduction 129 10 CASE STUDY OF EXTREME PROGRAMMING IN A STAGE-GATE CONTEXT 133 10.1 Introduction 133 10.1.1 Methodological Status 133 10.2 Case Study Design 134 10.2.1 Rationale 134 10.2.2 Objectives 134 10.2.3 Cases and Units of Analysis 135 10.2.4 Theoretical Frame of Reference 136 10.2.5 Research Questions 136 10.3 Planning 136 10.3.1 Methods of Data Collection 136 10.3.2 Selection of Data 137 10.3.3 Case Selection Strategy 137 10.3.4 Case Study Protocol 137 10.3.5 Ethical Considerations 137 10.4 Data Collection 139 10.5 Data Analysis 139 10.5.1 Threats to Validity 144 10.6 Reporting 144 10.6.1 Academics 144 10.6.2 Practitioners 144 10.7 Lessons Learned 146 11 TWO LONGITUDINAL CASE STUDIES OF SOFTWARE PROJECT MANAGEMENT 149 11.1 Introduction 149 11.2 Background to the Research Project 149 11.3 Case Study Design and Planning 150 11.3.1 Rationale 150 11.3.2 Objective 150 11.3.3 Definition of the Case 150 11.3.4 Units of Analyses 151 11.3.5 Theoretical Frame of Reference and Research Questions 151 11.3.6 Case Selection 151 11.3.7 Replication Strategy 152 11.3.8 Case Study Protocol 152 11.3.9 Quality Assurance, Validity, and Reliability 152 11.3.10 Legal, Ethical, and Professional Considerations 153 11.4 Data Collection 154 11.4.1 Sources of Data 154 11.5 Data Analysis 157 11.6 Reporting 159 11.6.1 Internal Reporting of Results 160 11.6.2 Dissemination of Artifacts 160 11.7 Lessons Learned 160 12 AN ITERATIVE CASE STUDY OF QUALITY MONITORING 163 12.1 Introduction 163 12.2 Case Study Design 164 12.2.1 Objectives 164 12.2.2 Cases and Units of Analysis 165 12.2.3 Theoretical Frame of Reference 165 12.2.4 Research Questions 165 12.3 Planning 165 12.3.1 Methods of Data Collection 165 12.3.2 Case Selection Strategy 167 12.3.3 Case Study Protocol 167 12.3.4 Ethical Considerations 167 12.3.5 Data Collection 168 12.3.6 Exploratory Study 168 12.3.7 Confirmatory Study 168 12.3.8 Explanatory Study 168 12.4 Data Analysis 169 12.5 Reporting 169 12.6 Lessons Learned 169 13 A CASE STUDY OF THE EVALUATION OF REQUIREMENTS MANAGEMENT TOOLS 171 13.1 Introduction 171 13.2 Design of the Case Study 172 13.2.1 Rationale 172 13.2.2 Objective 172 13.2.3 The Case and Its Context 173 13.2.4 The Units of Analyses 174 13.2.5 Theoretical Framework 175 13.2.6 Research Questions 175 13.2.7 Propositions, Concepts, and Measures 175 13.2.8 Case Study Protocol 175 13.2.9 Methods of Data Collection 176 13.2.10 Methods of Data Analysis 176 13.2.11 Case Selection Strategy 177 13.2.12 Data Selection Strategy 177 13.2.13 Replication Strategy 177 13.2.14 Quality Assurance, Validity, and Reliability 177 13.3 Data Collection 178 13.4 Data Analysis 179 13.5 Reporting and Dissemination 180 13.6 Lessons Learned 181 14 A LARGE-SCALE CASE STUDY OF REQUIREMENTS AND VERIFICATION ALIGNMENT 183 14.1 Introduction 183 14.2 Case Study Design 184 14.2.1 Rationale 184 14.2.2 Objectives 184 14.2.3 Cases and Units of Analysis 185 14.2.4 Theoretical Frame of Reference 186 14.2.5 Research Questions 187 14.3 Planning 188 14.3.1 Methods of Data Collection 189 14.3.2 Case Selection Strategy 190 14.3.3 Selection of Data 191 14.3.4 Case Study Protocol 191 14.3.5 Ethical Considerations 192 14.4 Data Collection 192 14.5 Data Analysis 193 14.6 Lessons Learned 195 14.6.1 Effort Estimation Lessons 195 14.6.2 Design and Planning Lessons 196 14.6.3 Data Collection Lessons 197 14.6.4 Data Analysis Lessons 198 14.6.5 Reporting Lessons 199 14.6.6 A General Lesson 199 EPILOGUE 201 Appendix A: CHECKLISTS FOR READING AND REVIEWING CASE STUDIES 203 A.1 Design of the Case Study 203 A.2 Data Collection 204 A.3 Data Analysis and Interpretation 204 A.4 Reporting and Dissemination 204 A.5 Reader's Checklist 205 Appendix B: EXAMPLE INTERVIEW INSTRUMENT (XP) 207 Appendix C: EXAMPLE INTERVIEW INSTRUMENT (REVV) 209 Appendix D: EXAMPLE OF A CODING GUIDE 213 D.1 Coding Instructions 213 D.2 Codes 214 D.2.1 High Level Codes: Research Questions 214 D.2.2 Medium Level Codes: Categories 216 D.2.3 Coding Example 216 Appendix E: EXAMPLE OF A CONSENT INFORMATION LETTER 219 REFERENCES 221 INDEX 235