Software Testing: Principles and Practices

PART I TESTING METHODOLOGY ; 1. Introduction to Effective Software Testing ; 1.1 Introduction ; 1.2 Evolution of Software Testing ; 1.3 Software Testing Myths ; 1.4 Goals of Software Testing ; 1.5 Psychology for Software Testing ; 1.6 Software Testing Definitions ; 1.7 Model for Software Testing ; 1.8 Effective Software Testing vs Exhaustive Software Testing ; 1.9 Effective Testing is hard ; 1.10 Software Testing as a Process ; 1.11 Schools of Software Testing ; 1.12 Software Failure Case Studies ; Summary ; 2. Software Testing Terminology and Methodology ; 2.1 Software Testing Terminology ; 2.1.1 Definitions ; 2.1.2 Life Cycle of a Bug ; 2.1.3 States of a Bug ; 2.1.4 Why do bugs occur? ; 2.1.5 Bug affects economics of software testing ; 2.1.6 Bug Classification based on Criticality ; 2.1.7 Bug Classification based on SDLC ; 2.1.8 Testing Principles ; 2.2 Software Testing Life Cycle (STLC) ; 2.3 Software Testing Methodology ; 2.3.1 Software Testing Strategy ; 2.3.2 Test Strategy Matrix ; 2.3.3 Development of Test Strategy ; 2.3.4 Testing Life Cycle Model ; 2.3.4.1 V Testing life cycle model ; 2.3.5 Validation Activities ; 2.3.6 Testing Tactics ; 2.3.7 Considerations in developing testing methodologies ; Summary ; 3. Verification and Validation ; 3.1 Verification and Validation Activities ; 3.2 Verification ; 3.2.1 Verification Activities ; 3.3 Verification of Requirements ; 3.3.1 Verification of Objectives ; 3.3.2 How to verify Requirements and Objectives ; 3.4 Verification of High level Design ; 3.4.1 How to verify High level Design? ; 3.4.4.1 Verification of Data Design ; 3.4.1.2 Verification of Architectural Design ; 3.4.1.3 Verification of User Interface Design ; 3.5 Verification of Low level Design ; 3.5.1 How to verify Low level Design? ; 3.6 How to verify Code? ; 3.6.1 Unit Verification ; 3.7 Validation ; 3.7.1 Validation Activities ; Summary ; PART II TESTING TECHNIQUES ; 4. Dynamic Testing : Black Box Testing Techniques ; 4.1 Boundary Value Analysis ; 4.1.1 Boundary value checking ; 4.1.2 Robustness Testing method ; 4.1.3 Worst Case Testing method ; 4.2 Equivalence Class Testing ; 4.2.1 Identification of Equivalence classes ; 4.2.2 Identifying the Test cases ; 4.3 State Table based Testing ; 4.3.1 Finite State Machine ; 4.3.2 State Graphs ; 4.3.3 State Tables ; 4.3.4 State table based testing ; 4.4 Decision Table based Testing ; 4.4.1 Formation of Decision Table ; 4.4.2 Test case design using decision table ; 4.4.3 Expanding the immaterial test cases in decision table ; 4.5 Cause Effect Graphing based Testing ; 4.5.1 Basic notations ; 4.6 Error Guessing ; Summary ; 5. Dynamic Testing : White Box Testing Techniques ; 5.1 Need of White box testing ; 5.2 Logic Coverage Criteria ; 5.3 Basis Path Testing ; 5.3.1 Control Flow Graph ; 5.3.2 Flow graph notations of different programming constructs ; 5.3.3 Path Testing Terminology ; 5.3.4 Cyclomatic Complexity ; 5.3.4.1 Formulae based on Cyclomatic complexity ; 5.3.4.2 Guidelines for Basis Path Testing ; 5.3.5 Applications of Path Testing ; 5.4 Graph Matrices ; 5.4.1 Graph Matrix ; 5.4.2 Connection Matrix ; 5.4.3 Use of connection matrix in finding cyclomatic complexity number ; 5.4.4 Use of graph matrix for finding the set of all paths ; 5.5 Loop Testing ; 5.6 Data Flow Testing ; 5.6.1 State of a Data Object ; 5.6.2 Data Flow Anomalies ; 5.6.3 Terminology used in Data Flow Testing ; 5.6.4 Static Data flow testing ; 5.6.4.1 Static analysis is not enough ; 5.6.5 Dynamic Data flow testing ; 5.6.6 Ordering of Data flow testing strategies ; 5.8 Mutation Testing ; 5.8.1 Primary Mutants ; 5.8.2 Secondary Mutants ; 5.8.3 Mutation Testing Process ; Summary ; 6. Static Testing ; 6.1 Inspections ; 6.1.1 Inspection Team ; 6.1.2 Inspection Process ; 6.1.3 Benefits of Inspection Process ; 6.1.4 Effectiveness of Inspection Process ; 6.1.5 Cost of Inspection Process ; 6.1.6 Variants of Inspection process ; 6.1.7 Reading Techniques ; 6.1.8 Checklists for Inspection Process ; 6.2 Walkthroughs ; 6.3 Technical Reviews ; Summary ; 7. Validation Activities ; 7.1 Unit Validation Testing ; 7.2 Integration Testing ; 7.2.1 Decomposition Based Integration ; 7.2.1.1 Types of Incremental Integration Testing ; 7.2.1.2 Comparison between Top-Down and Bottom-Up Integration Testing ; 7.2.1.3 Practical Approach for Integration Testing ; 7.2.2 Call Graph Based Integration ; 7.2.2.1 Pair-wise Integration ; 7.2.2.2 Neighborhood Integration ; 7.2.3 Path Based Integration ; 7.3 Function Testing ; 7.4 System Testing ; 7.4.1 Categories of Systems Tests ; 7.4.1.1 Recovery Testing ; 7.4.1.2 Security Testing ; 7.4.1.3 Performance Testing ; 7.4.1.4 Load Testing ; 7.4.1.5 Stress Testing ; 7.4.1.6 Usability Testing ; 7.4.1.7 Compatibility/Conversion/Configuration Testing ; 7.4.2 Guidelines for performing the system tests ; 7.5 Acceptance Testing ; 7.5.1 Alpha Testing ; 7.5.1.1 Entry to Alpha ; 7.5.1.2 Exit to Alpha ; 7.5.2 Beta Testing ; 7.5.2.1 Entry to Beta ; 7.5.2.2 Exit criteria ; Summary ; 8. Regression Testing ; 8.1 Progressive vs Regression Testing ; 8.2 Regression testing produces quality software ; 8.3 Regression Testability ; 8.4 Objectives of Regression Testing ; 8.5 When to do regression testing? ; 8.6 Regression Testing Types ; 8.7 Defining Regression Test Problem ; 8.7.1 Regression Testing is a problem? ; 8.7.2 Regression Testing Problem ; 8.8 Regression Testing Techniques ; 8.8.1 Selective Retest Techniques ; 8.8.1.1 Strategy for Test Case Selection ; 8.8.1.2 Selection Criteria Based on Code ; 8.8.1.3 Regression Test selection Techniques ; 8.8.1.4 Evaluating Regression Test Selection Techniques ; 8.8.2 Regression Test Prioritization ; 8.9 Benefits of Regression Testing ; Summary ; PART III MANAGING THE TEST PROCESS ; 9. Test Management ; 9.1 Test Organization ; 9.2 Structure of Testing Group ; 9.3 Test Planning ; 9.3.1 Test Plan Components ; 9.3.2 Test Plan Hierarchy ; 9.3.3 Master Test Plan ; 9.3.4 Verification Test Plan ; 9.3.5 Validation Test Plan ; 9.3.5.1 Unit Test Plan ; 9.3.5.2 Integration Test Plan ; 9.3.5.3 Function Test Plan ; 9.3.5.4 System Test Plan ; 9.3.5.5 Acceptance Test Plan ; 9.4 Detailed Test Design and Test Specifications ; 9.4.1 Test Design Specifications ; 9.4.2 Test Case Specifications ; 9.4.3 Test Procedure Specifications ; 9.4.4 Test Result Specifications ; 9.4.4.1 Test Log ; 9.4.4.2 Test Incident Report ; 9.4.4.3 Test Summary Report ; Summary ; 10. Software Metrics ; 10.1 Need of Software Measurement ; 10.2 Definition of Software Metrics ; 10.3 Classification of Software Metrics ; 10.3.1 Product vs. Process Metrics ; 10.3.2 Objective vs. Subjective Metrics ; 10.3.3 Primitive vs. Computed Metrics ; 10.3.4 Private vs. Public Metrics ; 10.4 Entities to be measured ; 10.5 Size Metrics ; 10.5.1 Line of Code (LOC) ; 10.5.2 Token Count (Halstead Product Metrics) ; 10.5.2.1 Program Vocabulary ; 10.5.2.2 Program Length ; 10.5.2.3 Program Volume ; 10.5.3 Function Point Analysis ; 10.5.3.1 Process used for calculating Function Points ; 10.5.3.2 Sizing Data Functions ; 10.5.3.3 Sizing Transactional Functions ; 10.5.3.4 Calculating Unadjusted Function Point (UFP) ; 10.5.3.5 Calculating Adjusted Function Point ; Summary ; 11. Testing Metrics for Monitoring and Controlling the Testing Process ; 11.1 Measurement Objectives for Testing ; 11.2 Attributes and Corresponding Metrics in Software Testing ; 11.3 Attributes ; 11.3.1 Progress ; 11.3.2 Cost ; 11.3.3 Quality ; 11.3.4 Size ; 11.4 Estimation models for estimating testing efforts ; 11.4.1 Halstead metrics for estimating testing effort ; 11.4.2 Development ratio method ; 11.4.3 Project staff ratio method ; 11.4.4 Test procedure method ; 11.4.5 Task planning method ; 11.5 Architectural Design Metric used for Testing ; 11.6 Information Flow Metrics used for Testing ; 11.6.1 Henry & Kafura Design Metric ; 11.7 Cyclomatic Complexity measures for Testing ; 11.8 Function Point Metrics for Testing ; 11.9 Test Point Analysis ; 11.9.1 Procedure for Calculating TPA ; 11.9.2 Calculating Dynamic Test Points ; 11.9.3 Calculating Static Test Points ; 11.9.4 Calculating Primary Test Hours ; 11.9.5 Calculating Total Test Hours ; 11.10 Some Testing Metrics ; Summary ; 12. Efficient Test Suite Management ; 12.1 Why Test Suite grows? ; 12.2 Minimizing the test suite and its benefits ; 12.3 Defining the Test Suite Minimization Problem ; 12.4 Test Suite Prioritization ; 12.5 Types of Test case Prioritization ; 12.6 Prioritization Techniques ; 12.6.1 Coverage based Test Case Prioritization ; 12.6.1.1Total Statement Coverage Prioritization ; 12.6.1.2 Additional Statement Coverage Prioritization ; 12.6.1.3 Total Branch Coverage Prioritization ; 12.6.1.4 Additional Branch Coverage Prioritization ; 12.6.1.5 Total Fault-Exposing-Potential (FEP) Prioritization ; 12.6.2 Risk based Prioritization ; 12.6.3 Prioritization based on Operational Profiles ; 12.6.4 Prioritization using Relevant Slices ; 12.6.4.1 Execution Slice ; 12.6.4.2 Dynamic Slice ; 12.6.4.3 Relevant Slice ; 12.6.5 Prioritization based on Requirements ; 12.7 Measuring Effectiveness of Prioritized Test Suite ; Summary ; PART IV QUALITY MANAGEMENT ; 13. Software Quality Management ; 13.1 Software Quality ; 13.2 Quality Types ; 13.3 Broadening the concept of Quality ; 13.4 Quality Cost ; 13.5 Benefits of Investment on Quality ; 13.6 Quality Control ; 13.7 Quality Assurance ; 13.8 Quality management ; 13.9 QM and Project management ; 13.10 Quality factors ; 13.11 Methods of Quality Management ; 13.11.1 Procedural Approach to QM ; 13.11.2 Quantitative Approach to QM ; 13.12 Software Quality Metrics ; Summary ; 14. Testing Process Maturity Models ; 14.1 Need for Test Process Maturity ; 14.2 Measurement and Improvement of a Test Process ; 14.3 Test Process Maturity Models ; 14.3.1 The Testing Improvement Model (TIM) ; 14.3.1.1 Maturity Model ; 14.3.1.2 Key areas of TIM ; 14.3.1.3 The Assessment procedure of TIM ; 14.3.2 Test Organization Model (TOM) ; 14.3.2.1 Questionnaire ; 14.3.2.2 Improvement Suggestions ; 14.3.3 Test Process Improvement (TPI) Model ; 14.3.3.1 Key Process Areas ; 14.3.3.2 Maturity Levels ; 14.3.3.3 Test Maturity Matrix ; 14.3.3.4 Checkpoints ; 14.3.3.5 Improvement Suggestions ; 14.3.4 Test Maturity Model (TMM) ; 14.3.4.1 TMM Components ; 14.3.4.2 TMM Levels ; 14.3.4.3 The Assessment Model ; Summary ; PART V TEST AUTOMATION ; 15. Automation and Testing Tools ; 15.1 Need of Automation ; 15.2 Categorization of Testing Tools ; 15.2.1 Static and Dynamic Testing Tools ; 15.2.2 Testing Activity Tools ; 15.3 Selection of Testing Tools ; 15.4 Costs incurred in Testing Tools ; 15.5 Guidelines for Automated Testing ; 15.6 Overview of some commercial Testing Tools ; Summary ; PART VI TESTING FOR SPECIALIZED ENVIRONMENT ; 16. Testing Object Oriented Software ; 16.1 OOT and Structured Approach ; 16.1.1 Terminology ; 16.1.2 Object-Oriented Modelling and UML ; 16.2 Object Oriented Testing ; 16.2.1 Differences between Conventional testing Object oriented Testing ; 16.2.2 Object-Oriented Testing and Maintenance Problems ; 16.2.3 Issues in OO Testing ; 16.2.4 Strategy and Tactics of Testing OOS ; 16.2.5 Verification of OOS ; 16.2.6 Validation Activities ; 16.2.7 Testing of OO Classes ; 16.2.7.1 Feature based Testing of Classes ; 16.2.7.2 Role of Invariants in Class Testing ; 16.2.7.3 State Based Testing ; 16.2.8 Inheritance Testing ; 16.2.8.1 Issues in Inheritance Testing ; 16.2.8.2 Inheritance of Invariants of Base Class ; 16.2.8.3 Incremental Testing ; 16.2.9 Integration Testing ; 16.2.9.1 Thread based Integration Testing ; 16.2.9.2 Implicit Control Flow based Integration Testing ; 16.2.10 UML based OO Testing ; 16.2.10.1 UML diagrams in software testing ; 16.2.10.2 System Testing based on Use cases ; Summary ; 17. Testing Web based Systems ; 17.1 Web-based System ; 17.2 Web Technology Evolution ; 17.2.1 First Generation/ 2-tier Web system ; 17.2.2 Modern 3-tier & n-tier architecture ; 17.3 Differences between Traditional Software and Web-based Software ; 17.4 Challenges in Testing for Web-based Software ; 17.5 Quality Aspects ; 17.6 Web Engineering (WebE) ; 17.6.1 Analysis and Design of Web-based systems ; 17.6.1.1 Conceptual Modeling ; 17.6.1.2 Navigation Modeling ; 17.6.1.3 Presentation Modeling ; 17.6.1.4 Web Scenarios Modeling ; 17.6.1.5 Task Modeling ; 17.6.1.6 Configuration Modeling ; 17.6.2 Design Activities ; 17.6.2.1 Interface Design ; 17.6.2.2 Content Design ; 17.6.2.3 Architecture Design ; 17.6.2.4 Presentation Design ; 17.6.2.5 Navigation design ; 17.7 Testing of Web-Based Systems ; 17.7.1 Interface Testing ; 17.7.2 Usability Testing ; 17.7.3 Content Testing ; 17.7.4 Navigation Testing ; 17.7.5 Configuration/Compatibility Testing ; 17.7.6 Security Testing ; 17.7.6.1 Security Test Plan ; 17.7.6.2 Various Threat Types and their corresponding Test cases ; 17.7.7 Performance Testing ; 17.7.7.1 Performance Parameters ; 17.7.7.2 Types of Performance Testing ; Summary ; PART VII TRACKING THE BUG ; 18. Debugging ; 18.1 Debugging: An art or technique? ; 18.2 Debugging Process ; 18.3 Debugging is Difficult? ; 18.4 Debugging Techniques ; 18.4.1 Debugging with Memory Dump ; 18.4.2 Debugging with Watch Points ; 18.4.3 BackTracking ; 18.5 Correcting the Bugs ; 18.5.1 Debugging Guidelines ; 18.6 Debuggers ; 18.6.1 Types of Debuggers ; Summary ; CASE STUDIES ; REFERENCES ; APPENDICES ; APPENDIX AANSWERS TO MULTIPLE CHOICE QUESTIONS ; APPENDIX A SRS VERIFICATION CHECKLIST ; APPENDIX BHLD VERIFICATION CHECKLIST ; APPENDIX CLLD VERIFICATION CHECKLIST ; APPENDIX D GENERAL SDD VERIFICATION CHECKLIST ; APPENDIX EGENERIC CODE VERIFICATION CHECKLIST ; INDEX