Fundamentals of Software Testing
Description
Software testing has greatly evolved since the first edition of this book in 2011. Testers are now required to work in "agile" teams and focus on automating test cases. It has thus been necessary to update this work, in order to provide fundamental knowledge that testers should have to be effective and efficient in today's world.
This book describes the fundamental aspects of testing in the different lifecycles, and how to implement and benefit from reviews and static analysis. Multiple other techniques are approached, such as equivalence partitioning, boundary value analysis, use case testing, decision tables and state transitions.
This second edition also covers test management, test progress monitoring and incident management, in order to ensure that the testing information is correctly provided to the stakeholders.
This book provides detailed course-study material for the 2023 version of the ISTQB Foundation level syllabus, including sample questions to help prepare for exams.
More details
Other editions
Additional editions
Person
Bernard Homès, founder of the ISTQB and an IEEE Senior member, has over 40 years of experience in IT and software testing. He headed the design for the Advanced Level syllabus and holds numerous testing certifications. This is his eighth book on software testing, which will once more focus on helping customers to improve the efficiency of their testing processes.
Content
Preface xi
Glossary xvii
Chapter 1 Fundamentals of Testing 1
1.1 What is testing? 1
1.1.1 Software and systems context 1
1.1.2 Causes of software defects 3
1.1.3 Role of testing in software development, maintenance and operations 5
1.1.4 Tests and quality 5
1.1.5 Terminology 7
1.2 What is testing? 8
1.2.1 Origin of defects 8
1.2.2 Common goals of testing 9
1.2.3 Examples of objectives for testing 9
1.2.4 Test and debugging 10
1.3 Paradoxes and main principles 11
1.3.1 Testing identifies the presence of defects 11
1.3.2 Exhaustive testing is impossible 11
1.3.3 Early testing 12
1.3.4 Defect clustering 12
1.3.5 Pesticide paradox 13
1.3.6 Testing is context dependent 13
1.3.7 Absence-of-errors fallacy 14
1.4 Test activities, testware and test roles 14
1.4.1 Planning 15
1.4.2 Monitoring and control 16
1.4.3 Test analysis and design 17
1.4.4 Test implementation 20
1.4.5 Test execution 21
1.4.6 Reporting 23
1.4.7 Test completion activities 24
1.4.8 The value of traceability 25
1.4.9 Impact of context on the test process 25
1.5 Roles in testing 26
1.6 Essential skills and "good practices" in testing 26
1.6.1 Generic skills 26
1.6.2 Specific skills 27
1.6.3 Whole team approach 27
1.6.4 Independence of testing 28
1.6.5 Levels of independence 29
1.6.6 Adapt to objectives 30
1.6.7 Destructive or constructive? 32
1.6.8 People skills 32
1.6.9 Change of perspective 33
1.7 Testers and code of ethics (FL 1.6) 33
1.7.1 Public 34
1.7.2 Client and employer 35
1.7.3 Product 35
1.7.4 Judgment 37
1.7.5 Management 37
1.7.6 Profession 38
1.7.7 Colleagues 39
1.7.8 Self 40
1.8 Sample exam questions 41
Chapter 2 Testing Throughout the Software Life Cycle 47
2.1 Testing through the software development life cycle 47
2.1.1 Sequential models 48
2.1.2 Iterative models 50
2.1.3 Incremental model 52
2.1.4 RAD 54
2.1.5 Agile models 55
2.1.6 Selection of a development model 61
2.1.7 Positioning tests 62
2.1.8 Test-first and shift-left approaches 63
2.2 Test levels and test types 64
2.2.1 Component-level testing or component tests 65
2.2.2 Integration-level testing or integration tests 66
2.2.3 System tests 68
2.2.4 Acceptance tests 70
2.2.5 Other levels 72
2.3 Types of tests 72
2.3.1 Functional tests 73
2.3.2 Nonfunctional tests 74
2.3.3 Tests based on the structure or architecture of the software 76
2.3.4 Tests associated with changes 77
2.3.5 Comparisons and examples 79
2.4 Test and maintenance 80
2.4.1 Maintenance context 81
2.4.2 Evolutive maintenance 81
2.4.3 Corrective maintenance 82
2.4.4 Retirement and replacement 83
2.4.5 Regression test policies 83
2.4.6 SLA validation and acceptance 86
2.5 Oracles 86
2.5.1 Problems with oracles 87
2.5.2 Sources of oracles 87
2.5.3 Oracle usage 88
2.6 Process improvements 89
2.6.1 Objectives 89
2.6.2 Measurements 89
2.6.3 Retrospectives and improvements 89
2.7 Specific cases 90
2.7.1 Performance tests 90
2.7.2 Maintainability tests 91
2.8 Sample exam questions 91
Chapter 3 Static Testing 97
3.1 Static techniques and the test process 97
3.2 Review process 100
3.2.1 Types of reviews 101
3.2.2 Roles and responsibilities during reviews 106
3.2.3 Phases of reviews 109
3.2.4 Success factors for reviews 122
3.2.5 Comparison of the types of reviews 123
3.3 Static analysis by tools 125
3.3.1 Types of static analysis 125
3.3.2 Types of defects that can be identified 130
3.3.3 Data flow analysis 131
3.4 Added value of static activities 136
3.5 Sample exam questions 137
Chapter 4 Test Design Techniques 141
4.1 The test development process 143
4.1.1 Terminology 143
4.1.2 Traceability 144
4.2 Categories of test design techniques 146
4.2.1 Black box, white box or gray box 147
4.2.2 Experience-based techniques 148
4.2.3 Test characteristics 149
4.2.4 Limitations and assumptions 149
4.3 Black-box techniques 151
4.3.1 Equivalence partitioning 152
4.3.2 Boundary value analysis 158
4.3.3 Decision tables 161
4.3.4 Other combinational techniques 166
4.3.5 State transition testing 166
4.3.6 Use case testing 173
4.3.7 Limitations and assumptions 175
4.4 Structure-based techniques 175
4.4.1 Statement testing and coverage 178
4.4.2 Decision testing and coverage 183
4.4.3 Other structure-based techniques 186
4.4.4 MC/DC coverage 188
4.4.5 Limitations and assumptions of structure-based testing 189
4.4.6 Coverage level and exit criteria 190
4.5 Experience-based technique 190
4.5.1 Attacks 191
4.5.2 Defect taxonomies 192
4.5.3 Error guessing and ad hoc testing 193
4.5.4 Exploratory testing 194
4.5.5 Limitations and assumptions 195
4.6 Collaboration-based test approaches 196
4.6.1 Collaborative user stories 196
4.6.2 Acceptance criteria 197
4.6.3 Acceptance test-driven development 197
4.7 Choosing test techniques 198
4.8 Sample exam questions 200
Chapter 5 Test Management 209
5.1 Test organization 209
5.1.1 Independence levels 209
5.1.2 Roles and responsibilities 212
5.1.3 Human and contractual aspects 214
5.2 Test planning and estimation 215
5.2.1 Planning and evaluation activities 218
5.2.2 Test planning activities 229
5.2.3 Test documentation 231
5.2.4 Entry and exit criteria for test activities 236
5.3 Test progress monitoring and control (FL 5.3) 240
5.4 Reporting 243
5.4.1 What to report, to whom and how? 243
5.4.2 Statistics and graphs 245
5.5 Transverse processes and activities 248
5.5.1 Test data definition 248
5.5.2 Configuration management (FL 5.4) 249
5.5.3 Change management 250
5.6 Risk management (FL 5.2) 250
5.6.1 Principles of risk management 251
5.6.2 Project risks and product risks 255
5.6.3 Introduction to risk management 256
5.7 Defect management (FL 5.5) 259
5.7.1 Introduction to defect management 260
5.7.2 Defect identification 261
5.7.3 Actions applied to defects 266
5.7.4 Defect disposition 266
5.8 Sample exam questions 267
Chapter 6 Tools Support for Testing 277
6.1 Types of test tools 277
6.1.1 Test tool classification 278
6.1.2 Tools supporting test management 278
6.1.3 Tools supporting requirement management 279
6.1.4 Tools supporting static tests 279
6.1.5 Modeling tools 280
6.1.6 Tools supporting test design and test data creation 280
6.1.7 Tools supporting test execution 281
6.1.8 Tools supporting test environment management 281
6.1.9 Tools supporting test data comparison 282
6.1.10 Tools supporting test coverage measurement 282
6.1.11 Other test supporting tools 282
6.2 Assumptions and limitations of test tools 283
6.2.1 Advantages and risks of the tools 283
6.2.2 Specific considerations for some tools 285
6.3 Selecting and introducing tools in an organization 289
6.3.1 Main principles 289
6.3.2 Tool selection process 290
6.3.3 Test tool implementation 293
6.3.4 To build or to buy test tools? 294
6.4 Sample exam questions 295
Chapter 7 Mock Exam 299
Chapter 8 Templates and Models 313
8.1 Master test plan 313
8.2 Test plan 315
8.2.1 Test plan as per IEEE 829-1998 315
8.2.2 Test plan as per IEEE 829-2008 315
8.3 Test design document 317
8.3.1 Test design specifications as per IEEE 829-1998 317
8.3.2 Test design document as per IEEE 829-2008 317
8.4 Test case 318
8.4.1 Test case document as per IEEE 829-1998 318
8.4.2 Test case document as per IEEE 829-2008 318
8.5 Test procedure 319
8.5.1 Test procedure document as per IEEE 829-1998 319
8.5.2 Test procedure document as per IEEE 829-2008 319
8.6 Test log 320
8.6.1 Test log as per IEEE 829-1998 320
8.6.2 Test log as per IEEE 829-2008 320
8.7 Defect report 320
8.7.1 Defect report as per IEEE 829-1998 320
8.7.2 Defect report as per IEEE 829-2008 321
8.8 Test report 322
8.8.1 Test report as per IEEE 829-1998 322
8.8.2 Interim test report as per IEEE 829-2008 322
8.8.3 Level test report as per IEEE 829-2008 323
8.8.4 Master test report as per IEEE 829-2008 323
Chapter 9 Answers to the Questions 325
9.1 Answers to the end-of-chapter questions 325
9.2 Correct answers to the sample paper questions 327
References 329
Index 333
Glossary
The definitions listed below have been extracted from the International Software Testing Qualifications Board (ISTQB) Standard Glossary of Terms used in Software Testing. Only the terms used for the Foundation Level certification exams are mentioned, so as not to drown the reader in terms that are used at other levels or in other syllabi.
Acceptance criteria: The criteria that a component or system must satisfy in order to be accepted by a user, customer or other authorized entity (from ISO 24765).
Acceptance test-driven development: A collaboration-based test-first approach that defines acceptance tests in the stakeholders' domain language. Abbreviation: ATDD.
Acceptance testing: Formal testing with respect to user needs, requirements and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entities to determine whether or not to accept the system. See also user acceptance testing.
ACM: (Association for Computer Machinery) professional and scientific association for the development of information technology.
Alpha testing: Simulated or actual operational testing by potential users/customers or an independent test team at the developers' site, but outside the development organization. Alpha testing is often employed as a form of internal acceptance testing.
Anomaly: A condition that deviates from expectation (from ISO 24765).
Attack: Directed and focused attempt to evaluate the quality, and especially the reliability, of a test object by attempting to force specific failures to occur.
Beta testing: Operational testing by potential and/or existing users/customers at an external site not otherwise involved with the developers, to determine whether or not a component or system satisfies the user/customer needs and fits within the business processes. Beta testing is often employed as a form of external acceptance testing in order to acquire feedback from the market.
Black-box technique: See also black-box testing.
Black-box test technique: A test technique based on an analysis of the specification of a component or system. Synonyms: black-box test design technique, specification-based test technique.
Black-box testing: Testing, either functional or nonfunctional, based on an analysis of the specification of the component or system. Synonym: specification-based testing.
Boundary value analysis: A black-box test technique in which test cases are designed based on boundary values. See also boundary values.
Branch coverage: The coverage of branches in a control flow graph (percentage of branches that have been exercised by a test suite). One hundred percent branch coverage implies both 100% decision coverage and 100% statement coverage.
Bug: See also defect.
Checklist-based testing: An experience-based test technique in which test cases are designed to exercise the items of a checklist.
Code coverage: An analysis method that determines which parts of the software have been executed (covered) by the test suite and which parts have not been executed, for example, statement coverage, decision coverage or condition coverage.
Collaboration-based test approach: An approach to testing that focuses on defect avoidance by collaborating among stakeholders.
Commercial off-the-shelf software (COTS): See also off-the-shelf software.
Compiler: A software tool that translates programs expressed in a high-order language into their machine language equivalents.
Complexity: The degree to which a component or system has a design and/or internal structure that is difficult to understand, maintain and verify. See also cyclomatic complexity.
Component integration testing: The testing executed to identify defects in the interfaces and interactions between integrated components. Synonyms: module integration testing, unit integration testing.
Component testing: A test level that focuses on individual hardware or software components. Synonyms: module testing, unit testing.
Configuration control: An element of configuration management, consisting of the evaluation, coordination, approval or disapproval, and implementation of changes to configuration items after formal establishment of their configuration identification.
Configuration item: An aggregation of hardware, software or both, that is designated for configuration management and treated as a single entity in the configuration management process.
Configuration management: A discipline applying technical and administrative direction and surveillance to: identify and document the functional and physical characteristics of a configuration item, control changes to those characteristics, record and report change processing and implementation status, and verify compliance with specified requirements.
Confirmation testing: A type of change-related testing performed after fixing a defect to confirm that a failure caused by that defect does not reoccur. Synonym: re-testing.
Control flow: An abstract representation of all possible sequences of events (paths) in the execution through a component or system.
Coverage: The degree to which specified coverage items are exercised by a test suite, expressed as a percentage. Synonym: test coverage.
Coverage item: An attribute or combination of attributes derived from one or more test conditions by using a test technique. See also coverage criteria.
Coverage measurement tool: See also coverage tool.
Coverage tool: A tool that provides objective measures of what structural elements, for example, statements, branches, have been exercised by the test suite.
Cyclomatic complexity: The number of independent paths through a program. Cyclomatic complexity is defined as: L - N + 2P, where:
- L = the number of edges/links in a graph;
- N = the number of nodes in a graph;
- P = the number of disconnected parts of the graph (e.g. a calling graph and a subroutine).
Data-driven testing: A scripting technique that stores test input and expected results in a table or spreadsheet, so that a single control script can execute all of the tests in the table. Data-driven testing is often used to support the application of test execution tools such as capture/playback tools. See also keyword-driven testing.
Data flow: An abstract representation of the sequence and possible changes of the state of data objects, where the state of an object can be creation, usage or destruction.
Debugging: The process of finding, analyzing and removing the causes of failures in a component or system.
Debugging tool: A tool used by programmers to reproduce failures, investigate the state of programs and find the corresponding defect. Debuggers enable programmers to execute programs step-by-step, halt a program at any program statement and set and examine program variables.
Decision coverage: The percentage of decision outcomes that have been exercised by a test suite. One hundred percent decision coverage implies both 100% branch coverage and 100% statement coverage.
Decision table testing: A black-box test technique in which test cases are designed to exercise the combinations of conditions inputs and/or stimuli (causes) shown in a decision table.
Defect: An imperfection or deficiency in a work product, which can cause the component or system to fail to perform its required function, for example, an incorrect statement or data definition (from ISO 24765). A defect, if encountered during execution, may cause a failure of the component or system. Synonyms: bug, fault.
Defect density: The number of defects identified in a component or system divided by the size of the component or system (expressed in standard measurement terms, for example, lines-of-code, number of classes or function points).
Defect management: The process of recognizing, recording, classifying, investigating, resolving and disposing of defects. It involves recording defects, classifying them and identifying the impact.
Defect management tool: See also incident management tool.
Defect report: Documentation of the occurrence, nature and status of a defect. Synonym: bug report.
Driver: A software component or test tool that replaces a component that takes care of the control and/or the calling of a component or system.
Dynamic analysis tool: A tool that provides run-time information on the state of the software code. These tools are most commonly used to identify unassigned pointers, check pointer arithmetic, and monitor the allocation, use and de-allocation of memory and highlight memory leaks.
Dynamic testing: Testing that involves the execution of the test item/software of a component or system (from ISO 29119-1). See...
