Software Testing
2nd Edition
Published on 4. August 2005
Book
Paperback/Softback
416 pages
978-0-672-32798-8 (ISBN)
Description
Software testing is one of the invisible jobs in the software industry. Everyone has heard of computer programmers but few people realize there are nearly as many people behind the scenes with job titles such as Software Tester, Software Quality Assurance Engineer, Software Test Engineer, and Software Test Technician. Microsoft alone hires hundreds of people for these positions each year. There are also many companies whose sole purpose is providing software test consulting and software testing services. The first edition of Software Testing was published in November 2000. Although the processes and techniques used in testing computer software are timeless, this title will be brought up-to-date by adding a chapter that specifically deals with testing software for security bugs and revisiting the rest of the book to update examples and references.
More details
Other editions
Previous edition
Person
Ron Patton is a software consultant living in Washington State. His software test experience is wide and varied from mission critical systems to painting programs for kids. In 1992 he joined Microsoft as a Software Test Lead in the Systems Group for Multimedia Viewer, the authoring tool and multimedia display engine used by Encarta, Cinemania, and Bookshelf. He moved on to become the Software Test Manager of the Kids Product Unit. Most recently, he was the Software Test Manager of the Microsoft Hardware Group responsible for the software shipped with the mouse, keyboard, gaming, telephony, and ActiMates product lines.
Content
Introduction.
About the Second Edition
Who Should Use This Book?
What This Book Will Do for You
Software Necessary to Use This Book
How This Book Is Organized
Part I: The Big Picture
Part II: Testing Fundamentals
Part III: Applying Your Testing Skills
Part IV: Supplementing Your Testing
Part V: Working with Test Documentation
Part VI: The Future
Appendix
Conventions Used in This Book
I. THE BIG PICTURE.
1. Software Testing Background.
Infamous Software Error Case Studies
Disney's Lion King, 1994-1995
Intel Pentium Floating-Point Division Bug, 1994
NASA Mars Polar Lander, 1999
Patriot Missile Defense System, 1991
The Y2K (Year 2000) Bug, circa 1974
Dangerous Viewing Ahead, 2004
What Is a Bug?
Terms for Software Failures
Software Bug: A Formal Definition
Why Do Bugs Occur?
The Cost of Bugs
What Exactly Does a Software Tester Do?
What Makes a Good Software Tester?
Summary
Quiz
2. The Software Development Process.
Product Components
What Effort Goes Into a Software Product?
What Parts Make Up a Software Product?
Software Project Staff
Software Development Lifecycle Models
Big-Bang Model
Code-and-Fix Model
Waterfall Model
Spiral Model
Summary
Quiz
3. The Realities of Software Testing.
Testing Axioms
It's Impossible to Test a Program Completely
Software Testing Is a Risk-Based Exercise
Testing Can't Show That Bugs Don't Exist
The More Bugs You Find, the More Bugs There Are
The Pesticide Paradox
Not All the Bugs You Find Will Be Fixed
When a Bug's a Bug Is Difficult to Say
Product Specifications Are Never Final
Software Testers Aren't the Most Popular Members of a Project Team
Software Testing Is a Disciplined Technical Profession
Software Testing Terms and Definitions
Precision and Accuracy
Verification and Validation
Quality and Reliability
Testing and Quality Assurance (QA)
Summary
Quiz
II. TESTING FUNDAMENTALS.
4. Examining the Specification.
Getting Started
Black-Box and White-Box Testing
Static and Dynamic Testing
Static Black-Box Testing: Testing the Specification
Performing a High-Level Review of the Specification
Pretend to Be the Customer
Research Existing Standards and Guidelines
Review and Test Similar Software
Low-Level Specification Test Techniques
Specification Attributes Checklist
Specification Terminology Checklist
Summary
Quiz
5. Testing the Software with Blinders On.
Dynamic Black-Box Testing: Testing the Software While Blindfolded
Test-to-Pass and Test-to-Fail
Equivalence Partitioning
Data Testing
Boundary Conditions
Sub-Boundary Conditions
Default, Empty, Blank, Null, Zero, and None
Invalid, Wrong, Incorrect, and Garbage Data
State Testing
Testing the Software's Logic Flow
Testing States to Fail
Other Black-Box Test Techniques
Behave Like a Dumb User
Look for Bugs Where You've Already Found Them
Think like a Hacker
Follow Experience, Intuition, and Hunches
Summary
Quiz
6. Examining the Code.
Static White-Box Testing: Examining the Design and Code
Formal Reviews
Peer Reviews
Walkthroughs
Inspections
Coding Standards and Guidelines
Examples of Programming Standards and Guidelines
Obtaining Standards
Generic Code Review Checklist
Data Reference Errors
Data Declaration Errors
Computation Errors
Comparison Errors
Control Flow Errors
Subroutine Parameter Errors
Input/Output Errors
Other Checks
Summary
Quiz
7. Testing the Software with X-Ray Glasses.
Dynamic White-Box Testing
Dynamic White-Box Testing Versus Debugging
Testing the Pieces
Unit and Integration Testing
An Example of Module Testing
Data Coverage
Data Flow
Sub-Boundaries
Formulas and Equations
Error Forcing
Code Coverage
Program Statement and Line Coverage
Branch Coverage
Condition Coverage
Summary
Quiz
III. APPLYING YOUR TESTING SKILLS.
8. Configuration Testing.
An Overview of Configuration Testing
Isolating Configuration Bugs
Sizing Up the Job
Approaching the Task
Decide the Types of Hardware You'll Need
Decide What Hardware Brands, Models, and Device Drivers Are Available
Decide Which Hardware Features, Modes, and Options Are Possible
Pare Down the Identified Hardware Configurations to a Manageable Set
Identify Your Software's Unique Features That Work with the Hardware Configurations
Design the Test Cases to Run on Each Configuration 136
Execute the Tests on Each Configuration
Rerun the Tests Until the Results Satisfy Your Team
Obtaining the Hardware
Identifying Hardware Standards
Configuration Testing Other Hardware
Summary
Quiz
9. Compatibility Testing.
Compatibility Testing Overview
Platform and Application Versions
Backward and Forward Compatibility
The Impact of Testing Multiple Versions
Standards and Guidelines
High-Level Standards and Guidelines
Low-Level Standards and Guidelines
Data Sharing Compatibility
Summary
Quiz
10. Foreign-Language Testing.
Making the Words and Pictures Make Sense
Translation Issues
Text Expansion
ASCII, DBCS, and Unicode
Hot Keys and Shortcuts
Extended Characters
Computations on Characters
Reading Left to Right and Right to Left
Text in Graphics
Keep the Text out of the Code
Localization Issues
Content
Data Formats
Configuration and Compatibility Issues
Foreign Platform Configurations
Data Compatibility
How Much Should You Test?
Summary
Quiz
11. Usability Testing.
User Interface Testing
What Makes a Good UI?
Follows Standards and Guidelines
Intuitive
Consistent
Flexible
Comfortable
Correct
Useful
Testing for the Disabled: Accessibility Testing
Legal Requirements
Accessibility Features in Software
Summary
Quiz
12. Testing the Documentation.
Types of Software Documentation
The Importance of Documentation Testing
What to Look for When Reviewing Documentation
The Realities of Documentation Testing
Summary
Quiz
13. Testing for Software Security.
WarGames-the Movie
Understanding the Motivation
Threat Modeling
Is Software Security a Feature? Is Security Vulnerability a Bug?
Understanding the Buffer Overrun
Using Safe String Functions
Computer Forensics
Summary
Quiz
14. Website Testing.
Web Page Fundamentals
Black-Box Testin
Text
Hyperlinks
Graphics
Forms
Objects and Other Simple Miscellaneous Functionality
Gray-Box Testing
White-Box Testing
Configuration and Compatibility Testing
Usability Testing
Introducing Automation
Summary
Quiz
IV. SUPPLEMENTING YOUR TESTING.
15. Automated Testing and Test Tools.
The Benefits of Automation and Tools
Test Tools
Viewers and Monitors
Drivers
Stubs
Stress and Load Tools
Interference Injectors and Noise Generators
Analysis Tools
Software Test Automation
Macro Recording and Playback
Programmed Macros
Fully Programmable Automated Testing Tools
Random Testing: Monkeys and Gorillas
Dumb Monkeys
Semi-Smart Monkeys
Smart Monkeys
Realities of Using Test Tools and Automation
Summary
Quiz
16. Bug Bashes and Beta Testing.
Having Other People Test Your Software
Test Sharing
Beta Testing
Outsourcing Your Testing
Summary
Quiz
V. WORKING WITH TEST DOCUMENTATION.
17. Planning Your Test Effort.
The Goal of Test Planning
Test Planning Topics
High-Level Expectations
People, Places, and Things
Definitions
Inter-Group Responsibilities
What Will and Won't Be Tested
Test Phases
Test Strategy
Resource Requirements
Tester Assignments
Test Schedule
Test Cases
Bug Reporting
Metrics and Statistics
Risks and Issues
Summary
Quiz
18. Writing and Tracking Test Cases.
The Goals of Test Case Planning
Test Case Planning Overview
Test Design
Test Cases
Test Procedures
Test Case Organization and Tracking
Summary
Quiz
19. Reporting What You Find.
Getting Your Bugs Fixed
Isolating and Reproducing Bugs
Not All Bugs Are Created Equal
A Bug's Life Cycle
Bug-Tracking Systems
The Standard: The Test Incident Report
Manual Bug Reporting and Tracking
Automated Bug Reporting and Tracking
Summary
Quiz
20. Measuring Your Success.
Using the Information in the Bug Tracking Database
Metrics That You'll Use in Your Daily Testing
Common Project-Level Metrics
Summary
Quiz
VI. THE FUTURE.
21. Software Quality Assurance.
Quality Is Free
Testing and Quality Assurance in the Workplace
Software Testing
Quality Assurance
Other Names for Software Testing Groups
Test Management and Organizational Structures
Capability Maturity Model (CMM)
ISO 9000
Summary
Quiz
22. Your Career as a Software Tester.
Your Job as a Software Tester
Finding a Software Testing Position
Gaining Hands-On Experience
Formal Training Opportunities
Websites
Professional Organizations Dedicated to Software or Software Quality
Further Reading
Summary
Quiz
APPENDIX.
A. Answers to Quiz Questions.
Chapter 1
Chapter 2
Chapter 3
Chapter 4
Chapter 5
Chapter 6
Chapter 7
Chapter 8
Chapter 9
Chapter 10
Chapter 11
Chapter 12
Chapter 13
Chapter 14
Chapter 15
Chapter 16
Chapter 17
Chapter 18
Chapter 19
Chapter 20
Chapter 21
Chapter 22
Index.
About the Second Edition
Who Should Use This Book?
What This Book Will Do for You
Software Necessary to Use This Book
How This Book Is Organized
Part I: The Big Picture
Part II: Testing Fundamentals
Part III: Applying Your Testing Skills
Part IV: Supplementing Your Testing
Part V: Working with Test Documentation
Part VI: The Future
Appendix
Conventions Used in This Book
I. THE BIG PICTURE.
1. Software Testing Background.
Infamous Software Error Case Studies
Disney's Lion King, 1994-1995
Intel Pentium Floating-Point Division Bug, 1994
NASA Mars Polar Lander, 1999
Patriot Missile Defense System, 1991
The Y2K (Year 2000) Bug, circa 1974
Dangerous Viewing Ahead, 2004
What Is a Bug?
Terms for Software Failures
Software Bug: A Formal Definition
Why Do Bugs Occur?
The Cost of Bugs
What Exactly Does a Software Tester Do?
What Makes a Good Software Tester?
Summary
Quiz
2. The Software Development Process.
Product Components
What Effort Goes Into a Software Product?
What Parts Make Up a Software Product?
Software Project Staff
Software Development Lifecycle Models
Big-Bang Model
Code-and-Fix Model
Waterfall Model
Spiral Model
Summary
Quiz
3. The Realities of Software Testing.
Testing Axioms
It's Impossible to Test a Program Completely
Software Testing Is a Risk-Based Exercise
Testing Can't Show That Bugs Don't Exist
The More Bugs You Find, the More Bugs There Are
The Pesticide Paradox
Not All the Bugs You Find Will Be Fixed
When a Bug's a Bug Is Difficult to Say
Product Specifications Are Never Final
Software Testers Aren't the Most Popular Members of a Project Team
Software Testing Is a Disciplined Technical Profession
Software Testing Terms and Definitions
Precision and Accuracy
Verification and Validation
Quality and Reliability
Testing and Quality Assurance (QA)
Summary
Quiz
II. TESTING FUNDAMENTALS.
4. Examining the Specification.
Getting Started
Black-Box and White-Box Testing
Static and Dynamic Testing
Static Black-Box Testing: Testing the Specification
Performing a High-Level Review of the Specification
Pretend to Be the Customer
Research Existing Standards and Guidelines
Review and Test Similar Software
Low-Level Specification Test Techniques
Specification Attributes Checklist
Specification Terminology Checklist
Summary
Quiz
5. Testing the Software with Blinders On.
Dynamic Black-Box Testing: Testing the Software While Blindfolded
Test-to-Pass and Test-to-Fail
Equivalence Partitioning
Data Testing
Boundary Conditions
Sub-Boundary Conditions
Default, Empty, Blank, Null, Zero, and None
Invalid, Wrong, Incorrect, and Garbage Data
State Testing
Testing the Software's Logic Flow
Testing States to Fail
Other Black-Box Test Techniques
Behave Like a Dumb User
Look for Bugs Where You've Already Found Them
Think like a Hacker
Follow Experience, Intuition, and Hunches
Summary
Quiz
6. Examining the Code.
Static White-Box Testing: Examining the Design and Code
Formal Reviews
Peer Reviews
Walkthroughs
Inspections
Coding Standards and Guidelines
Examples of Programming Standards and Guidelines
Obtaining Standards
Generic Code Review Checklist
Data Reference Errors
Data Declaration Errors
Computation Errors
Comparison Errors
Control Flow Errors
Subroutine Parameter Errors
Input/Output Errors
Other Checks
Summary
Quiz
7. Testing the Software with X-Ray Glasses.
Dynamic White-Box Testing
Dynamic White-Box Testing Versus Debugging
Testing the Pieces
Unit and Integration Testing
An Example of Module Testing
Data Coverage
Data Flow
Sub-Boundaries
Formulas and Equations
Error Forcing
Code Coverage
Program Statement and Line Coverage
Branch Coverage
Condition Coverage
Summary
Quiz
III. APPLYING YOUR TESTING SKILLS.
8. Configuration Testing.
An Overview of Configuration Testing
Isolating Configuration Bugs
Sizing Up the Job
Approaching the Task
Decide the Types of Hardware You'll Need
Decide What Hardware Brands, Models, and Device Drivers Are Available
Decide Which Hardware Features, Modes, and Options Are Possible
Pare Down the Identified Hardware Configurations to a Manageable Set
Identify Your Software's Unique Features That Work with the Hardware Configurations
Design the Test Cases to Run on Each Configuration 136
Execute the Tests on Each Configuration
Rerun the Tests Until the Results Satisfy Your Team
Obtaining the Hardware
Identifying Hardware Standards
Configuration Testing Other Hardware
Summary
Quiz
9. Compatibility Testing.
Compatibility Testing Overview
Platform and Application Versions
Backward and Forward Compatibility
The Impact of Testing Multiple Versions
Standards and Guidelines
High-Level Standards and Guidelines
Low-Level Standards and Guidelines
Data Sharing Compatibility
Summary
Quiz
10. Foreign-Language Testing.
Making the Words and Pictures Make Sense
Translation Issues
Text Expansion
ASCII, DBCS, and Unicode
Hot Keys and Shortcuts
Extended Characters
Computations on Characters
Reading Left to Right and Right to Left
Text in Graphics
Keep the Text out of the Code
Localization Issues
Content
Data Formats
Configuration and Compatibility Issues
Foreign Platform Configurations
Data Compatibility
How Much Should You Test?
Summary
Quiz
11. Usability Testing.
User Interface Testing
What Makes a Good UI?
Follows Standards and Guidelines
Intuitive
Consistent
Flexible
Comfortable
Correct
Useful
Testing for the Disabled: Accessibility Testing
Legal Requirements
Accessibility Features in Software
Summary
Quiz
12. Testing the Documentation.
Types of Software Documentation
The Importance of Documentation Testing
What to Look for When Reviewing Documentation
The Realities of Documentation Testing
Summary
Quiz
13. Testing for Software Security.
WarGames-the Movie
Understanding the Motivation
Threat Modeling
Is Software Security a Feature? Is Security Vulnerability a Bug?
Understanding the Buffer Overrun
Using Safe String Functions
Computer Forensics
Summary
Quiz
14. Website Testing.
Web Page Fundamentals
Black-Box Testin
Text
Hyperlinks
Graphics
Forms
Objects and Other Simple Miscellaneous Functionality
Gray-Box Testing
White-Box Testing
Configuration and Compatibility Testing
Usability Testing
Introducing Automation
Summary
Quiz
IV. SUPPLEMENTING YOUR TESTING.
15. Automated Testing and Test Tools.
The Benefits of Automation and Tools
Test Tools
Viewers and Monitors
Drivers
Stubs
Stress and Load Tools
Interference Injectors and Noise Generators
Analysis Tools
Software Test Automation
Macro Recording and Playback
Programmed Macros
Fully Programmable Automated Testing Tools
Random Testing: Monkeys and Gorillas
Dumb Monkeys
Semi-Smart Monkeys
Smart Monkeys
Realities of Using Test Tools and Automation
Summary
Quiz
16. Bug Bashes and Beta Testing.
Having Other People Test Your Software
Test Sharing
Beta Testing
Outsourcing Your Testing
Summary
Quiz
V. WORKING WITH TEST DOCUMENTATION.
17. Planning Your Test Effort.
The Goal of Test Planning
Test Planning Topics
High-Level Expectations
People, Places, and Things
Definitions
Inter-Group Responsibilities
What Will and Won't Be Tested
Test Phases
Test Strategy
Resource Requirements
Tester Assignments
Test Schedule
Test Cases
Bug Reporting
Metrics and Statistics
Risks and Issues
Summary
Quiz
18. Writing and Tracking Test Cases.
The Goals of Test Case Planning
Test Case Planning Overview
Test Design
Test Cases
Test Procedures
Test Case Organization and Tracking
Summary
Quiz
19. Reporting What You Find.
Getting Your Bugs Fixed
Isolating and Reproducing Bugs
Not All Bugs Are Created Equal
A Bug's Life Cycle
Bug-Tracking Systems
The Standard: The Test Incident Report
Manual Bug Reporting and Tracking
Automated Bug Reporting and Tracking
Summary
Quiz
20. Measuring Your Success.
Using the Information in the Bug Tracking Database
Metrics That You'll Use in Your Daily Testing
Common Project-Level Metrics
Summary
Quiz
VI. THE FUTURE.
21. Software Quality Assurance.
Quality Is Free
Testing and Quality Assurance in the Workplace
Software Testing
Quality Assurance
Other Names for Software Testing Groups
Test Management and Organizational Structures
Capability Maturity Model (CMM)
ISO 9000
Summary
Quiz
22. Your Career as a Software Tester.
Your Job as a Software Tester
Finding a Software Testing Position
Gaining Hands-On Experience
Formal Training Opportunities
Websites
Professional Organizations Dedicated to Software or Software Quality
Further Reading
Summary
Quiz
APPENDIX.
A. Answers to Quiz Questions.
Chapter 1
Chapter 2
Chapter 3
Chapter 4
Chapter 5
Chapter 6
Chapter 7
Chapter 8
Chapter 9
Chapter 10
Chapter 11
Chapter 12
Chapter 13
Chapter 14
Chapter 15
Chapter 16
Chapter 17
Chapter 18
Chapter 19
Chapter 20
Chapter 21
Chapter 22
Index.