Automated Software Testing
Introduction, Management, and Performance: Introduction, Management, and Performance
Published on 23. July 1999
Book
Paperback/Softback
608 pages
978-0-201-43287-9 (ISBN)
Description
With the urgent demand for rapid turnaround on new software releases-without compromising quality-the testing element of software development must keep pace, requiring a major shift from slow, labor-intensive testing methods to a faster and more thorough automated testing approach. This book is a comprehensive, step-by-step guide to the most effective tools, techniques, and methods for automated testing. Using numerous case studies of successful industry implementations, this book presents everything you need to know to successfully incorporate automated testing into the development process.
More details
Language
English
Place of publication
Harlow
United Kingdom
Publishing group
Pearson Education Limited
Target group
College/higher education
Dimensions
Height: 236 mm
Width: 184 mm
Thickness: 32 mm
Weight
1030 gr
ISBN-13
978-0-201-43287-9 (9780201432879)
Copyright in bibliographic data and cover images is held by Nielsen Book Services Limited or by the publishers or by their respective licensors: all rights reserved.
Schweitzer Classification
Other editions
Previous edition
Elfriede Dustin
CD L0652
Software
Addison Wesley
Unfortunately, price unknown
Article exhausted; check for reprint
Persons
Elfriede Dustin is the lead author of Automated Software Testing (Addison-Wesley, 1999), and Quality Web Systems (Addison-Wesley, 2002). An acknowledged expert in software engineering and testing practices, she has assisted numerous companies in the definition and implementation of QA and testing processes. For more information, please see her Web site at www.effectivesoftwaretesting.com.
Jeff Rashka is coauthor of Automated Software Testing. He has significant industry experience as a test/quality assurance manager, software development program manager, and software division director. In addition, he has served as a SEI CMM /ISO 9000 Process Improvement Director.
John Paul has worked as a senior programmer/analyst on financial and budgeting systems as well as a host of other information systems. His software development leadership responsibilities have included system analysis and design, application prototyping, and application development using a number of different methodologies and programming techniques. His software development responsibilities have included application testing using automated test tools. John has also assumed a lead role in the performance of year 2000 compliance testing.
0201432870AB12052002
Jeff Rashka is coauthor of Automated Software Testing. He has significant industry experience as a test/quality assurance manager, software development program manager, and software division director. In addition, he has served as a SEI CMM /ISO 9000 Process Improvement Director.
John Paul has worked as a senior programmer/analyst on financial and budgeting systems as well as a host of other information systems. His software development leadership responsibilities have included system analysis and design, application prototyping, and application development using a number of different methodologies and programming techniques. His software development responsibilities have included application testing using automated test tools. John has also assumed a lead role in the performance of year 2000 compliance testing.
0201432870AB12052002
Content
Preface.
I. WHAT IS AUTOMATED TESTING?
1. The Birth and Evolution of Automated Testing.
Automated Testing.
Background on Software Testing.
The Automated Test Life-Cycle Methodology (ATLM).
Decision to Automate Test.
Test Tool Acquisition.
Automated Testing Introduction Phase.
Test Planning, Design, and Development.
Execution and Management of Tests.
Test Program Review and Assessment.
ATLM's Role in the Software Testing Universe.
ATLM Relationship to System Development Life Cycle.
Test Maturity Model (TMM)-Augmented by Automated SoftwareTesting Maturity.
Test Automation Development.
Test Effort.
Software Testing Careers.
2. Decision to Automate Test.
Overcoming False Expectations for Automated Testing.
Automatic Test Plan Generation.
Test Tool Fits All.
Immediate Test Effort Reduction.
Immediate Schedule Reduction.
Tool Ease of Use.
Universal Application of Test Automation.
One Hundred Percent Test Coverage.
Benefits of Automated Testing.
Production of a Reliable System.
Improvement of the Quality of the Test Effort.
Reduction of Test Effort and Minimization of Schedule.
Acquiring Management Support.
Test Tool Proposal.
3. Automated Test Tool Selection and Evaluation.
Organization's Systems Engineering Environment.
Third-Party Input from Management, Staff, and Customers andUsers.
Tool Criteria Reflecting the Systems Engineering Environment.
Level of Software Quality.
Help Desk Problem Reports.
Budget Constraints.
Types of Tests.
Long-Term Investment Considerations.
Test Tool Process.
Avoiding Shortcuts.
Tools That Support the Testing Life Cycle.
Business Analysis Phase Tools.
Requirements Definition Phase Tools.
Tools for the Analysis and Design Phase.
Programming Phase Tools.
Metrics Tools.
Other Testing Life-Cycle Support Tools.
Testing Phase Tools.
Test Tool Research.
Improvement Opportunities.
Evaluation Domain Definition.
Hands-On Tool Evaluation.
Evaluation Report.
License Agreement.
II. INTRODUCTION OF AUTOMATED TESTING TO A PROJECT.
4. Automated Testing Introduction Process.
Test Process Analysis.
Process Review.
Goals and Objectives of Testing.
Case Study: Test Objectives and Strategies.
Test Strategies.
Test Tool Consideration.
Review of Project-Specific System Requirements.
Application-Under-Test Overview.
Review of Project Schedule.
Test Tool Compatibility Check.
Demonstration of the Tool to the Project Team.
Test Tool Support Profile.
Review of Training Requirements
5. Test Team Management.
Organizational Structure of a Test Team.
Stovepipe Test Team.
Centralized Test Team.
IV&V Test Team.
Systems Methodology and Test Team.
Test Team Summary.
Test Program Tasks.
Test Effort Sizing.
Test Team Sizing Methods: An Overview.
Development Ratio Method.
Percentage Method.
Test Procedure Method.
Task Planning Method.
Test Effort Sizing Factors.
Test Engineer Recruiting.
Test Engineer Qualities.
Test Team Composition.
Job Requisition.
Recruiting Activities.
Locating Test Engineers.
Test Engineer Interviews.
Distinguishing the Best Candidate.
Roles and Responsibilities.
III. TEST PLANNING AND PREPARATION.
6. Test Planning: Smart Application of Testing.
Test Planning Activities.
Test Program Scope.
System Description.
Critical/High-Risk Functions.
Test Goals, Objectives, and Strategies.
Test Tools.
Test Program Parameters.
Verification Methods.
Test Requirements Definition.
Test Requirements Management.
Requirements Management Tools.
Assessing the Test Requirements Risk.
Prioritization of Tests.
Requirements Traceability Matrix.
Test Program Events, Activities, and Documentation.
Events.
Activities.
Documentation.
The Test Environment.
Test Environment Preparations.
Test Environment Integration and Setup.
The Test Plan.
Test Completion/Acceptance Criteria.
Sample Test Plan.
7. Test Analysis and Design.
Test Requirements Analysis.
Development-Level Test Analysis (Structural Approach).
System-Level Test Analysis (Behavioral Approach).
Test Program Design.
Test Program Design Models.
White-Box Techniques (Development-Level Tests).
Black-Box Techniques (System-Level Tests).
Test Design Documentation.
Test Procedure Design.
Test Procedure Definition.
Automated Versus Manual Test Analysis.
Automated Test Design Standards.
Case Study: Naming Conventions.
Manual Test Design Guidelines.
Detailed Test Design.
Test Data Requirements.
8. Test Development.
Test Development Architecture.
Technical Environment.
Environment Readiness Checks.
Automation Reuse Analysis.
Test Procedure Development/Execution Schedule.
Modularity-Relationship Analysis.
Explanation of the Sample Modularity- Relationship Matrix.
Calibration of the Test Tool.
Compatibility Work-Around Solutions.
Case Study: Incompatibility Work-Around Solution.
Manual Execution of Test Procedures.
Test Procedure Inspections-Peer Reviews.
Test Procedure Configuration Management.
Test Development Guidelines.
Design-to-Development Transiftion.
Reusable Test Procedures.
Case Study: Navigation Using Tabs or Mouse Clicks.
Case Study: Testing Bitmaps Using a Capture/Playback Tool.
Maintainable Test Procedures.
Case Study: Automating Documentation.
Case Study: Automated Random Testing.
Other Guidelines.
Automation Infrastructure.
Table-Driven Test Automation.
PC Environment Automated Setup Script.
Automated Recording Options.
Login Function.
Exit Function.
Navigation.
Verifying GUI Standards.
Smoke Test.
Case Study: Smoke Test Application.
Error-Logging Routine.
Help Function Verification Script.
Timed Message Boxes Function.
Advanced Math Functions.
IV. TEST EXECUTION AND REVIEW.
9. Test Execution.
Executing/Evaluating Test Phases.
Unit Test Execution and Evaluation.
Integration Test Execution and Evaluation.
System Test Execution and Evaluation.
Test Results Analysis of Regression Tests.
User Acceptance Test Execution and Evaluation.
Defect Tracking and New Build Process.
Defect Life-Cycle Model.
Test Program Status Tracking.
Earned Value Management System.
Case Study: System Test Status Tracking.
Test Metrics Collection and Analysis.
10. Test Program Review and Assessment.
Test Program Lessons Learned-Corrective Actions andImprovement Activity.
Test Program Return on Investment.
Case Study: TestProgram Return on Investment.
Case Study: Quantify Tool Return on Investment.
V. APPENDIXES.
A. How to Test Requirements.
Requirements Testing Approach.
Abstract.
The Quality Gateway.
Make the Requirement Measurable.
Quantifiable Requirements.
Nonquantifiable Requirements.
Keeping Track.
Coherency and Consistency.
Completeness.
Relevance.
Requirement or Solution?
Stakeholder Value.
Traceability.
Order in a Disorderly World.
Conclusions.
References.
B. Tools That Support the Automated Testing Life Cycle.
Introduction.
Business Analysis Phase.
Business Modeling Tools.
Configuration Management Tools.
Defect Tracking Tools.
Technical Review Management.
Documentation Generators.
Requirements Definition Phase.
Requirements Management Tools.
Requirements Verifiers.
Use Case Generators.
Analysis and Design Phase.
Visual Modeling Tools.
Structure Charts, Flowcharts, and Sequence Diagrams.
Test Procedure Generators.
Programming Phase.
Syntax Checkers/Debuggers.
Memory Leak and Runtime Error Detection Tools.
Code Checkers.
Static and Dynamic Analyzers.
Unit Testing Tools.
Metrics Tools.
Code (Test) Coverage Analyzers and Code Instrumentors.
Usability Measurement Tools.
Testing Support Tools.
Test Data Generators.
File Comparison Tools.
Simulation Tools.
Testing Phase.
Test Management Tools.
Network Testing Tools.
GUI Application Testing Tools.
Load/Performance Testing Tools.
Web Testing Tools.
Year 24710 Testing Tools.
Other Test Tool Vendors.
C. Test Engineer Development.
Technical Skills Stage.
Test Process Stage.
Team Effort Stage.
Technical Stewardship Stage.
Test/Project Management Stage.
Business/Product Management Stage.
D. Sample Test Plan.
Introduction.
Purpose.
Background.
System Overview.
Applicable Documents.
Master Schedule.
Roles and Responsibilities.
Project Organization.
Project Roles and Responsibilities.
Test Task Structure.
Test Team Resources.
Test Program.
Scope.
Test Approach.
Test Strategies.
Automated Tools.
Qualification Methods.
Test Requirements.
Test Design.
Test Development.
Test Environment.
Test Environment Configuration.
Test Data.
Test Execution.
Test Program Reporting.
Test Program Metrics.
Defect Tracking.
Configuration Management.
Detailed Test Schedule.
Appendixes.
D.A Test Procedure Development Guidelines.
D.B Test Verification Summary and Matrix.
D.C Test Procedures and Test Scripts
E. Best Practices.
Documented Process.
Managing Expectations.
Pilot Project.
Test Tool Compatibility Checks.
Test Tool Upgrades.
Baselined System Setup and Configuration.
Software Installations in the Test Environment Baseline.
Overall Test Program Objectives.
Keep Automation Simple.
Test Procedure Design and Development Standards.
Automated Versus Manual Test Analysis.
Reuse Analysis.
Test Team Communication with Other Teams.
Schedule Compatibility.
Customer Involvement.
Defect Documentation and Reporting.
Automated Test Advocates and Experts.
Test Team Assignments.
User Group Participation.
Test Tool Improvement Suggestions.
Become a Beta Testing Site.
Specialty Topic Experts. 0201432870T04062001
I. WHAT IS AUTOMATED TESTING?
1. The Birth and Evolution of Automated Testing.
Automated Testing.
Background on Software Testing.
The Automated Test Life-Cycle Methodology (ATLM).
Decision to Automate Test.
Test Tool Acquisition.
Automated Testing Introduction Phase.
Test Planning, Design, and Development.
Execution and Management of Tests.
Test Program Review and Assessment.
ATLM's Role in the Software Testing Universe.
ATLM Relationship to System Development Life Cycle.
Test Maturity Model (TMM)-Augmented by Automated SoftwareTesting Maturity.
Test Automation Development.
Test Effort.
Software Testing Careers.
2. Decision to Automate Test.
Overcoming False Expectations for Automated Testing.
Automatic Test Plan Generation.
Test Tool Fits All.
Immediate Test Effort Reduction.
Immediate Schedule Reduction.
Tool Ease of Use.
Universal Application of Test Automation.
One Hundred Percent Test Coverage.
Benefits of Automated Testing.
Production of a Reliable System.
Improvement of the Quality of the Test Effort.
Reduction of Test Effort and Minimization of Schedule.
Acquiring Management Support.
Test Tool Proposal.
3. Automated Test Tool Selection and Evaluation.
Organization's Systems Engineering Environment.
Third-Party Input from Management, Staff, and Customers andUsers.
Tool Criteria Reflecting the Systems Engineering Environment.
Level of Software Quality.
Help Desk Problem Reports.
Budget Constraints.
Types of Tests.
Long-Term Investment Considerations.
Test Tool Process.
Avoiding Shortcuts.
Tools That Support the Testing Life Cycle.
Business Analysis Phase Tools.
Requirements Definition Phase Tools.
Tools for the Analysis and Design Phase.
Programming Phase Tools.
Metrics Tools.
Other Testing Life-Cycle Support Tools.
Testing Phase Tools.
Test Tool Research.
Improvement Opportunities.
Evaluation Domain Definition.
Hands-On Tool Evaluation.
Evaluation Report.
License Agreement.
II. INTRODUCTION OF AUTOMATED TESTING TO A PROJECT.
4. Automated Testing Introduction Process.
Test Process Analysis.
Process Review.
Goals and Objectives of Testing.
Case Study: Test Objectives and Strategies.
Test Strategies.
Test Tool Consideration.
Review of Project-Specific System Requirements.
Application-Under-Test Overview.
Review of Project Schedule.
Test Tool Compatibility Check.
Demonstration of the Tool to the Project Team.
Test Tool Support Profile.
Review of Training Requirements
5. Test Team Management.
Organizational Structure of a Test Team.
Stovepipe Test Team.
Centralized Test Team.
IV&V Test Team.
Systems Methodology and Test Team.
Test Team Summary.
Test Program Tasks.
Test Effort Sizing.
Test Team Sizing Methods: An Overview.
Development Ratio Method.
Percentage Method.
Test Procedure Method.
Task Planning Method.
Test Effort Sizing Factors.
Test Engineer Recruiting.
Test Engineer Qualities.
Test Team Composition.
Job Requisition.
Recruiting Activities.
Locating Test Engineers.
Test Engineer Interviews.
Distinguishing the Best Candidate.
Roles and Responsibilities.
III. TEST PLANNING AND PREPARATION.
6. Test Planning: Smart Application of Testing.
Test Planning Activities.
Test Program Scope.
System Description.
Critical/High-Risk Functions.
Test Goals, Objectives, and Strategies.
Test Tools.
Test Program Parameters.
Verification Methods.
Test Requirements Definition.
Test Requirements Management.
Requirements Management Tools.
Assessing the Test Requirements Risk.
Prioritization of Tests.
Requirements Traceability Matrix.
Test Program Events, Activities, and Documentation.
Events.
Activities.
Documentation.
The Test Environment.
Test Environment Preparations.
Test Environment Integration and Setup.
The Test Plan.
Test Completion/Acceptance Criteria.
Sample Test Plan.
7. Test Analysis and Design.
Test Requirements Analysis.
Development-Level Test Analysis (Structural Approach).
System-Level Test Analysis (Behavioral Approach).
Test Program Design.
Test Program Design Models.
White-Box Techniques (Development-Level Tests).
Black-Box Techniques (System-Level Tests).
Test Design Documentation.
Test Procedure Design.
Test Procedure Definition.
Automated Versus Manual Test Analysis.
Automated Test Design Standards.
Case Study: Naming Conventions.
Manual Test Design Guidelines.
Detailed Test Design.
Test Data Requirements.
8. Test Development.
Test Development Architecture.
Technical Environment.
Environment Readiness Checks.
Automation Reuse Analysis.
Test Procedure Development/Execution Schedule.
Modularity-Relationship Analysis.
Explanation of the Sample Modularity- Relationship Matrix.
Calibration of the Test Tool.
Compatibility Work-Around Solutions.
Case Study: Incompatibility Work-Around Solution.
Manual Execution of Test Procedures.
Test Procedure Inspections-Peer Reviews.
Test Procedure Configuration Management.
Test Development Guidelines.
Design-to-Development Transiftion.
Reusable Test Procedures.
Case Study: Navigation Using Tabs or Mouse Clicks.
Case Study: Testing Bitmaps Using a Capture/Playback Tool.
Maintainable Test Procedures.
Case Study: Automating Documentation.
Case Study: Automated Random Testing.
Other Guidelines.
Automation Infrastructure.
Table-Driven Test Automation.
PC Environment Automated Setup Script.
Automated Recording Options.
Login Function.
Exit Function.
Navigation.
Verifying GUI Standards.
Smoke Test.
Case Study: Smoke Test Application.
Error-Logging Routine.
Help Function Verification Script.
Timed Message Boxes Function.
Advanced Math Functions.
IV. TEST EXECUTION AND REVIEW.
9. Test Execution.
Executing/Evaluating Test Phases.
Unit Test Execution and Evaluation.
Integration Test Execution and Evaluation.
System Test Execution and Evaluation.
Test Results Analysis of Regression Tests.
User Acceptance Test Execution and Evaluation.
Defect Tracking and New Build Process.
Defect Life-Cycle Model.
Test Program Status Tracking.
Earned Value Management System.
Case Study: System Test Status Tracking.
Test Metrics Collection and Analysis.
10. Test Program Review and Assessment.
Test Program Lessons Learned-Corrective Actions andImprovement Activity.
Test Program Return on Investment.
Case Study: TestProgram Return on Investment.
Case Study: Quantify Tool Return on Investment.
V. APPENDIXES.
A. How to Test Requirements.
Requirements Testing Approach.
Abstract.
The Quality Gateway.
Make the Requirement Measurable.
Quantifiable Requirements.
Nonquantifiable Requirements.
Keeping Track.
Coherency and Consistency.
Completeness.
Relevance.
Requirement or Solution?
Stakeholder Value.
Traceability.
Order in a Disorderly World.
Conclusions.
References.
B. Tools That Support the Automated Testing Life Cycle.
Introduction.
Business Analysis Phase.
Business Modeling Tools.
Configuration Management Tools.
Defect Tracking Tools.
Technical Review Management.
Documentation Generators.
Requirements Definition Phase.
Requirements Management Tools.
Requirements Verifiers.
Use Case Generators.
Analysis and Design Phase.
Visual Modeling Tools.
Structure Charts, Flowcharts, and Sequence Diagrams.
Test Procedure Generators.
Programming Phase.
Syntax Checkers/Debuggers.
Memory Leak and Runtime Error Detection Tools.
Code Checkers.
Static and Dynamic Analyzers.
Unit Testing Tools.
Metrics Tools.
Code (Test) Coverage Analyzers and Code Instrumentors.
Usability Measurement Tools.
Testing Support Tools.
Test Data Generators.
File Comparison Tools.
Simulation Tools.
Testing Phase.
Test Management Tools.
Network Testing Tools.
GUI Application Testing Tools.
Load/Performance Testing Tools.
Web Testing Tools.
Year 24710 Testing Tools.
Other Test Tool Vendors.
C. Test Engineer Development.
Technical Skills Stage.
Test Process Stage.
Team Effort Stage.
Technical Stewardship Stage.
Test/Project Management Stage.
Business/Product Management Stage.
D. Sample Test Plan.
Introduction.
Purpose.
Background.
System Overview.
Applicable Documents.
Master Schedule.
Roles and Responsibilities.
Project Organization.
Project Roles and Responsibilities.
Test Task Structure.
Test Team Resources.
Test Program.
Scope.
Test Approach.
Test Strategies.
Automated Tools.
Qualification Methods.
Test Requirements.
Test Design.
Test Development.
Test Environment.
Test Environment Configuration.
Test Data.
Test Execution.
Test Program Reporting.
Test Program Metrics.
Defect Tracking.
Configuration Management.
Detailed Test Schedule.
Appendixes.
D.A Test Procedure Development Guidelines.
D.B Test Verification Summary and Matrix.
D.C Test Procedures and Test Scripts
E. Best Practices.
Documented Process.
Managing Expectations.
Pilot Project.
Test Tool Compatibility Checks.
Test Tool Upgrades.
Baselined System Setup and Configuration.
Software Installations in the Test Environment Baseline.
Overall Test Program Objectives.
Keep Automation Simple.
Test Procedure Design and Development Standards.
Automated Versus Manual Test Analysis.
Reuse Analysis.
Test Team Communication with Other Teams.
Schedule Compatibility.
Customer Involvement.
Defect Documentation and Reporting.
Automated Test Advocates and Experts.
Test Team Assignments.
User Group Participation.
Test Tool Improvement Suggestions.
Become a Beta Testing Site.
Specialty Topic Experts. 0201432870T04062001