Domain-Specific Model-Driven Testing

1 Testing of Software (p. 1)

Testing is defined as the execution of a system with the purpose of finding faults in the system. It encompasses a wide range of different approaches with different goals and motivations. In this chapter, we discuss both the role of testing for software quality and the role of MTCC in the field of testing.

We start with an examination of the role of testing in Section 1.1. In order to be more efficient and effective, often, various parts of the testing process are automated. The extent, to which testing is automated, varies with different approaches, but the execution of tests implemented as test scripts and the recording of test results form the core of test automation. Section 1.1.3 examined the goals of test automation and the challenges that must be met to establish an automated testing regime. Acceptance tests involve domain experts in system testing.

Section 1.2 motivates the involvement of domain experts into the testing process and discusses acceptance tests and their automation. The quality of a test is defined by its ability to find faults. Tests must therefore be based on the requirements as defined by the domain experts for a system and must be able the assess a testee based on this requirements.

Section 1.3 discusses the meaning of quality for tests and test software and applies the quality attributes of software engineering to test software.

1.1 Testing and Software

Quality Testing is a method of quality assurance that verifies the behavior of a system against a set of requirements that are expressed as tests. The notion of quality and its relation to the requirements of a system, the specifications that represent these requirements and the needs of the users of a system are central to testing.

If quality is defined as the capability of a system to fulfill the requirements of its users or customers [JG88], the question must be considered how these requirements can be captured and formalized as tests. One source of requirements are domain experts. Domain experts are customers or users of a system who are familiar with the functions that the systems needs to fulfill and who are often experienced in the use of similar systems.

The direct and continuous involvement of domain expert in the software development process as a methodology belongs to the group of agile approaches to software development [BA04] but is considered as part of other approaches to software development as well [Eva04].

One method to make the knowledge of domain experts available for the software development process is the definition of acceptance tests. In acceptance tests, domain experts either test a system themselves or specify the functionality to be tested and the conditions for the success of a test in a format that is either suitable for execution by a human tester or for execution by an automated testing system. One purpose of test automation is to allow the execution of tests without human involvement. Automation is desirable both for economic reasons and to allow for greater effectiveness in testing.

1.1.1 Software Quality

Juran [JG88] defines quality as fitness for use. This definition raises a number of questions: fitness for what use? What constitutes fitness? How can it be determined, if and to what degree a system is fit to be used? In order to evaluate the fitness of a system, some sort of specification is needed.

A specification is a document that describes the requirements for a system. The quality of a system can also be defined as the degree to which its implementation complies with its specification. The specification must therefore reflect the requirements for a system. Functional as well as non-functional quality attributes are relevant to software systems. Software engineering defines a number of methods that serve the purpose of creating a high-quality products or evaluating its quality.

1.1.1.1 Different Notions of Quality

Two questions have to be addressed in the discussion of quality assurance.