Automated Extraction of GUI Models for Testing

Pekka Aho*,†; Teemu Kanstrén*,‡; Tomi Räty*,§; Juha Röning¶    * VTT Technical Research Centre of Finland, Oulu, Finland
† Department of Computer Science, University of Maryland, College Park, Maryland, USA
‡ Department of Computer Science, University of Toronto, Toronto, Canada
§ Department of Electrical Engineering and Computer Science, University of California, Berkeley, California, USA
¶ Department of Computer Science and Engineering, University of Oulu, Oulu, Finland

Abstract

A significant challenge in applying model-based testing on software systems is that manually designing the test models requires considerable amount of effort and deep expertise in formal modeling. When an existing system is being modeled and tested, there are various techniques to automate the process of producing the models based on the implementation. Some approaches aim to fully automated creation of the models, while others aim to automate the first steps to create an initial model to serve as a basis to start the manual modeling process. Especially graphical user interface (GUI) applications, including mobile and Web applications, have been a good domain for model extraction, reverse engineering, and specification mining approaches. In this chapter, we survey various automated modeling techniques, with a special focus on GUI models and their usefulness in analyzing and testing of the modeled GUI applications.

Keywords

Graphical user interfaces

Model-based GUI testing

MBGT

Test automation

Model extraction

Reverse engineering

Specification mining

1 Introduction

Increasing, more ubiquitous use of software systems makes our daily lives more dependent on the software functioning without errors. In the worst case, a slight error in an airline coordinating system could cause a fatal accident, and a minor leak in Internet banking system could lead to loss of customers’ money and major financial problems. As software systems are simplified models of their real-world counterparts, no one can claim that they are perfect and free of defects [1]. Especially, errors or weaknesses in critical infrastructure are risks that have to be minimized, and that requires software testing. Software testing is a dynamic technique for increasing confidence in the correctness of the software, i.e., that the software behaves reliably as expected [2].

Graphical user interfaces (GUIs) constitute a large part of the software being developed today [3]. Most software today is interactive, and the code related to the user interface (UI) can be more than half of all code [4]. The UI of a computer program is the part that handles the output to the display and the input from the person using the program [5]. GUI is a graphical front-end to a software system that accepts user- and system-generated events as input and produces deterministic graphical output [6]. GUI is intended to make software easier to use by providing the user with visual controls [7]. GUI software is often large and complex and requires developers to deal with elaborate graphics, multiple ways of giving commands, multiple asynchronous input devices, and rapid semantic feedback [5]. Complexity is further increased when the goal is to create simple but flexible GUIs, e.g., providing default settings for quick and simple usage but allowing more skillful users to modify the settings. The correctness of GUI's behavior is essential to the correct execution of the overall software [7].

For the most part, the challenges in automating GUI testing have remained the same for a decade [8]. It is still challenging to define and implement coverage criteria for not measuring only code coverage of the test suite, but also coverage of all the possible interactions between the GUI and the underlying application. As the amount of possible interactions tends to be impractically large for nontrivial GUI applications, the challenge is to select a test suite having a good coverage with a smaller amount of test cases. While automatically crafting the models for model-based GUI testing (MBGT) has solved some issues, it still remains challenging to verify whether the GUI executes correctly as specified or expected. A GUI test case requires interleaving the oracle invocation with the test case execution, because an incorrect GUI state can lead to an unexpected screen making further test case execution useless [8]. Also, pinpointing an error's actual cause may otherwise be difficult, especially when the final output is correct but the intermediate outputs are incorrect [8]. Automated creation of test oracles is easier for regression testing, because the behavior of an existing, presumably correct version of the GUI software can be used for extracting the expected behavior, that is then stored as oracle information [9].

Most GUI applications are built using UI software tools [10] and iterative process of redesigning the UI after user testing [5]. Most designers think that the behavior is more difficult to prototype than the appearance and communicating the behavior to the developers is more difficult than the appearance [11]. Due to the iterative “rapid prototyping” development processes, the requirements, design, and implementation of GUI software change often, making GUI testing a challenging field for test automation. The maintenance effort required for updating the test suites because of the constant changes in the GUI decreases the benefits gained from test automation. Another challenge in GUI test automation is that automated testing through a GUI is more difficult than testing the application through its API (Application Programming Interface) because it requires additional programming effort to simulate user actions, to observe the outputs produced and to check its correctness, even when using auxiliary libraries [12], such as Microsoft UI Automation [13] or Jemmy [14]. Furthermore, it is even more difficult to automate the evaluation of the user experience, as it depends also on the needs of the targeted users, and usually, end users evaluate concepts and prototypes already during development [4].

Manual testing and the use of capture/replay (CR) tools have been the most popular GUI testing approaches in industry [15]. While CR tools are an easy and straightforward first step toward more effective use of the testing resources, a large number of test cases have to be manually updated after changes in the GUI [16]. The next step, also used by the latest CR tools, is to use GUI automation at a higher level of abstraction, describing test cases with action or key words and using auxiliary libraries to translate and execute the user actions and observe the produced outputs. Usually, this protects the test cases from breaking due to the smallest changes in the GUI, and sometimes the same test cases can be used for testing the software on different platforms and test environments just by changing the auxiliary library.

A number of research results have shown that model-based testing (MBT) reduces the maintenance costs compared to the CR tools [17]. Traditionally in MBT, an abstract test model is created manually based on the requirements of the system under test (SUT), and then an MBT tool is used to generate tests based on the model and execute the tests to verify if the SUT implements the requirements correctly [18]. A significant challenge in applying MBT on software systems is that manually designing the test models requires considerable amount of effort and deep expertise in formal modeling, even more so during iterative development processes or when the documentation of the system is not accurate or updated [12]. Modeling has been one of the most significant obstacles to why MBT has not been taken into use by industry on a large scale [19]. It also requires consolidating the differences between the specification that is modeled and the implementation that is tested and providing mapping between the model and the implementation to be able to execute the derived test cases on a concrete system [12]. The manual effort in designing and maintaining the models negates a significant part of the increased efficiency gained from automated testing [16].

When an existing system is being modeled, there are various model extraction techniques to automate the process of producing the models based on the implementation. The process of automatically creating models or other specifications based on existing design and implementation artifacts is also referred to as model extraction, reverse engineering, specification mining, or specification inference. In this chapter, these terms are used interchangeably unless otherwise specified. Especially GUI applications, including mobile and Web applications, have been a good domain for reverse engineering and specification mining approaches. Usually, the goal of reverse engineering is to analyze the software and represent it in another level of abstraction, and using the abstract form for software maintenance, migrating, testing, reengineering, reuse, documenting purposes, or just to understand how the software works...