Exploiting Hardware Monitoring in Software Engineering

Kristen R. Walcott-Justice,    Department of Computer Science, University of Colorado at Colorado Springs, USA

Abstract

Program monitoring is a key component of many software engineering tasks. Traditionally, instrumentation is used to complete such tasks. However, instrumentation can prohibitively increase the time and especially the memory overhead of an application. As an alternative to instrumentation, hardware monitoring has been shown to aid in developing more efficient techniques.

In this chapter, we examine efforts in applying hardware monitoring to a number of software engineering tasks including profiling, dynamic optimization, and software testing. We discuss improvements in using instrumentation for monitoring, how hardware mechanisms can provide an alternative, and the success that has been revealed in software engineering research when applying hardware monitoring approaches.

Keywords

Performance monitoring

Hardware monitoring

Software engineering

Software testing

Dynamic optimization

Path profiling

1 Introduction

Many tasks in software engineering rely on the ability to monitor a program’s execution. For example, in software testing parts of the program under test are tracked to provide information about the quality of the program’s test suite. Similarly, in tasks requiring profiling information, such as runtime code optimization, the program is first compiled with counters inserted to generate profile information. The profile-enabled program is then run to collect profile data. Many of these software engineering tasks depend on accurate, often fine-grained information, but common approaches to collecting profiles at run-time result in significant slowdowns during program execution, high code growth, and necessitate recompilation.

Monitoring is traditionally performed using instrumentation. To instrument code, the program is analyzed, either statically or dynamically, to determine points of interest. Each is marked by a probe, which is usually a jump or call to payload code. When the probe is reached during program execution, the payload is executed, and the payload analyzes and records the monitoring information needed.

In most tools, instrumentation is statically placed in the source or binary code before execution and remains in the code until execution terminates, which leads to high overheads. For example, the time overhead of monitoring the execution of all branches has been reported to be, on average, between and , with code growth ranging from to [1–3]. Even though coverage may only require one instantiation of a code element, the instrumentation generally stays in the code, causing unnecessary time overhead. Static instrumentation may also be placed along infeasible paths, which leads to imprecision as well as avoidable code growth [4].

Dynamic instrumentation tools, such as Pin [5], add instrumentation each time a selected program element is going to be executed. A just-in-time compiler is used to compile new code for the straight line code sequence starting at the instruction to be instrumented. For each insertion, the tool transfers control to the generated sequence [5]. While dynamic techniques are advantageous in that they do not require static code modification, the time, and memory overheads are generally substantially higher than that of static instrumentation [6,7].

When monitoring large scale programs or more complex structures, the overall cost of monitoring can become prohibitive in both time and space, particularly in resource-constrained environments such as mobile devices and embedded systems. Despite the popularity of such systems, there exists little support for monitoring execution behavior on the devices themselves. As an example, mobile applications generally are first evaluated within an emulator, and then they are later subjected to field testing for additional evaluation. Emulators provide an inexpensive way to evaluate applications on mobile phones to which developers may not have physical access. However, developers recognize that when building applications, it is important that the application is tested adequately on a real device before releasing it to users [8]. Traditional program monitoring approaches using static or dynamic instrumentation can prohibitively impact the ability to monitor behavior on the devices themselves due to their increased memory footprints.

As an alternative to software-based monitoring through instrumentation, recent advances in computer architecture have brought new opportunities for producing more efficient and effective software development tools. Modern-day processors in commodity machines, mobile devices, and embedded systems include sets of hardware performance counters and monitors that were built to assist in tracking and measuring various aspects of program execution in situ. For example, the Intel Nehalem processor provides the capability to track more than 175 different kinds of execution events [9]. The Cortex-A8 and A9 processors, used in the iPhone 3GS, Samsung Galaxy Nexus, iPad2, and the Amazon Kindle Fire, have more than 50 hardware counters that can be exploited using a variety of tools. Multicore processors can enable more efficient program analysis and modification, while hardware performance mechanisms enable architectural event monitoring at very low cost.

The term hardware performance mechanism (HPM) encompasses three architectural concepts. The first is hardware performance counters, which are simply counters for architectural events. Hardware performance counts are stored in registers and accessible using privileged instructions. Using hardware performance counters, users can count the number of events that have occurred during execution across all processor cores or on a single core. The second concept is that of information that can be gathered using a performance monitoring unit (PMU) or debugging unit, which we call hardware performance monitors. Hardware performance monitors gather hardware snapshots of program execution based on a hardware performance counter. When a counter reaches a user designated value, a sample of the currently executing instructions is taken. A sample includes current instruction, memory contents, the PC, etc… Finally, the third mechanism consists of more advanced debug monitoring support on the processor that can monitor series of instructions at a time.

Compared to instrumentation, the use of hardware mechanisms is attractive as they can perform monitoring with very little overhead, and their use can remove the need for instrumentation. When monitoring using hardware mechanisms, a counter and mechanism need only be set up once per core during test execution, and reading of the mechanism is inexpensive. For example, Dey et al. [10] report that the initial setup for a counter takes approximately , and reading a counter value takes only on average. Setup and sampling of events is higher for more complex mechanisms [11]. Using hardware performance monitors in lieu of instrumentation incurs no code growth and does not require recompilation.

Although the access/reading of hardware mechanisms is inexpensive, it is not free, particularly when more complex hardware mechanisms are being used. Therefore, a balance needs to be found between the number of times a counter is read or a sample is taken and the time overhead for gathering such information. In some software engineering tasks, such as determining dynamic optimizations and performing trace selection, sparse samples can be used to estimate profiles or partial program behavior [12]. Other tasks, such as measuring the coverage of a test suite, rely on more exact information, necessitating more frequent sampling [13]. For example, in branch testing, instrumentation is used to monitor all source code level branches with which the tester is concerned and monitor only those branches. While hardware mechanisms tracking a particular event will observe all events of that type during program execution, sampled data may miss certain events such as infrequently executed branches. Also, the use of hardware mechanisms implies that samples are likely to include branches that are not associated with the test program such as those in setup, teardown, or library code.

When executing and evaluating programs, particularly those that will run in environments that are constrained by time or memory, techniques that are efficient in terms of both time and memory are necessary. However, these techniques should also be effective, providing an accurate estimation of program execution behavior.

This chapter presents an overview of the state of the art and practice in efficiently and effectively monitoring program behavior through the exploitation of hardware mechanisms in a variety of software engineering tasks. First, we discuss instrumentation practices that have been used to improve the time and memory overheads incurred by monitoring. Then, we provide an overview of hardware mechanisms, how they can be used, and several challenges that must be considered when using hardware mechanisms in software engineering tasks. Finally, we describe how hardware mechanisms have been used in recent software engineering research and discuss future directions.

2 Reducing the Overhead of Program Monitoring

A number of techniques have been developed based on instrumentation in an attempt to reduce the overheads of program monitoring. This...