Synchronized Performance Evaluation Methodologies for Communication Systems

Researchers and developers typically rely on analytical techniques, network simulations or real-world experiments for the performance evaluation of communication systems. Simulations are well suitable for conducting virtual experiments of very high scale. Performance evaluations carried out with real-world systems allow for investigations with the topmost possible level of realism and detail. Finally, analytical techniques abstract away from the technical environment of a communication system and hence enable the deduction of general evidence. Unfortunately, all these techniques also suffer from a number of individual shortcomings. The analytical formalization of complex communication protocols and their behavior is often very difficult. Network simulations model only the essential functionality of a communication system, making it problematic to apply this technique for analyzing system-specific effects. In addition, evaluations carried out with real-world software prototypes are very costly and often perturbed by uncontrollable conditions in the environment.

Hybrid performance evaluation methodologies are one approach for overcoming these issues. One such method is network emulation, which combines the flexibility and scalability of network simulation with the credibility and the level of detail associated with performance evaluations using real-world systems. In this work we discuss and evaluate different contributions to the domain of hybrid performance evaluation. We first discuss different methodological enhancements that target a better applicability of network emulation. Second, we show how similar concepts can be employed for creating a hardware-software co-design framework. Finally, we demonstrate that our emulation technology can be extended for setting up a distributed debugging framework.