Can static type systems speed up programming? An experimental evaluation of static and dynamic type systems
1st Edition
Published in December 2012
114 pages
978-3-95489-540-3 (ISBN)
Description
Programming languages that use the object-oriented approach have been around for quite a while now. Most of them use either a static or a dynamic type system. However, both types are very common in the industry. But, in spite of their common use in science and practice, only very few scientific studies have tried to evaluate the two type systems' usefulness in certain scenarios. There are arguments for both systems. For example, static type systems are said to aid the programmer in the prevention of type errors, and further, they provide documentation help for, there is an explicit need to annotate variables and methods with their respective types.
This book describes a controlled experiment that was conducted to shed some light into the presented matter. Which of the type systems can live up to its promises? Is one of these better suited for a particular task? And which type system is the most supportive in a problem solving? The main hypothesis claims that a static type system is faster in a problem solving in use of an undocumented API. Thus, in the study, the participants need to solve different programming tasks in an undocumented API environment with the help of the static type system (Java), and the dynamic type system (Groovy).
The author starts with a short introduction to the topic, the experimentation, and the motivation. Then, he describes a list of related works, and proceeds to the description of the experiment, its evaluation, and finally, the discussion of the results.
This book should prove interesting reading for anyone who is interested in the mechanics that drive programmer productivity and performance that depend on the kind of technology used, as well as for anyone who might be interested in empirical research in software engineering, in general.
This book describes a controlled experiment that was conducted to shed some light into the presented matter. Which of the type systems can live up to its promises? Is one of these better suited for a particular task? And which type system is the most supportive in a problem solving? The main hypothesis claims that a static type system is faster in a problem solving in use of an undocumented API. Thus, in the study, the participants need to solve different programming tasks in an undocumented API environment with the help of the static type system (Java), and the dynamic type system (Groovy).
The author starts with a short introduction to the topic, the experimentation, and the motivation. Then, he describes a list of related works, and proceeds to the description of the experiment, its evaluation, and finally, the discussion of the results.
This book should prove interesting reading for anyone who is interested in the mechanics that drive programmer productivity and performance that depend on the kind of technology used, as well as for anyone who might be interested in empirical research in software engineering, in general.
More details
Other editions
Additional editions
Sebastian Kleinschmager
Can static type systems speed up programming? An experimental evaluation of static and dynamic type systems
Book
12/2012
1st Edition
Anchor Academic Publishing
€29.99
Shipment within 7-9 days
Person
Sebastian Kleinschmager is a software engineer from Germany, and has a special interest in creating a scientific foundation for his field. During his studies of applied computer science (Bachelor's degree), and business information systems (Master), his research focused on the conduction of empirical experiments in order to evaluate programming techniques.
During his day-to-day job, he specializes in software development where he uses the.NET Framework and the newest web technologies, and therefore, has the chance to put theory into practice.
During his day-to-day job, he specializes in software development where he uses the.NET Framework and the newest web technologies, and therefore, has the chance to put theory into practice.
Content
1 - Can static type systems speed up programming? An experimental evaluation of static and dynamic type systems [Seite 1]
1.1 - Abstract [Seite 3]
1.2 - Zusammenfassung (German Abstract) [Seite 3]
1.3 - Table of Contents [Seite 4]
1.4 - Directory of Figures [Seite 7]
1.5 - Directory of Tables [Seite 8]
1.6 - Directory of Listings [Seite 11]
1.7 - 1. Introduction [Seite 13]
1.8 - 2. Motivation & Background [Seite 15]
1.8.1 - 2.1 Motivation [Seite 15]
1.8.2 - 2.2 Maintenance and Debugging [Seite 15]
1.8.2.1 - 2.2.1 Maintenance in a Nutshell [Seite 15]
1.8.2.2 - 2.2.2 Debugging in a Nutshell [Seite 16]
1.8.3 - 2.3 Documentation and APIs [Seite 17]
1.8.3.1 - 2.3.1 Documentation of Software Systems [Seite 17]
1.8.3.2 - 2.3.2 APIs and Application of their Design Principles in General Programming [Seite 19]
1.8.4 - 2.4 Type Systems [Seite 20]
1.8.5 - 2.5 Empirical Research in Software Engineering [Seite 22]
1.8.5.1 - 2.5.1 On Empirical Research [Seite 22]
1.8.5.2 - 2.5.2 Controlled Experiments [Seite 23]
1.8.5.3 - 2.5.3 Current State of Empirical Research in Software Engineering [Seite 25]
1.9 - 3. Related Work [Seite 27]
1.9.1 - 3.1 Gannon (1977) [Seite 27]
1.9.2 - 3.2 Prechelt and Tichy (1998) [Seite 28]
1.9.3 - 3.3 Daly, Sazawal and Foster (2009) [Seite 29]
1.9.4 - 3.4 Hanenberg (2010) [Seite 30]
1.9.5 - 3.5 Steinberg, Mayer, Stuchlik and Hanenberg - A running Experiment series [Seite 31]
1.9.5.1 - 3.5.1 Steinberg (2011) [Seite 31]
1.9.5.2 - 3.5.2 Mayer (2011) [Seite 31]
1.9.5.3 - 3.5.3 Stuchlik and Hanenberg (2011) [Seite 31]
1.10 - 4. The Experiment [Seite 33]
1.10.1 - 4.1 The Research Question [Seite 33]
1.10.2 - 4.2 Experiment Overview [Seite 34]
1.10.2.1 - 4.2.1 Initial Considerations [Seite 34]
1.10.2.2 - 4.2.2 Further Considerations: Studies on Using Students as Subjects [Seite 36]
1.10.2.3 - 4.2.3 Design of the Experiment [Seite 37]
1.10.3 - 4.3 Questionnaire [Seite 39]
1.10.4 - 4.4 Hard- and Software Environment [Seite 40]
1.10.4.1 - 4.4.1 Environment [Seite 40]
1.10.4.2 - 4.4.2 Programming Languages [Seite 41]
1.10.4.2.1 - 4.4.2.1 Java [Seite 41]
1.10.4.2.2 - 4.4.2.2 Groovy [Seite 41]
1.10.5 - 4.5 Workspace Applications and Tasks [Seite 42]
1.10.5.1 - 4.5.1 The Java Application - A Labyrinth Game [Seite 42]
1.10.5.2 - 4.5.2 The Groovy Application - A simple Mail Viewer [Seite 43]
1.10.5.3 - 4.5.3 Important Changes made to both Parts [Seite 44]
1.10.5.4 - 4.5.4 The Tasks [Seite 45]
1.10.5.4.1 - 4.5.4.1 The Task Types [Seite 45]
1.10.5.4.2 - 4.5.4.2 Tasks 1 and 10 - 2 Types to identify [Seite 47]
1.10.5.4.3 - 4.5.4.3 Tasks 2 and 11 - 4 Types to identify [Seite 48]
1.10.5.4.4 - 4.5.4.4 Tasks 4 and 13 - Semantic Error [Seite 50]
1.10.5.4.5 - 4.5.4.5 Tasks 5 and 14 - Semantic Error [Seite 51]
1.10.5.4.6 - 4.5.4.6 Tasks 6 and 15 - 8 Types to identify [Seite 52]
1.10.5.4.7 - 4.5.4.7 Tasks 7 and 16 - Stack size 2 and branch size 3 [Seite 53]
1.10.5.4.8 - 4.5.4.8 Tasks 8 and 17 - 12 types to identify [Seite 55]
1.10.5.4.9 - 4.5.4.9 Tasks 9 and 18 - Stack size 2 and branch size 5 [Seite 57]
1.10.5.4.10 - 4.5.4.10 Summary of Variables and Mapping of Tasks to Hypotheses [Seite 59]
1.10.6 - 4.6 Experiment Implementation [Seite 60]
1.11 - 5. Threats to Validity [Seite 61]
1.11.1 - 5.1 Internal Validity [Seite 61]
1.11.2 - 5.2 External Validity [Seite 63]
1.12 - 6. Analysis and Results [Seite 65]
1.12.1 - 6.1 General Descriptive Statistics [Seite 65]
1.12.2 - 6.2 Statistical Tests and Analysis [Seite 67]
1.12.2.1 - 6.2.1 Within-Subject Analysis on the complete data [Seite 67]
1.12.2.2 - 6.2.2 Analysis for residual effects between the two ParticipantGroups [Seite 69]
1.12.2.3 - 6.2.3 Within-Subject Analysis on the two Participant Groups [Seite 71]
1.12.2.3.1 - 6.2.3.1 Participants that started with Groovy [Seite 71]
1.12.2.3.2 - 6.2.3.2 Participants that started with Java [Seite 74]
1.12.2.4 - 6.2.4 Exploratory Analysis of the Results based on Participants' Performance [Seite 76]
1.12.2.4.1 - 6.2.4.1 Participants that started with Groovy [Seite 77]
1.12.2.4.2 - 6.2.4.2 Participants that started with Java [Seite 79]
1.12.2.5 - 6.2.5 Hypotheses and Task based Analysis [Seite 82]
1.12.2.5.1 - 6.2.5.1 Tasks 1, 2, 3, 6 and 8 [Seite 82]
1.12.2.5.2 - 6.2.5.2 Hypothesis 2-1 and Tasks 7 and 9 [Seite 84]
1.12.2.5.3 - 6.2.5.3 Hypothesis 2-2 and Tasks 4 and 5 [Seite 86]
1.13 - 7. Summary and Discussion [Seite 88]
1.13.1 - 7.1 Final Remarks [Seite 88]
1.13.2 - 7.2 Result Summary [Seite 88]
1.13.3 - 7.3 Discussion [Seite 90]
1.14 - 8. Conclusion [Seite 93]
1.15 - References [Seite 95]
1.16 - A. Appendix [Seite 100]
1.16.1 - A.1 Statistical Methods and Tests [Seite 100]
1.16.1.1 - A.1.1. Box plots (box-whisker-diagrams) [Seite 100]
1.16.1.2 - A.1.2. Kolmogorov-Smirnov and Shapiro-Wilk [Seite 101]
1.16.1.3 - A.1.3. Independent and Dependent t-test [Seite 101]
1.16.1.4 - A.1.4. Wilcoxon Signed Rank Test [Seite 101]
1.16.1.5 - A.1.5. Mann-Whitney-U and Kolmogorov-Smirnov Z test [Seite 102]
1.16.1.6 - A.1.6. Regression Analysis [Seite 102]
1.16.2 - A.2 Supplemental Data [Seite 103]
1.16.2.1 - A.2.1. Participant Results Tasks 1 to 9 (Java) [Seite 103]
1.16.2.2 - A.2.2. Participant Results Tasks 1 to 9 (Groovy) [Seite 104]
1.16.2.3 - A.2.3. Results of the Tests for Normal Distribution on the results split by the two Groups [Seite 105]
1.16.2.4 - A.2.4. Results of Tests for Normal Distribution for the Participant Performance Analyses [Seite 106]
1.16.2.5 - A.2.5. Participant Performance Analysis based on the complete data [Seite 107]
1.16.2.5.1 - A.2.5.1. Outperformers [Seite 107]
1.16.2.5.2 - A.2.5.2. Underperformers [Seite 108]
1.16.2.6 - A.2.6. Demographic of participants and Questionnaire Results [Seite 110]
1.16.3 - A.3 An Example of a problematic Experiment Design and Analysis [Seite 111]
1.1 - Abstract [Seite 3]
1.2 - Zusammenfassung (German Abstract) [Seite 3]
1.3 - Table of Contents [Seite 4]
1.4 - Directory of Figures [Seite 7]
1.5 - Directory of Tables [Seite 8]
1.6 - Directory of Listings [Seite 11]
1.7 - 1. Introduction [Seite 13]
1.8 - 2. Motivation & Background [Seite 15]
1.8.1 - 2.1 Motivation [Seite 15]
1.8.2 - 2.2 Maintenance and Debugging [Seite 15]
1.8.2.1 - 2.2.1 Maintenance in a Nutshell [Seite 15]
1.8.2.2 - 2.2.2 Debugging in a Nutshell [Seite 16]
1.8.3 - 2.3 Documentation and APIs [Seite 17]
1.8.3.1 - 2.3.1 Documentation of Software Systems [Seite 17]
1.8.3.2 - 2.3.2 APIs and Application of their Design Principles in General Programming [Seite 19]
1.8.4 - 2.4 Type Systems [Seite 20]
1.8.5 - 2.5 Empirical Research in Software Engineering [Seite 22]
1.8.5.1 - 2.5.1 On Empirical Research [Seite 22]
1.8.5.2 - 2.5.2 Controlled Experiments [Seite 23]
1.8.5.3 - 2.5.3 Current State of Empirical Research in Software Engineering [Seite 25]
1.9 - 3. Related Work [Seite 27]
1.9.1 - 3.1 Gannon (1977) [Seite 27]
1.9.2 - 3.2 Prechelt and Tichy (1998) [Seite 28]
1.9.3 - 3.3 Daly, Sazawal and Foster (2009) [Seite 29]
1.9.4 - 3.4 Hanenberg (2010) [Seite 30]
1.9.5 - 3.5 Steinberg, Mayer, Stuchlik and Hanenberg - A running Experiment series [Seite 31]
1.9.5.1 - 3.5.1 Steinberg (2011) [Seite 31]
1.9.5.2 - 3.5.2 Mayer (2011) [Seite 31]
1.9.5.3 - 3.5.3 Stuchlik and Hanenberg (2011) [Seite 31]
1.10 - 4. The Experiment [Seite 33]
1.10.1 - 4.1 The Research Question [Seite 33]
1.10.2 - 4.2 Experiment Overview [Seite 34]
1.10.2.1 - 4.2.1 Initial Considerations [Seite 34]
1.10.2.2 - 4.2.2 Further Considerations: Studies on Using Students as Subjects [Seite 36]
1.10.2.3 - 4.2.3 Design of the Experiment [Seite 37]
1.10.3 - 4.3 Questionnaire [Seite 39]
1.10.4 - 4.4 Hard- and Software Environment [Seite 40]
1.10.4.1 - 4.4.1 Environment [Seite 40]
1.10.4.2 - 4.4.2 Programming Languages [Seite 41]
1.10.4.2.1 - 4.4.2.1 Java [Seite 41]
1.10.4.2.2 - 4.4.2.2 Groovy [Seite 41]
1.10.5 - 4.5 Workspace Applications and Tasks [Seite 42]
1.10.5.1 - 4.5.1 The Java Application - A Labyrinth Game [Seite 42]
1.10.5.2 - 4.5.2 The Groovy Application - A simple Mail Viewer [Seite 43]
1.10.5.3 - 4.5.3 Important Changes made to both Parts [Seite 44]
1.10.5.4 - 4.5.4 The Tasks [Seite 45]
1.10.5.4.1 - 4.5.4.1 The Task Types [Seite 45]
1.10.5.4.2 - 4.5.4.2 Tasks 1 and 10 - 2 Types to identify [Seite 47]
1.10.5.4.3 - 4.5.4.3 Tasks 2 and 11 - 4 Types to identify [Seite 48]
1.10.5.4.4 - 4.5.4.4 Tasks 4 and 13 - Semantic Error [Seite 50]
1.10.5.4.5 - 4.5.4.5 Tasks 5 and 14 - Semantic Error [Seite 51]
1.10.5.4.6 - 4.5.4.6 Tasks 6 and 15 - 8 Types to identify [Seite 52]
1.10.5.4.7 - 4.5.4.7 Tasks 7 and 16 - Stack size 2 and branch size 3 [Seite 53]
1.10.5.4.8 - 4.5.4.8 Tasks 8 and 17 - 12 types to identify [Seite 55]
1.10.5.4.9 - 4.5.4.9 Tasks 9 and 18 - Stack size 2 and branch size 5 [Seite 57]
1.10.5.4.10 - 4.5.4.10 Summary of Variables and Mapping of Tasks to Hypotheses [Seite 59]
1.10.6 - 4.6 Experiment Implementation [Seite 60]
1.11 - 5. Threats to Validity [Seite 61]
1.11.1 - 5.1 Internal Validity [Seite 61]
1.11.2 - 5.2 External Validity [Seite 63]
1.12 - 6. Analysis and Results [Seite 65]
1.12.1 - 6.1 General Descriptive Statistics [Seite 65]
1.12.2 - 6.2 Statistical Tests and Analysis [Seite 67]
1.12.2.1 - 6.2.1 Within-Subject Analysis on the complete data [Seite 67]
1.12.2.2 - 6.2.2 Analysis for residual effects between the two ParticipantGroups [Seite 69]
1.12.2.3 - 6.2.3 Within-Subject Analysis on the two Participant Groups [Seite 71]
1.12.2.3.1 - 6.2.3.1 Participants that started with Groovy [Seite 71]
1.12.2.3.2 - 6.2.3.2 Participants that started with Java [Seite 74]
1.12.2.4 - 6.2.4 Exploratory Analysis of the Results based on Participants' Performance [Seite 76]
1.12.2.4.1 - 6.2.4.1 Participants that started with Groovy [Seite 77]
1.12.2.4.2 - 6.2.4.2 Participants that started with Java [Seite 79]
1.12.2.5 - 6.2.5 Hypotheses and Task based Analysis [Seite 82]
1.12.2.5.1 - 6.2.5.1 Tasks 1, 2, 3, 6 and 8 [Seite 82]
1.12.2.5.2 - 6.2.5.2 Hypothesis 2-1 and Tasks 7 and 9 [Seite 84]
1.12.2.5.3 - 6.2.5.3 Hypothesis 2-2 and Tasks 4 and 5 [Seite 86]
1.13 - 7. Summary and Discussion [Seite 88]
1.13.1 - 7.1 Final Remarks [Seite 88]
1.13.2 - 7.2 Result Summary [Seite 88]
1.13.3 - 7.3 Discussion [Seite 90]
1.14 - 8. Conclusion [Seite 93]
1.15 - References [Seite 95]
1.16 - A. Appendix [Seite 100]
1.16.1 - A.1 Statistical Methods and Tests [Seite 100]
1.16.1.1 - A.1.1. Box plots (box-whisker-diagrams) [Seite 100]
1.16.1.2 - A.1.2. Kolmogorov-Smirnov and Shapiro-Wilk [Seite 101]
1.16.1.3 - A.1.3. Independent and Dependent t-test [Seite 101]
1.16.1.4 - A.1.4. Wilcoxon Signed Rank Test [Seite 101]
1.16.1.5 - A.1.5. Mann-Whitney-U and Kolmogorov-Smirnov Z test [Seite 102]
1.16.1.6 - A.1.6. Regression Analysis [Seite 102]
1.16.2 - A.2 Supplemental Data [Seite 103]
1.16.2.1 - A.2.1. Participant Results Tasks 1 to 9 (Java) [Seite 103]
1.16.2.2 - A.2.2. Participant Results Tasks 1 to 9 (Groovy) [Seite 104]
1.16.2.3 - A.2.3. Results of the Tests for Normal Distribution on the results split by the two Groups [Seite 105]
1.16.2.4 - A.2.4. Results of Tests for Normal Distribution for the Participant Performance Analyses [Seite 106]
1.16.2.5 - A.2.5. Participant Performance Analysis based on the complete data [Seite 107]
1.16.2.5.1 - A.2.5.1. Outperformers [Seite 107]
1.16.2.5.2 - A.2.5.2. Underperformers [Seite 108]
1.16.2.6 - A.2.6. Demographic of participants and Questionnaire Results [Seite 110]
1.16.3 - A.3 An Example of a problematic Experiment Design and Analysis [Seite 111]
Text Sample: Chapter 4.3, Questionnaire: The questionnaire that was handed out to all participants consisted of two parts: The first was the Big Five questionnaire, also called the NEO-FFI [Costa 1992] and the second part was made of a few questions about the participant's programming experience. The Big Five questionnaire was chosen because it is one of the most popular and widely used psychological tests. It tries to measures five personality dimensions called 'Openness to Experience', 'Conscientiousness', 'Extraversion', 'Agreeableness' and 'Neuroticism', each with its own set of questions that altogether sum up to about 60 questions. Evaluation of the questionnaire data was not planned to be part of the experiment but it made a lot of sense to gather as much additional data as possible for future research and analysis. The NEO-FFI part was included for a possible exploratory analysis of personality traits/types and experiment performance, and the programming experience part was mainly included to serve as additional data for possible grouping and a meta-analysis of participant questionnaires from different experiments. The latter is a continuation of work already done by this author and Hanenberg [Kleinschmager and Hanenberg 2011], where participant questionnaire data was correlated to participants' performance in the experiments to try to find meaningful connections. 4.4, Hard- and Software Environment: 4.4.1, Environment: All participants completed the study on the same Lenovo R60 Thinkpad Computers which were provided by the University, along with a mouse for every laptop. The prepared software environment was installed on an 8 GB USB Stick. It was a Ubuntu Live Installation of Version 11.04, configured and intended to run and boot only from the stick. The only applications/libraries that were installed apart from the experiment application were XVidCapture (a tool used to record screencasts of the whole experiment for each participant) and the Sun Java Runtime and SDK of Version 1.6_25. The videos from the screen logging application were used as backup for potential problems during the experiment and as a source of information that could give answers to questions which an analysis of the log files alone could not provide. The experiment IDE (integrated development environment) itself was called Emperior, an editor specifically designed for empirical programming experiments (it was originally created by Steinberg for his master thesis [Steinberg 2011], where it is also explained in much more detail). Emperior provides very simple editing features like a search function or syntax highlighting and also logs different types of information into log files which can be analyzed later. When the participants clicked on either the 'Test' or the 'Run' buttons, Unix bash (command-line) and Ant (a Java library for batch and command-line processing, [Apache Foundation]) scripts worked behind the scenes. Both were tasked with setting correct paths variables, compile the application, test projects and return possible run output to the console or call JUnit (a unit testing library for Java, see [JUnit]) to show the test results.
