1
Software Crisis 2.0

Brian Fitzgerald

Lero - The Irish Software Research Centre, University of Limerick, Limerick, Ireland

1.1 Software Crisis 1.0

In 1957, the eminent computer scientist, Edsger Dijkstra, sought to record his profession as "Computer Programmer" on his marriage certificate. The Dutch authorities, probably more progressive than most, refused on the grounds that there was no such profession. Ironically, just a decade later, the term "software crisis" had been coined, as delegates at an international conference in 1968 reported a common set of problems, namely that software took too long to develop, cost too much to develop, and the software which was eventually delivered did not meet user expectations.

In the early years of computing during the 1940s, the computer was primarily used for scientific problem solving. A computer was needed principally because of its speed of mathematical capability, useful in areas such as the calculation of missile trajectories, aerodynamics, and seismic data analysis. The users of computers at the time were typically scientific researchers with a strong mathematical or engineering background who developed their own programs to address the particular areas in which they were carrying out research. For example, one of the early computers, ENIAC (Electronic Numerical Integrator and Calculator), which became operational in 1945, by the time it was taken out of service in 1955 "had probably done more arithmetic than had been done by the whole human race prior to 1945" [1].

During the 1950s, the use of computers began to spread beyond that of scientific problem solving to address the area of business data processing [2]. These early data processing applications were concerned with the complete and accurate capture of the organization's business transactions, and with automating routine clerical tasks to make them quicker and more accurate. This trend quickly spread, and by 1960, the business data processing use of computers had overtaken the scientific one [3]. Once underway, the business use of computers accelerated at an extremely rapid rate. The extent of this rapid expansion is evidenced by the fact that in the United States, the number of computer installations increased more than twentyfold between 1960 and 1970 [4].

However, this rapid expansion did not occur without accompanying problems. The nature of business data processing was very different from the computation-intensive nature of scientific applications. Business applications involved high volumes of input and output, but the input and output peripherals at the time were very slow and inefficient. Also, memory capacity was very limited, and this led to the widespread conviction among developers that good programs were efficient programs, rather than clear, well-documented, and easily understood programs [3]. Given these problems, writing programs required much creativity and resourcefulness on the part of the programmer. Indeed, it was recognized that it was a major achievement to get a program to run at all in the early 1960s [5].

Also, there was no formal training for developers. Programming skills could only be learned through experience. Some programmers were drawn from academic and scientific environments and thus had some prior experience. However, many programmers converted from a diverse range of departments. As Friedman [3] describes it

People were drawn from very many areas of the organization into the DP department, and many regarded it as an 'unlikely' accident that they became involved with computers.

Also, during the 1960s, the computer began to be applied to more complex and less-routinized business areas. Aron [6] identifies a paradox in that as the early programmers improved their skills, there was a corresponding increase in the complexity of the problem areas for which programs had to be written.

Thus, while the term "software" was only introduced in 1958 [7], within 10 years, problems in the development of software led to the coining of the phrase "software crisis" at the NATO Conference in Garmisch [8]. The software crisis referred to the reality that software took longer to develop and cost more than estimated, and did not work very well when eventually delivered.

Over the years, several studies have confirmed these three aspects of the software crisis. For example, in relation to development timescales: Flaatten et al. [9] estimated development time for the average project to be about 18 months - a conservative figure perhaps given that other estimates put the figure at about 3 years [10] or even up to 5 years [11]. Also, an IBM study estimated that 68% of projects overran schedules [12]. In relation to the cost, the IBM study suggested that development projects were as much as 65% over budget [12], while a Price Waterhouse study in the United Kingdom in 1988 concluded that £500 million was being lost per year through ineffective development. Furthermore, in relation to performance, the IBM study found that 88% of systems had to be radically redesigned following implementation [12]. Similarly, a UK study found that 75% of systems delivered failed to meet users expectations. This has led to the coining of the term "shelfware" to refer to those systems that are delivered but never used.

Notwithstanding the bleak picture painted above, the initial software crisis has largely been resolved, while the Standish Chaos Report continues to report high rates of software project failure - estimated at 68%, for example [13].1 Although there has been no "silver bullet" advance, using Brooks [16] term, which affords an order of magnitude improvement in software development productivity, a myriad of advances have been made in more incremental ways, and software is now routinely delivered on time, within budget, and meets user requirements well. Software is really the success story of modern life. Everything we do, how we work, travel, communicate, entertain ourselves has been dramatically altered and enhanced by the capabilities provided by software.

1.2 Software Crisis 2.0

However, a new software crisis is now upon us, one that I term "Software Crisis 2.0." Software Crisis 2.0 is fuelled by a number of "push factors" and "pull factors." Push factors include advances in hardware such as that perennially afforded by Moore's law, multiprocessor and parallel computing, big memory servers, IBM's Watson platform, and quantum computing. Also, concepts such as the Internet of Things and Systems of Systems have led to unimaginable amounts of raw data that fuel the field of data analytics. Pull factors include the insatiable appetite of digital native consumers - those who have never known life without computer technology - for new applications to deliver initiatives such as the quantified self, lifelogging, and wearable computing. Also, the increasing role of software is evident in the concept of software-defined * (where * can refer to networking, infrastructure, data center, enterprise). The Software Crisis 2.0 bottleneck arises from the inability to produce the volume of software necessary to leverage the absolutely staggering increase in the volume of data being generated, which in turn allied to the enormous amount of computational power offered by the many hardware devices also available, and both complemented by the appetite of the newly emerged "digital native" consumer and a world where increasingly software is increasingly the key enabler (see Figure 1.1).

Figure 1.1 Software Crisis 2.0.

1.2.1 Hardware Advances

There are many eye-catching figures and statistics that illustrate the enormous advances in the evolution of hardware capacity over the past half-century or so. Moore's law, for example, predicted the doubling of hardware capacity roughly every 18 months or so. To illustrate this in a more familiar context, if one had invested just a single dollar in some shares when Moore was declaring his prediction initially in 1964, and if the stock market return on this shares had kept pace accordingly with Moore's prediction, the individual's net worth would now exceed $17 billion - not bad for a $1 investment. On each occasion when hardware appears to be halted due to an insurmountable challenge in the fundamental laws of physics - the impurity of atoms, sculpting light-wavelength limits, heat generation, radiation-induced forgetfulness, for example - new advances have emerged to overcome these problems as we move into the quantum computing area.

Moore's law is paralleled by similar "laws" in relation to storage capacity (Kryder's law) and network capacity (Butter's law) that portray similar exponential performance with decreasing costs. Big memory servers are disrupting the technology landscape with servers now capable of providing terabytes of physical memory. This has led to the observation that disks have become the new magnetic tape, as it is now possible to use physical memory for random access operations and to reserve the traditional random access disk for purely sequential operations.

1.2.1.1 Parallel Processing

In the era of Software Crisis 1.0, the programming paradigm was one based on serial computation. Instructions were executed one at a time on a single processor. Parallel processing allows programs to...