Introduction

The Lost Potential of Agile Development

I wrote this book because I want to see Agile development fulfill its promises. My assertion is that organizations that retain Waterfall planning have not obtained the true benefits of Agile. I found that organizations often referred to their software development frameworks as a "hybrid" approach. After almost a decade of software development consulting, I learned to recognize that the term "hybrid" was a sure sign of a failed Agile implementation. It meant that engineering was still held to fixed schedule and release content commitments. This book will show that Waterfall planning is the antithesis of Agile. However, product managers can now adopt new planning methods that support Agile development while meeting the business need for predictability. Engineering can finally do Agile with content flexibility.

After a successful career in software development, I joined Construx Software as a consultant. I gave seminars on requirements, innovation, and project management. I also conducted software development organizational assessments, which included many Fortune 500 companies. I reviewed their practices and organizational structures to recommend how to improve software development productivity and predictability, and to reduce time-to-market. I had the opportunity to see what was really happening "under the hood" in Agile.

While at Construx, I also had the opportunity to moderate group sessions at the annual Construx Leadership Summit where software executives shared their challenges and solutions. I heard the same concerns over and over. Engineering was still being held accountable for schedule, scope and budget commitments made during the release planning stage. Many reported that they had not been able to deliver on the expectations of Agile because of the continued use of Waterfall planning.

In this book, we will consider "Waterfall planning" to be any development framework where release development cost, schedule, and scope are fixed during an initial planning period. Scope is typically defined in terms of features. Development may be split up into several increments of release functionality, but the overall release schedule and scope have already been determined.

Waterfall planning with fixed schedule, scope, and resources undermines the fundamental Agile principles of evolving requirements to incorporate feedback and additional knowledge during development. Agile teams were supposed to pull work from queues at their own pace to ensure they can incorporate feedback and new requirements during the development cycle. Variable content was supposed to give teams time to correct defects to maintain "potentially shippable" software instead of building up mounds of defects that had to be corrected near the end of Waterfall projects. Agile teams were supposed to deploy software faster, with the ultimate goal of continuous delivery. However, planning begins with the assumption of large releases that are then "filled up" with myriad features of questionable value.

This book mainly addresses issues observed in larger organizations with separate product management departments. I've seen many cases where Agile works great with small teams under the guidance of a knowledgeable product owner who truly understands customers and the market. However, something seems to go wrong when silos go up around formalized product management departments. There is a major disconnect between the Waterfall planning perpetuated by product managers and the variable content necessary for Agile development. Product managers have not been provided alternatives to Waterfall planning that meet the predictability needs of the business and the schedule and/or content flexibility required by engineering.

I've also observed that there are common characteristics of the Agile examples touted by the Agile community. They are mostly in companies with a Business to Consumer (B-C) business model where the engineers have a solid understanding of the domain. Consider Facebook or Apple versus a medical device company serving international patients. The second characteristic of these example companies is the ability to attract the top percentiles of engineering talent. The third is that engineers are not constrained by Waterfall planning mandated to try to meet quarterly and annual financial targets.

Engineers with deep domain knowledge, especially when they are potential consumers of the product, can work effectively with little guidance. They can fill in the blanks of high-level user stories without frequent interaction with product owners. The Business to Business (B-B) companies that I assessed have complex domains and were never able to convey the necessary requirements detail through user stories alone. As an example, I worked with a company where Indian engineers were implementing highly regulated casino management software from user stories when they had never been in a casino in their lives! It wasn't Agile development - just code and fix.

Talent is another critical factor. I've worked with teams of talented engineers that can deliver software with little or no software development framework or methodology. These are the top 15% of self-motivated engineers who will overcome anything in their way. And they can write virtually error-free code. Who gets this demographic? The "cream of the crop" goes to startups or fashionable companies like Google, Microsoft, Facebook, and Spotify. The rest of the industry is left with average engineers who need a solid development framework and detailed requirements.

The third Agile success element is the main subject of this book. The example companies have abandoned Waterfall release planning and embraced Agile planning, often because they are dominant in their markets and can set their own schedules. They are probably not like your company with a demanding sales team committed to quarterly revenue based on what is available for them to sell. I have not seen an effective Agile implementation where Waterfall planning persists.

Putting these three Agile success elements together, one realizes that most of the "ideal Agile" examples and literary advice is based on a small fraction of the software industry comprised of startups and sexy B-C companies that have embraced Agile planning and are loaded with engineering talent. I always felt bad for the Construx summit attendees who listened with envy about amazing Agile implementations and wondered what was wrong with them. It is not them. They could be making the same presentations given the same ideal conditions discussed above.

This book is for the frustrated "silent majority" of software development organizations where Agile has not met expectations. It addresses the typical large company with complex domains and average talent. These companies can now adopt Agile planning and deployment and improve predictability for the business. This book describes how highly motivated Agile product teams can be built around business problems with quantified value. Engineers are provided with the problem to solve instead of trying to convey detailed implementation requirements through user stories with an uninvolved or uninformed product owner.

The word Agile in this book refers to methods other than just Scrum, like Kanban. I use Scrum terminology in this book, but the same investment planning principles and methods apply to any Agile method using iterative development with evolving requirements where self-directed teams pull their work from backlogs.

There is one other term used throughout this book that needs to be clarified for those of you with some knowledge of company financials. When I use the term "income" I am referring to "Net Income," which is the income received with product cost, expenses and taxes removed.

Missed Business Expectations

Agile has not addressed the real-world need for financial predictability in business. We do know that roadmap commitments were usually not met with Waterfall development. We learned there is a limit to the accuracy of software development effort and schedule estimation. With the advent of Agile, engineering was now expected to make commitments even before requirements were developed, exacerbating the predictability problem.

The Agile community response of, "The business needs to be more Agile" did nothing to help product and project managers who were still held accountable for long-term commitments of schedule, scope, and cost. This widened the division between product management and engineering. Engineering is frustrated by not being able to do Agile development the way they were taught, having to resort to "hybrid" methods. Product and project management are frustrated because of the reluctance of engineering to make release commitments.

Certainly, there are organizations that have been able to manage variable content with Waterfall planning by reserving software development contingency. But these organizations are rare because contingency doesn't typically survive the pressure to commit during release planning.

I don't blame product management for the continued use of Waterfall planning. They weren't provided with an alternative. Product managers were left out of Agile because, in the view of the Agile community, they could be replaced with product owners who sit with a team to provide verbal clarification for everything they need to know. And, according to the Agile community,...