We are doing it wrong – the truth about agile development

We are doing it wrong – the truth about agile development

When I started writing code, we tracked projects and tasks in something called a “Gannt chart.” It was often carved into a stone wall using a hammer and chisel or sometimes Microsoft Project. If the scope of the project changed or if something took longer (or in the exceptional case - shorter) than anticipated, the project manager would break out the hammer and chisel and very angrily chop away at the stone wall to update dependencies, due dates, and anything else that needed to be changed. Done enough times, the stone wall containing the “project plan” would become an unintelligible mess and the project itself would come in grossly over time and over budget. In the best-case scenario, the project would be done on time and on budget, but would include a feature set that no longer reflected the business need (as these things change rapidly). 

Believe it or not, this happened enough times that software projects ended up getting a bum rap. Countless case studies were published as academics and industry luminaries tried to explain why software projects were harder/different/easier/less predictable/more predictable than similar hardware (construction) projects. My favorite was the ‘old house analogy that was used to explain how evolving requirements were the cause for poor software project performance – it was clearly the business’ fault. Meanwhile, the business was convinced that the issue was “programmers” (as we were called back then) either not understanding what was being asked of them or simply not caring and choosing to build whatever it was they felt like building with little to no grounding in actual business need – which did happen.

The industry tried to tackle this problem by putting more discipline and rigor around how software projects and teams were run. “Programmers” morphed into “software engineers” overnight and with little to no training. More case studies were written studying the detailed steps of the waterfall method of software development and both failed and successful projects were decomposed and analyzed to see what worked and what did not. In 1996, the Project Management Institute (PMI) released the Project Management Body of Knowledge (PMBOK) which did functionally nothing but allow us to more accurately measure how badly we were failing.

Some adventurous souls founded new methods of software development like rapid application development, joint application development, the spiral model, and others. Each of these new methodologies included components of what would ultimately yield a solid practice. These components included rapid iterations, rapid development tools to accelerate iteration cycles, and more involvement from stakeholders. None of them had all the key pieces necessary for success. Worse yet, none of them were taught effectively to students of either management or computer science. By 2000, we were still pumping out computer science graduates who were, at best, somewhat familiar with the waterfall method of developing software, were vaguely aware of other options, and could write siloed code, reasonably well, exactly once (don’t ask them to change anything).

In 2001, Kent Beck, Ken Schwaber, Martin Fowler, Jeff Sutherland, Dave Thomas, and 12 others got together and drafted and signed the Agile Manifesto, pulling together years’ worth of research and progress into a cohesive strategy for building software in a way that aligned with and strove to effectively manage rather than avoid the one constant in what we do – change. In those early days, getting any project manager (aka PMP) or business stakeholder to take you serious as an agile evangelist was an uphill battle. No way to forecast what and when something you must deliver in one year will look like, you say? Aghast! Trust that iterations and frequent deliverables will get us to where we need to go? Bah! The PMI and their ilk met Agile Development with the warmth of the catholic church welcoming Galileo’s proclamation that the earth was not the center of the solar system. The Scrum Alliance was founded and the processes that were advocated lent credibility to the cause – giving the bean counters something they could hold on to. Somehow, through the warm welcome our new methodology was greeted with, we evangelists nonetheless pressed on and the rest is history.

Agile development is a lever that can be exploited. It is based on the notion of evolutionary design and postulates that teams can skip the complexities of upfront design and move right into an iterative development approach. If you follow the values and ceremonies of Scrum, you’ll have a high functioning team and a successful project – right? Wrong.

We are still failing. We are failing less than we did before and in less spectacular and public ways only because our iterations are smaller and we’re able to detect issues earlier and course correct more often. Software projects are still painful to run and fail in some way more often than not. Why? To borrow from Martin Folwer’s amazing essay “Is Design Dead?”, it happens because we attempt to exploit the benefits of agile development without employing the enabling practices. We’re essentially doing the worst parts of evolutionary design without any of the controls. All the certified Scrum Masters from the Agile Alliance are following their ceremonies to the best of their abilities. Business stakeholders are participating more than ever before. Developers are doing their level best to apply what they learned in school or code camps and still we fail.

This is a problem in how we train our developers. We’ve ritualized out all of the enabling practices of agile development. Somewhere along the past 20 years, the rituals caught on, but what the developers need to be doing to make the rituals work were dropped. We don’t teach them (well) in university programs and the code camps are not teaching it, either. To know it exists, you have to have read something like Folwer’s seminal work on refactoring, or Rework, or Extreme Programming Explained. And to read those texts, you need to know they exist and that the body of knowledge in those tomes are essential to our success. We are failing our current generation of engineers and the organizations that count on them to make the new software-powered world turn.

How do we do better? Fortunately, we already know the path. We need to flatten, as much as possible, the cost-of-change curve. To do that, business stakeholders, developers, and project managers all have key roles to play.

If you represent the business, it’s important to recognize that the role of software has never been more central to what most businesses do. Software is (with rare exceptions) core to your ability to competitively differentiate and not something that should be kept at arm’s length and left up to someone else. It’s important to stay close to your development teams, value the transparency they offer and work with them to provide clarity, solve problems, and clear hurdles. Most importantly, be a good partner in allocating time to the development team to refactor (enhancing the structure of the code without changing the functionality). It’s important. Without it, your team will get slower and slower as time goes on and the cost of change increases. At some point, the pain of working on the code will be high enough that you see turnover or grumblings of the need for a re-write. A re-write represents a tremendous loss of time and business value (all those edge cases were coded for a reason).

If you are the scrum master, keep doing what you are doing. Keep coaching the development team on our core values of openness, commitment, focus, respect, and courage. Remind us that development is a team sport. Keep the sprints tight. Make sure the development team integrates early and often and that working software is demoed at the end of each sprint. Conduct meaningful retrospectives and track action items for implementing improvements. Track and review metrics with the team – you need some way of knowing if the retrospectives are doing anything. Model good communication. And finally, help the development team clear any hurdles as soon as possible.

If you are a developer (aka “software engineer”, aka “programmer”, aka “coder), you have some homework to do. Read the book on refactoring. THE book. Martin Fowler’s book. At the very least, read his amazing essay, “Is Design Dead?”. Practice refactoring and write unit tests so that you can do so fearlessly. You should be spending at least 20% of your time each sprint refactoring. Other than delivering value to our stakeholders, nothing is more important. Read Rework. Consider reading Extreme Programming Explained. Extreme Programming shares much of its DNA with Scrum, but has more of a developer-centric focus and the text does a decent job of highlighting what you should be thinking about.

If you don’t have the time to do your homework, I’m going to give you a cheat sheet:

  1. Defer decisions until the last possible moment. Wait until you have all the information you can get – you’ll make better choices. If you make a decision too early, you may get it wrong.
  2. Make your decisions reversible. Remember - courage. Sometimes, new stuff comes to light. Be okay with trying a different path. Let go of ego.
  3. Continuous integration, testing, and refactoring are THE pillars on which agile development rests. If you are not doing this or striving toward it, you are not agile.
  4. Testing means you – not someone else. If you have no unit tests, don’t spend six months writing them. Every time you touch code, write a test. Do that enough and you’ll have respectable coverage soon enough. Your quality team also tests. They are not an excuse for you not to do it.
  5. Refactor each sprint. 20% of your time. Every. Sprint.
  6. Prefer simple designs. A simple design runs all tests, has no duplication, reveals all the intentions, and has the fewest possible number of classes or methods.
  7. Learn design patterns and when to apply them.
  8. Be a good partner to DevOps. They are the first to get called when you stuff breaks.
  9. Deliver business value often. If what you’ve coded is useful, release it. Release often.

We’ve been doing it wrong a long time, but it doesn’t have to be this way. Never stop honing your craft. Follow all the principles of agile development. We have the tools to be successful. Let’s use them.

Lynette Mason

Building on program/project management, government partnership, healthcare, technical, and training experiences to achieve results.

5y

“If you represent the business, it’s important to recognize that the role of software has never been more central to what most businesses do.” How true this statement is. For that reason, as a previous boss impressed upon us project managers, it's crucial that software development is not looked at as an IT project, but a business project supported by IT. This would shift some of the responsibility to the business leader to be involved sufficiently instead of punting a project to IT. Agile does build that into its model with the product owner, but I've seen cases where the product owner is a business analyst in IT.

Uppili Srinivasan

COO, iGTB at Intellect Design Arena Ltd

5y

Some points debatable but could relate to the overall thought process and cheat sheet items..

Nyla Owens, PMP

Director Portfolio Management Governance and Compliance

5y

I have been a part of teams trying strictly agile and strictly waterfall, both have flaws and projects where they work better than others. In my opinion no project is entirely Business, Code/Software/IT Infrastructure or Physical (Engineering, Mechanical or Real-estate). The state of the world and companies today integrate these elements to deliver to the customer. One change has the potential to impact multiple areas. I prefer to build the structure and framework of project management with controls and processes that are fluent and allow for delivery to be both in waterfall and agile. By allowing the framework to support both processes and teams to be educated on both types of delivery we can work in tandem and support the delivery as appropriate. Strip back some of the complications and rigidity of the traditional Project Management processes and encourage/support the automation of processes and collaboration of teams, and you can find success.

Michael De Klein

Cyber Security Specialist at Eye Security

5y

Leuk om eens door te lezen, Erik Stoel!

To view or add a comment, sign in