Bill S. • This question and the FYI by themselves (without answering the question) spark interesting comments. Why is there concern about fairness? Life is not fair. Is fairness really the value that the question is trying to measure? If I accurately malign the waterfall process, isn't that fair? And is that maligning if it is indeed accurate?

The answer to the question would definitely have much different answers as each person answering will have their own perspective. Some might think that everything negative that Scrum people say about Waterfall is malignant. Some might think that everything negative that Scrum people say about Waterfall is accurate (and therefore fair?).

Anything that becomes rote means that the person has accepted the statement as fact and does not look to change it, challenge it, question it, and find other ways to look at an issue and resolve a problem. Whether or not the statements are fair or accurate or malignant, if someone is just pressing the Play button to spit out that Agile takes no discipline or that waterfall is too monolithic has stopped thinking (about that topic, maybe more).

Matthew W. • Bill, interesting thoughts. True, life is not always fair. I say fair in this context in that often waterfall is caricatured in a manner that is completely unrealistic. Ah, accurately malign? That is the real question. While it is true that maligning waterfall serves the immediate purpose of many Agile and Scrum evangelists, is this based on an accurate view and understanding of waterfall?

I sat in a presentation by Ken Schwaber this morning in Reston, VA. He was very surprising in stating that waterfall was equally as valid as Scrum. He drew distinctions in that their respective benefit and value needed to be determined almost case-by-case with more complex projects potentially tilting toward waterfall.

Personally, while the maligned view of waterfall might justify Agile or Scrum to the hard core believers, it is less convincing to those who know how to use waterfall, understand that it is more agile than credited, and offers less rework that might be required of a Scrum project. And, further, waterfall normally doesn't end with any technical debt.

Why did I ask this question on an Agile, a Scrum, and a PM-oriented group? Because I am curious to hear and engage in the comparative perspectives that will come from each group. I suspect it will be enlightening, if for no other reason that to see the misconceptions we hold about each approach.

For disclosure--my perspective--I think that waterfall is the natural way we do things in our lives and that it is the best big picture approach. Agile is a very integral part of this and, in my opinion, people have held the two as opposing approaches when they really are not. I do not see Scrum as Agile as Scrum comes across as very intolerant, especially or maybe because of the way so many people advocate it.

Harry L. • I firmly believe it has to do with the temperament of the client, more than anything else. Having used both a Waterfall and a Agile approach, and having experienced the mis-use of both of the "frameworks" I can see where there is a tendency to try to incorporate "fairness" into the equation.
The fact is that the Waterfall approach has been in use for a LONG, LONG time, in all manner of projects. It turns out that it is not particularly effective in Software Development, because of all the "bugs" or "inconsistencies" that have been baked into the process over time. I find that READING the PMI, the Waterfall approach "sounds" almost agile. But SEEING it put into practice, especially for Software Development leaves you agape in wonder at the inadequacies of the practitioner, the implementation of the tool, or both.
Agile/Scrum are process improvement ideas born out of a desire to "get around" the blocking issue of the waterfall approach. In that sense, "waterfall" will always be the enemy (in lieu of bad management, inappropriate metrics, lack of real communication, lack of care or interest in people, a deadline focus - in short everything that is crap in corporate America and beyond) and thus will always be disparaged by those that are able to find something better.
That being said - Agile and Scrum are developing their own anti patterns relatively fast, part of being adopted by an all to eager business (who always thirsts for results) and part of the practitioners starting to "bake in" some process efficiencies. In short, all Agile and Scrum need to be equally "as good" or "as bad" as waterfall is time. But no PROCESS can overtake shortages in LEADERSHIP. We have tried to "can managers" only to see this fail in an EPIC scale (business in the US is the case study).
In that sense, Agile's ATTEMPT at taking into account individuals and people over tools and processes do make it a "better" option than a mis-implemented waterfall approach.

My background? I am a ScrumMaster/PM that essentially takes care of the team. I have been accused of being an excellent PM because I "manage the project" and am considered a good SM because I "remove blocks and let us get work done". I find that if you help the team around you to achieve success, you will get there.

Valentin-Tudor M. • "Big picture" yes, "Waterfall" not.

Waterfall offers a big picture, but then does not take care good enough of this “big picture” and “big plan”.

One problem: Even you have a “big good plan”, the real world will change during Waterfall project execution. That mean in BusinessTime2 we will have a solution for older BusinessTime1. And that is waste and what is most important is that customer business changes are not timely served.

Second Problem: By default does not take care good enough of “big plan”, because cannot measure progress and manage risks – the only valid way to that in software development is progress based on “working software” and feedback about it. Waterfall is a method without enough feedback and that is known from mathematics and theory of the systems (control systems) that does not work for complex and big problems.

About Agile: Agile offer response to business, offer valid progress, but does not offer log term plans and “big pictures”.

If you want all, you need to try an iterative risk driven approach (~ derived from Unified Process). Anyway, there are some difficulties also in this case:
* there are few peoples that really understand the method (need work and abilities)
* there is tough to be efficiently applied

“Waterfall peoples” does not want to “complicate” their life with iterative work (no Agile, no UP)
“Agile peoples” recognize that Waterfall is bad and does not recognize that UP exist. Some of them are still practice Waterfall in small projects and are claiming Agility.
“Unified Process peoples” – where are they? (Not the simple ceremonialists)

Curtis S. • The word "malign" is a subjective value judgement about which I cannot comment. For instance, take the following statement and tell me if it maligns waterfall when it was mentioned in a paper I read.

"…the implementation described above is risky and invites failure. The problem is illustrated in figure 4. The testing phase which occurs at the end of the development cycle is the first event for which timing, storage, input/output transfers, etc., are experienced as distinguished from analyzed. These phenomena are not precisely analyzable. They are not the solutions to the standard partial differential equations of mathematical physics for instance. Yet if these phenomena fail to satisfy the various external constraints, then invariably a major redesign is required. A simple octal patch or redo of some isolated code will not fix these kinds of difficulties. The required design changes are likely to be so disruptive that the software requirements upon which the design is based and which provides the rationale for everything are violated. Either the requirements must be modified, or a substantial change in the design is required. In effect the development process has returned to the origin and one can expect up to a 100-percent overrun in schedule and/or costs.”

Many people with whom I have talked find the above claim about waterfall being inherently risky and prone to failure to be a statement maligning waterfall rather than a statement of fact.

Jim H. • Waterfall has become (for some time) to be associated with bloat (process, paperwork, etc.) on a project. In some situations you need to be acting in a Waterfall manner (DoD, FDA, NASA) for the project because you need to be able to fully audit it. In other situations you don't worry so much about Audit and want to get things done. Thus the Agile movement (Scrum is a management method in Agile, so I am led to believe), which in itself is a compressed version of the Sprial/Iterative model (like RUP & MSF).

The whole Agile (or agile) movement has had some good impacts/benefits to software. And some detrimental ones (I've seen it cause a devolve back to the days of Cowboy style programming, no thank you) that make you wonder what the hell were people thinking.

Typically the whole thing with any project is that all people/groups have to agree to "buy-in" to the process and support it. Once everyone is on the same page and doing their part then things tend to go better. This is true for any SDLC (Waterfall, Iterative/Spiral, Agile). The key thing about the agile movement that I have bought into is that it is all about the people (team), and them communicating and working together in parallel to get the job done.

So is Waterfall maligned, yeah to a good degree. Is it invalid and a dinosaur, nope. Is Agile the end all solution, nope. Is agile worth all the hype, not totally. Does getting buy-in and team cooperation make things better, definitely. And you don't have to be 'agile' to do it.

Larry U. • I have worked on both and seen both work well, and both fail miserably. The arguments come very close to those of quality control - is quality fitness for use or is it conformance to the requirements.
Waterfall does require an ability to know the requirements - which sometimes even the users can not properly define. Then agile can provide a vehicle which can rapidly adapt to feedback.

But I feel the maligning of waterfall is many times the result of "cowboy programming", NIH syndrome, or "we need to start coding now".

Stephen V. • Practice well, both can serve with the best being dictated by the context. Practiced badly, both can create total disasters. To the extent that "fair" matters, "all is fair in love and war" and the Agile/waterfall debate too closely resembles a war in many organizations.

In general I tend to be an Agile promoter. I think that most software development is served better by an Agile/lean-style process than a waterfall-style process. However, for software that has human safety implications or where the cost-of-change economics differ as in hardware systems, I'll advocate and execute a waterfall-like or hybrid process.

And when I make or influence these decisions, my arguments are not "rote" but are based on empirical and experiential first principles of appropriateness, a critical evaluation of the context.

Jay C. • Here is the case I make in my Agile training and coaching...

Agile was developed to a counter a history of dysfunction in software development project management. The traditional approach demanded inappropriate certainty about detailed specifications that are generally not knowable in advance. This is done in the name of a false standard of professionalism. Agile, done right, permits and demands honesty about what we know and don't know. A User Story is at the right level of abstraction to have appropriate certainty. Then the details are discovered in the interaction between the developers and the business users.

This is not to say that known details should not be captured in advance of the Story being selected for a Sprint. Jeff Sutherland said that at PatientKeeper, he expected his POs to have 3 to 30 pages of backup for proposed stories. So that remains the challenge - to distinguish the known from the to-be-discovered and gracefully adjust to discovery mistaken opinions on this. And that's being Agile - vs doing Agile.

Regarding fairness, I take that to mean that the system permits people to do their best. Contrast that with Deming's position on the system as the typical cause of people's failures in jobs. To the extent that's the case, it's fair to acknowledge that cause and it's unfair to blame the employee. In this context the comment "life is not fair" is irrelevant, or worse, an evasion of responsibility for ethical behavior. That what gives business management a bad name. I know many ethical, fair business leaders. The others deserve to have good employees shun them and their companies.

Jay C. • Matthew - when you used the phrase "as is rote" was that meant to be - as written? Or as in rote learning? Thanks.

Matthew W. • Jay, I mean as in that diehard Agile and Scrum advocates pretty much without exception caricature and demean waterfall almost as a religious article of faith than as a meaningful conclusion. As well, too many see Scrum as Agile, which it is not in my opinion, and also see Agile as an opposite of Waterfall, which is also not correct, again in my opinion.

Jay C. • Matthew, I agree completely. That's why I care so much about defining
terms precisely and usefully. So much of our Agile community's as well as
the world's problems come from sloppy use of language yielding sloppy
thinking.


What do you think of my comment about the traditional approach demanding
inappropriate certainty?

Curtis S. • @Matthew - It's probably outside the scope of the OP but you can replace "Agile" and "Scrum" with "Sequential" and "Waterfall" in your statement and it will be just as true. Dogmatists are fairly common no matter the specific canon. Attempting to correct their predetermined conclusions will usually result in ridicule and no small amount of frustration.

Jason S. • How about we forget the words "Agile" and "Waterfall" for awhile and think about what works best. In my experience the most successful projects include:

-A clear, big picture vision from an organization about what needs to be accomplished

-Free, continuous, and respectful communication across all project members, business and tech alike. I like the term "conversations" as Marty Fowler has labeled them. The fewer layers here, the better the conversations are.

-The shortest feedback loops as are reasonably possible. This is were we find out if the code we wrote matches what's needed and is valuable. Consumers of the project's output must be a part of these loops or the loops are ultimately useless.

-We continuously evaluate the outcome of the feedback loops:
Did the line of code I just write provide the value it was intended to?
Did the chunk of functionality we just coded achieve the value desired by the organization for this project?
Are there any strategies we are using that are working well?
Are there any strategies that aren't working well?
Can we fix the things that aren't working well? If yes, then how? If no, then who do we talk to that can fix them?

I think I'm going to call this methodology, "Stuff that committed people do that works". Patent, $2000/seat training courses, and certifications pending.

Andy D. • Why do so many people behave as if we leaped magically from Waterfall to Agile/Scrum?

Boehm's Spiral, Henderson-Sellars' Fountain and others provided iterative development models for a good decade or so in between. It's like Waterfall provides a big bogeyman but the others are a bit too close to provide enough scary contrast.

Scott A. • Process discussions are often religious in nature, often boiling down to "my holy approach" can beat up your "unholy approach". One of the things that I've been trying to do over the past few years is to gather data comparing the approaches. I've been publishing my results in an open manner at http://www.ambysoft.com/surveys/ so that we can move away from the process holy wars that are all too prevalent in the IT world today. Granted, surveys aren't perfect, but they're a heck of a lot better than nothing.

Interestingly, there isn't much difference in the results between agile and iterative projects, regardless of the rhetoric. What agilists are actually doing in practice often varies from much of the mainstream agile rhetoric (for example, it is quite common for agile teams to invest time up front to form the big picture that Jason refers to). At scale, there is no statistical difference (yet) between agile, iterative, and waterfall, so claims that waterfall is better (or worse) than agile at scale aren't backed up by actual data that I know of.

Anyway, regardless of what the survey data shows your mileage may vary (YMMV).

Anthony D. • @Scott. You seem to imply that agile projects are different than iterative projects. Aren't iterations baked in to all agile processes/practices? Otherwise, they'd be waterfall projects?

Gaston N. • @Anthony: you could be following the Spiral (Boehm) process, which is iterative and incremental in nature, but nonetheless, it is not Agile.

Curtis S. • @Anthony & Gaston: The difference is not one of agile, but rather iterative versus sequential. The project management frameworks most often associated with the agile movement (ie: scrum, XP, kanban, RUP, etc) are generally iterative. This has lead to some confusion and mixing of terms. The problems with a sequential process are the ones I quoted above, which are primarily risk and cost related. It is risky to assume you can chart a course completely from initial planning. You may get lucky or be working in completely familiar waters, in which case its feasible. Iterative project management techniques seek to mitigate the risk through earlier review and adoption, which lead to course corrections. The risk in this strategy is it actually increases the amount of planning necessary over the entire length of the project since the meetings and such must be done multiple times.

Anthony D. • @Gaston. I think I understand why the classic waterfall process is not agile: it's a one-shot, sequential pass culminating in a "big bang". But, what features disqualify Boehm's spiral process from being considered agile? Thanks