Too many reasons ... I recommend "Code Complete" - it is a great read.

posted October 17, 2009

The reasons you gave are all valid. There is another though which I think is especially important. It is communication and goal definition. What I see time and again is that an organization sets forth on a software project without a really detailed idea of where they are heading. Developers make decisions based on what they think the end result will be but often this turns out to be somewhat different. They also tend to over engineer when they don't know the goal, anticipating changes.

The worst thing is developers not knowing where they are heading. All the other ills you enumerated flow or are enhanced by that. I am especially wary of non technical people using fancy words that they don't know the meaning of. A good exercise is to ask people what they mean by things .... details, down to equipment, tools and hiring.

posted October 17, 2009

In my experience there are billions of reasons why software development efforts fail. Mostly it's all about (not having) focus, consistent approach and control.

PRINCE2 is the international standard method for project management and provides benefits to the managers and directors of a project and to an organisation, through the controllable use of resources and the ability to manage business and project risk more effectively.
PRINCE2 embodies established and proven best practice in project management.
PRINCE2 encourages formal recognition of responsibilities within a project and focuses on what a project is to deliver, why, when and for whom. PRINCE2 provides projects with:
•A controlled and organised start, middle and end
•Regular reviews of progress against plan and against the Business Case flexible decision points
•Automatic management control of any deviations from the plan
•The involvement of management and stakeholders at the right time and place during the project
•Good communication channels between the project, project management, and the rest of the organisation.

Links:

• http://www.prince2.com/

posted October 17, 2009

Totally agreed with Geoff and Robert. As far as I have experienced, we should spend good (really good) amount of time defining the problem statement. (NOTE: this has to be quality time, not just loitering around with second class documents.)

Though there is one caveat here. When we have defined the problem all parties SHOULD understand the definition of problem very very well and agree that what is documented is actually the problem. Does not matter how trivial it is, it should be agreed by all.
Second, change control management. Its very important to control the changes flowing in and out of specifications. This is something our BA & architect have been doing very well.

If things go wrong after this, you will exactly know the name and address of the entity to shoot! :P

Cheers,
Sunny

posted October 17, 2009

I agree with several of the others: the most common cause of failure is not having a clear understanding of the needs and objectives coupled with a lack of "buy-in" to these and the plan by all stakeholders.

posted October 17, 2009

I have many years of watching the repetitive nature of this process over and over again. In my experience it usually begins with the idea that if it takes one woman 9 months to have a baby we can get nine women and have a baby in a month. This is coupled with the idea that the design phase of the project is considered by developers to be the lull time prior to programming.

Software development, IMHO, needs to learn to be more of a science than the arts and crafts practiced today. It needs to be more like other disciplines of engineering where rules defining the science and/or engineering practices are applied. This is true in most other engineering disciplines, like civil or mechanical engineering. Our form of engineering is just too young to have applied these processes. There is no such thing as as a licensed Professional Engineer of Software Engineering.

If this were a science and engineering principles were applied properly to the process the life curve of the development effort could be measured from a financial standpoint, as well as, a scientifically defined percentage of the project complete. This is the idea behind the algorithmic processes of I-J CPM project management. While some companies employ the usage of software based on these algorithms, ie, Microsoft Project, there is no inherent understanding by either the project managers or of the engineers of the actuality of what the math represents to the processes or activities defined. Things like Carnegie-Mellons Software Engineering Institute (SEI) have been working on such definitions for years. Yet over the years all I hear is that we do not have the time to study or educate anyone in the associated processes. Thusly, only about 35% of the projects started today in IT will complete on time, on budget and meeting the initial requirements.

If these kind of processes were studied and understand by the corporate entities to be, I believe this would be conducive to the project completion. If this were practiced, the initial requirements of the project would be more guided and thus focused on the desired outcome. This is usually where I have seen the process go awry to begin with. Then there would be a more level expectation of the time line and expenditures by the corporate entity as to the completion of the project. This would also constrain the engineer into the fundamentals and principles of good programming practices. It would eliminate many who call themselves software engineers, but who have no grasp of the fundamentals or processes of their respective discipline.

The funny thing is that there has been very little change in the bell curve of software development life cycle for the 20 years of my experience. To implement the science of this curve it requires more investment of time and resources in the initial gathering of requirements and in the design and review process. This front loads the curve with expenditure and no visible return. Instead of the business looking at the overall return on investment (ROI) of the project, they take the short sighted approach of looking for something to physically show for the dollar. So what happens is they quickly get something to show for the investment and spend countless monies fixing, repairing or enhancing the product made. It makes the total ROI for the project much higher. Until business, even though they know the financial realities of what is stated here is true, is willing to accept that better product comes out of better planning, this will remain the same as it has always been.

Sorry y'all for the diatribe, but this question hits the rawest nerve I possess as an IT based engineer.

posted October 17, 2009

It is hardly ever a single reason, although it is often easy to identify the one or two major contributors to a failure.

Software development is engineering. It is an applied science. Until it is properly treated as such then expect failures. There should be no room for cowboys, heroes, cavaliers or any other types that are common in the field.

posted October 17, 2009

Requirements, requirements and requirements. If these are not defined before you start then an lot of projects ends up with a "house of cards." Now, defining requirements and understanding requirements is also crtical, which means that did you understand the product requirement from customer prospective instead of your own. Most common issues with a failed product is we start with a small set of requirements, and then more are added, development team adapts, which is fine but the product adapts. So, in my mind collect as many requirements as you can, generate more, anticipate requirements, re-iterate to confirm common understanding and deliver.

posted October 17, 2009

The worst failures occur when management has no idea what it wants, lots of resources, and a need to do SOMETHING. Nothing can be done about this but to get another job--an option that may soon be forced on you in any event.

The most common problem I've encountered is underestimating the time required for the project. This is often done deliberately. Delays have never been fatal where I have worked, but real deadlines coupled with anything but highly skilled time estimates could be disastrous.

In my jobs the biggest killer was doing something no one wanted. If the customer is contractually obligated to pay, the company/department/whatever doing the work may count the project as a success anyway, but if the product was to be sold commercially, this is an absolute failure.

Trying to do something that is technically impossible can kill a project also. Presumably a sufficiently skilled technical staff could do anything, but even a bunch of Einsteins would have some limits; there has to be a match between worker skill and management expectations.

The last two problems can be solved, or at least mitigated, by breaking a project into small pieces. Get something, however trivial, working, then keep enhancing it, step by step, till the entire project is done. Bad specifications, whether they specify the impossible or the unwanted, can be spotted quickly and cause delays of only days or weeks, as opposed to causing two years of wasted labor. If the entire project can never work, you'll learn about it as early as possible and limit your losses.

The last big cause of failure is getting sidetracked, which the above method can encourage. Changing the whole project's direction can make sense, but remember you are trying to make money, trying to keep the department in the next building happy, feed the CEO's ego, or something. Does that cool new feature the client just suggested and that you'd love to program do that in a cost-effective way? Always keep focused on the primary goal, whatever it might be.

posted October 17, 2009

I agree with the people who point to unknown or constantly shifting goals as a frequent cause of failure. It's sad that we now have "methodologies" that enshrine lack of planning and try to promote it as a benefit.

The one thing I can add is, having worked on the production of commercial software, I identify a lack of understanding of who the customer is and how the customer might use the product as the seed of bad goal definition.

posted October 17, 2009

Simple answer, all the above factors are involved with same weightage for development failure..

posted October 17, 2009

There are many different factors.

1. Design issues
2. Implementation issues

As far as design issues are concerned, the development team estimates certain XX number on manhours. Then the customer compares various bids and may arrive at another figure that may be unrealistic for you. But then due to the fear of losing the order, you agree for the same and try to manage by adding more resources. This may lead to some issues.

The customer may get different proposals and when they compare the technical bids, they may add some best features of all the bids and ask the one who is successful to add these features. If the bidder has not implemented similar things earlier, the chances of bugs in the code may be a possibility.

If you have enough time ( which is a reltive term :-), you can plan well and execute well.

Then the implementation is very critical as one need to take all stake holders into confidence. Else there are bound to be friction and that leads to many other unanticipated issues.

Dont be over confident in developing the code. I keep telling my boys to think like a crook as some one out there may be looking to break your software. Once you have covered all angles, you will produces a better software.

We have been delivering error free ( zero failure) online exainations for our biggest client. This year we have administered online examination for 120,000 students in 20 centres in 30 days. Not a student went back without taking the online test. That was our committment, despite power failures, internet connection failures, PC failures we managed administer the test to all the candiates.

It is a real hard work that is required to ensure fail proof software. But it is not impossible. Believe me.

Ramesh
The Human Search Engine

posted October 17, 2009

Software development efforts fail to deliver as promised because computers generate problems faster than human organizations can recognize and cope with them.

As computation and communication speeds increase, combinational complexity increases apace. Runaway collapse threatens. How to manage?

A few thoughts-
Keep requirements under control--small, modest, verifiable
Design for adaptability. Your product will be doing lots of it.
Engineer for stability.
Think about time: Microseconds, minutes, years. Define your system's behavior over the time frames. Test for it.
Think about state. Define snapshots. Verify periodically.
Don't go nuts with threads

Re your specifics:
The staff is bright enough, invariably has a matrix of strengths and weaknesses, and generally learns what they need for the task at hand in a few weeks.
Management has its own set of motivations, which I would say combine a general benevolent hope for a good outcome, and a suspicion that the technicians have underestimated money, time, and risk factors.
The resources that do not acknowledge, and combat, combinational complexity are not addressing the primary problem. Hence, they only occasionally produce successful outcomes.

What about success?
To paraphrase Voltaire, things don't have to be perfect to be successful. Good examples: TCP, HTTP, Ethernet, Google (the search app). Modestly stated ends. Stable behavior. Lots of engineering.
HTH

posted October 17, 2009

Milton: It is easy. Wedge projects by hiding assumptions, e.g. that software is to reflect management organization. Encourage thrashing of efforts by changing mind back and forth when close to completion. Systematically eliminate intrepid problem-solvers, especially after each delivery. Ensure that equipment does not work as advertised. Adjust schedules to fall outside internet time. Keep various teams in parallel universes. Outlaw open-source solutions. Make a mockery of quality testing. Change the hardware at the last minute. Base rewards on anything other than successful outcome. Thanks.

posted October 17, 2009

All of the reasons you listed, and probably more. A lot of the responses here have included "failure to adequately plan" -- to have a good idea of where you're headed before committing resources. I have seen many projects where a great deal of planning /was/ done, but it turned out the be wrong kind or sloppily done. At the end of months of meetings, only a hazy outline of the desired result had emerged, and all too often it was infeasible anyway (given technology, resources, time). So yes, failure to properly plan up front is a big killer. At the same time, beware of the "paralysis by analysis" trap, where nothing moves forward until every i is dotted and every t crossed. At worst, by the time the planning is done, the needs have changed, the market has moved on, and the project is useless. A balance has to be struck between complete requirements and the need to get moving on the thing.

Let's not even mention middle and upper level management who firmly believe that 9 women can deliver a baby in 1 month. These idiots, who've never heard of Fred Brooks, simply throw in more resources as the project slips further and further behind. Sometimes tasks /can/ be parallelized (that were serial), to make good use of additional resources, but that requires a deft touch. As Brooks explained, the more people and groups you have, the need for communication among them (which consumes time) grows exponentially, and that's something often not taken into consideration. I think that the right development group architecture would include "gatekeepers" at the top of each sub-group, to do all the communication with the other gatekeepers. Within the sub-group, its developers would only have to talk to each other. This keeps the number of communications paths down to a reasonable number, but requires someone /very/ competent (who understands the entire project) to act as a gatekeeper.

I think that the end customer should be in the loop, and they should be shown prototypes to play with at frequent intervals. That way, they can't surprise you with "it's what we asked for, but it's not what we wanted". The worst thing you can do is to get a list of requirements from them, and X months (or years) later present them with the finished product. They have to see it at every step of the way, to ensure that it's usable and meets their needs. This shows them (and the execs) that progress is being made and permits feedback on what's good and what's bad. The downside, of course, is feature creep -- "can you add this one little thing for us", which could mean a major rearchitecture! Someone in power has to champion the project, and act as a gatekeeper to say "no, it's not feasible to get that feature in to the initial release; maybe version 2". Of course, they also have to have development smarts so they can understand what should be accepted and what should be rejected (for now). It's entirely possible that the customer, looking at an early to intermediate milestone (prototype), may conclude that it's all wrong and you need to start over. As long as someone is willing to fund the resulting extra cost, that /may/ be better than stubbornly plodding along and delivering an unwanted and useless product! It should rarely come to this, if adequate planning was done in the first place, but it is possible, particularly if the end customer's business changes during the project.

Will there ever be a "scientific process" to plan and execute software development on schedule and on spec? One problem, in contrast to something like designing a car or a cell phone or a building, is that you are often creating new technology /during/ the product development. It's true that new technologies may be used in, say, a car, but someone has already worked out (most of) the kinks and it's usually more or less ready to use. Everything else is pretty much "off the shelf". Can SW development be changed to use only OTS components, without constantly reinventing the wheel?

posted October 17, 2009

Based on my experience, I would say none of the mentioned reasons are appropriate to be the realy cause failure of development efforts. Since having skilled technical staff, realisic management expectations and reasonable resources cannot ensure success, hence lack of these cannot be main reason of failure.

In my opinion the most important reason of failure is the communication GAP between management and development team. Management thinks they are clear about the goals and it has been communicated to the development team. At the same time development team thinks that they have understood the requirement and are delivering the same. At the end there are mismatch because of two reasons:

1) Management learns or get educated in due course of the project development time that changes their goals.

2) Development team had mis-understood the requirement at the first place and that has been ignored by management in initial milestones.

Now the question is how to control this?

1) Have small mile stones.
2) Management team should have a close look to ensure they have received per their expectation.
3)Any discrepencies/changes should be comunicated back to development team. (sooner the better).
4)Developement team should consider provision in thery project schedule for efforts required for these feedback from management.

posted October 17, 2009

I've never seen a software project fail, but I've seen two projects 'succeed' by the skin-of-their-teeth, and the reason might be unexpected - reuse of code.

It seems to be a good thing, reusing existing code, but in the cases I mention there was less than perfect understanding of the code; the code was getting hoary, optimized for a compiler several revisions back; and the new code had to be tweaked significantly to make it work with the old code.

It's also very hard to schedule the reuse of code. It will likely take less time than rewriting the code from scratch, but it will take more than zero time.

posted October 17, 2009

it's when one tries to reach the unreachable.
That may mean far too many things - lack of resources, knowledge, attitude, time, goal, clear vision, and basic understanding that software is just a reflection of people's knowledge/state-of-mind. Hence it might not be very rational...

Links:

• http://www.biblenews1.com/philosophy/philosop.html
• http://www.amazon.com/Laws-Software-Process-Production-Management/dp/084931...
• http://www.cs.utexas.edu/~EWD/transcriptions/EWD10xx/EWD1036.html

posted October 17, 2009

I'm afraid I disagree with those who said the problem is an inadequate problem or goal definition. I've worked on a number of projects that have had very complete requirements and that have still failed. I would also say that it has almost nothing to do with technical skill or even resources.

After 30+ yrs in this business, my feeling is that it comes down to the following:

1. In many companies, upper management believes they can treat software development as though it was something like building a house or selling a product. As a result, they both ask for and reward the wrong things. The reality is that big software projects are highly creative, particularly with the key aspects of architecture and design, which drive the rest. It's like writing a novel: you can't just give an author a well-defined subject, page count and schedule and expect the final product to be any good.

2. You can constrain any three of cost, schedule, features or quality, but not all four. Most software engineers I talk to understand and agree with this intuitively, yet a surprisingly large number of companies and projects don't put this basic rule into practice.

3. Many software engineers and managers seem to think they are smarter than those that have come before them, particularly when it comes to a development process. As a result, they often end up with an ad hoc process that feels right to them, but that is actually broken in many ways. Without a sound development process (and the tools, infrastructure and management support to go with it), most software projects are doomed to fail. The unfortunate truth is that many highly skilled programmers don't have a clue on how to correctly manage a large scale project.

I talk about these issues in more detail in the last chapter of my new book: Ultra-Fast ASP.NET, published by Apress.

posted October 17, 2009

I agree with Robert Gusnowski, Geoff Feldman and Sunny Arora who all say that software development efforts fail because of a lack of effort placed into identifying the goals, clearly communicating them and always ensuring that this is the case, hence the need for good change control management due to changes of scope.

I also wholeheartedly agree with Paul Anderson and John Ratcliffe, who state that software development is a science, not an art; more specifically, it is the applied science of engineering. There is a known engineering process that product development follows, and software is a product, yet development of software rarely follows this product development process.

In product development (which includes house design) the time is spent up front to first define the problem – gathering and analysing requirements – and then planning and designing a product that best fits all functional and non-functional (constraints) requirements. Only after this is done, well-documented so that it may be communicated clearly to all stakeholders, and well-reviewed to ensure that all stakeholders agree that the requirements are correct and the design is sound, does the building (i.e. Coding) of the product begin. Any changes to requirements (“scope changes”) will affect the design and the building/coding phases, so must be well-managed to clearly understand the impact of the proposed changes. There is room to be creative in product development (and software development), and that is during the Design stage, not during the Building/Coding phase. Testing then is straightforward: verify that the product was built right (built according to the detailed plans/designs) and validate that the right product was built (the product meets the original requirements).

Remember the saying “Measure twice, cut once”? Careful goal identification and communication, by gathering and understanding requirements, and then carefully planning and designing is “measuring twice”. “Cut once” by building only once; having to build/code more than once costs a lot of time, effort and money. Any software development process that places the important work of the project at the tail end – a lot of coding and re-coding and a lot of testing quality into the product without knowing what they should be testing for – is not following this age-old advice and is costing the project a lot of time and money.

The possible causes listed above can all be caused by a lack of up-front goal definition and planning:

1)“the technical staff is not skilled enough for the work”: They may not be skilled enough if you do not know what you are doing so that you cannot define what you need,

2)“management has unrealistic expectations”: Management may have unrealistic expectations if the end goal has not been clearly identified and communicated to them,

3)“lack of reasonable resources to perform the effort”: There may be a lack of reasonable resources to perform the effort if you do not know what you are doing so that you cannot plan for the resources needed to perform the effort.

Remember: “Measure twice, cut once” and “If you fail to plan, you plan to fail”.

posted October 17, 2009

There are so many reasons. The question really is, how should we classify them?

I favour classifying them as follows:
- Lack of appropriate buy-in and support from project stakeholders.
- Excessive length of time before discovery that you have gone off track.
- Shortage of ability to respond appropriately once it is known you are off track.

In my experience, ALL project failures fall into one or another of these categories, so if you get all three right, you won't fail. You may, of course, choose to cancel the project in a timely manner, in response to a known risk that has come to pass; a risk you chose to accept earlier in the project.

I favour nested planning and checking cycles at four or more levels of depth, say Project; Release; Iteration; Daily commitment. That way most 'going off track' incidents will be detected within a day.

For purposes of learning, each failed project comes down to a question we didn't think to ask; find out what that was, so you know to ask next time. Do the same for all cases of near failure, all cases of potential failure... and learn from other people's failures.

posted October 18, 2009

I've been involved in IT projects for a quite long time, and I think that it is not possible to treat software engineering like other engineering disciplines. It is simply far too complex. Applying ideas from other engineering disciplines to software engineering makes for bad software engineering solutions. SW has remained a craft for exactly that reasons, and not for lack of people trying to make it into an engineering discipline.

I have found that incrementally building prototype systems (aka Rapid Prototyping, Agile Programming etc..) works far better than the classical requirement specification approach. For one, requirements change quite fast; for two, people usually don't know what they want from a system until you can show them a little of what would be possible. I have never done a requirement specification in my life, and practically all of my projects were successful.

posted October 18, 2009

Someone once said that failure can only occur when time and resources are limiting factors. In the case of software, all of the above are true, though the most consistent cause I see is that the process of doing the following in order:
1) Set a completion date
2) Define the requirements
3) Design the software
4) Develop the software
5) Change the requirements
6) Wonder what went wrong

Agile is a good step in preventing failure from the above process except that even shops that use Agile often face that the end date is set before work begins and that unrealistic expectations are set at the same time.

Another ongoing issue is that management's reaction to bad news about meeting functionality or a date is to throw more people on the project and demand more frequent meetings which pull the people most capable of solving the issue away from solving the issue. This trains developers to not communicate issues until the last minute, which accelerates this vicious cycle.

As Dennis Miller used to say “But that’s just my opinion. I could be wrong”

posted October 18, 2009

To me reason 3 is the major issue and secondly it is issue 1. It is very difficult to completly define the problem, it is only experience with the problem that gives you the solution. ie iterative development is the answer.

posted October 18, 2009

2 and 3 are related and I would sugest are more common causes of failure

posted October 18, 2009

Software development efforts rarely fail, it is the software development projects that fail more often than we'd like.

Most often projects fail because there is not a coherent idea of the project's objective, resources, and expectations.

The technical staff more often than not is fully qualified to a deliver a project - but is not fully informed which project management would like to have delivered.
The management has reasonable expectations of the skills and capacity of the technical team - but has little clue of what is going to be successful with the customer, and how to explain that to the engineers.
Occasionally, there will be a true lack of resources on the project - but adding resources is almost always cheaper than failing the entire project, so this is hardly ever a reason for big failures.

The biggest problem in software development is communication failure - either lack of communication, or miscommunication. This can happen at any stage of the project - early on, in the middle of development, or just before shipment. Or in any iteration when iterative life cycle is used. Often the problems become more noticeable later in the project, or closer to delivery date, even if the communication failure started happening on day one in the planning stage.

Successful teams talk more and hear each other better - and fail less.

posted October 19, 2009

Milton,

There are plenty of great answers here, and the causes of project failures are many and varied, including skills lacking for a particular project, too many features, too little time, and not enough resources. I advocate small projects, and as Scott Nelson points out the use of Agile development is a good step in avoiding failure. Prioritized users stories (of high business value) delivered in short sprints, with working software as your primary measure of progress will allow the business to be adaptable to change while meeting the very real demands of today.

I would also state that many software project “failures” are really failures on the predictability front, something that is difficult to achieve in practice and not necessarily the most desirable attribute. I wrote about the difficulties of software predictability in a recent blog post titled Can Software Projects be Predictable?

Links:

• http://softwareresults.blogspot.com/2009/10/can-software-projects-be-predic...

posted October 19, 2009

Many of the previous responses show a frustration with management’s inability to effectively communicate their vision of a new product, with specific details, and without the delays of adjustments or additions as development progresses. But, the responses are minus one key point. Management is the customer. I’m not sure there is any business where it is the customer’s responsibility to understand product development, whether the product is a draper platform harvester or billing software.

Many middle and upper level managers in most professions could be considered functionally illiterate when it comes to using a computer. Because of this reality, I believe every project team must include “interpreters” who have at least a basic understanding of software development and functional knowledge in the area where the new software will be deployed. It would be a stroke of raw luck, in any profession, to create a useful product for another profession where developer has no functional knowledge.

I intended to include my business experiences that brought me to this realization, but the answer was getting lengthy and I did not want to dilute my point that having functional “interpreters” is key to providing a useful software product. This is true when the product is for another department, you are hired as a consultant to design and implement, or the product is to be marketed to the general public.

posted October 19, 2009