Software Testing - Is it a cost or an Investment ?
When it comes to test the products, there are many limitations in terms of resources, time lines, tools and finally it matters to the budget.
Organizations need to support thier products and the limitations above leading to increasing support costs.
Clarification added September 21, 2007:
Let me thank you all for your time & views over the context. It's interesting to see a great discussion here.
The context here is for Product Based companies with over hundread plus production deployments of the product.
From my perspective, i see it as an investment for most of the product based organizations & cost cuttings here might have lot of impact on the sales, support costs & the perceived value of quality for the end user.
The Test Managers might be familiar with the following scenarios & It's good to see that expectations might be high.
Planning & Estimation
It takes as much as time as it takes to development for the effective test design, Development & execution. But timelines & resources will be limited
Change Management & Regression Testing
Are we effectively deploying the efforts to update the old test suites based on the changes for the release ? It's same as the code base but let focussed area
Last Minutes Changes
Development Cycles gets delayed & since there are promises made for release, it will affect Test Execution timelines
1. What will be the risk, if we need to ship it in next week ?
2. Can we deploy some developers since timelines have reduced to get it on time
Resources
Do we need so many testers ? what should be the ideal ratio between development & test groups
Tools & Test Labs
Do we have the enough budgets for the Tools & Machines that are required for your test lab and automation.
Recrutment
We might be familiar with the following in the recruitment process
1. Why do we need so many testers ? what should be the ideal ratio between development & test groups ?
2. This resource is good but may not fit into our budgets
3. This resource expectations are high. We are not even paying that much to development groups.
Regards,
Venkat.
Clarification added September 26, 2007:
Some Links that refer to this thread.
1. http://venkatreddyc.wordpress.com/2007/09/24/software-testing-is-it-a-cost-or-an-investment-for-stakeholders/
2. http://qualsys.wordpress.com/2007/09/26/software-testing-cost-or-investment/
3. http://www.dzone.com/links/software_testing_is_it_a_cost_or_an_investment_.html
Good Answers (8)
Hi Venkat,
I work in product management at an Internet and mobile product company - it is a startup and I also double as a project manager leading to release of products. One of my biggest pain points is that in the race to release a product in time, QA gets neglected. And having faced the consequences of improper test management, and hence risk management, I am convinced that we do not invest as much time into software testing as we should.
Sure there are mitigating factors like release timelines, pressure of time to market, costs etc but fact remains that testing is an integral part of the development life cycle. And should not be cut corners with.
Software testing is a necessity. To create adequate functioning software you will need to test in different stages of development. Software testing can be best viewed as a investment. Testing in different stages of the production will even cut back production efforts. It can save a lot of time in development. Software does not always have to be tested by expensive experts, depending on your target audience you can ask students or other (open source) people to assist you.
Software testing will eventually save money in support costs. By having a complete overview of the weakness of your product you can give more precise feedback to your customers. Less costumers will need support if you do your testing. Money is always a issue. When it comes to testing, it is well spend.
It's a cost that needs to be acknowledged and budgeted.
Up front you need to establish what your quality needs are for your product. There should be a shared understanding of this: Will someone end up in the hospital if the software crashes, or are the users so addicted to it that they'll put up with a browser crash every couple hours? Then, you need the ability to assess that quality before releases, or accept the risk of not assessing it. Either extreme is fine, if it's in sync with your product goals. Don't set the bar too high, as this will undermine the whole process. What I mean is: don't have a requirement that there are no cosmetic bugs if that's not important to the business; this will devalue the QA effort.
Once you have your quality requirements, and the proper resources to ensure it, I'd encourage you to hang on tight.
Make quality the last thing to go. When push comes to shove, drop features first, slip the schedule second, but don't skimp on testing.
Not having a feature is easy to explain. Releasing late may have repercussions, but it's understandable. Releasing bad quality will have more unpredictable affects throughout the organization and customer base. From my experience, it's the most expensive way to go.
Software testing - Is a must. Defect free software - If people think can be delivered, then I think they have to wake up. Boss, this is not a fairyland. Humans err - Face the fact.
But yes, software testing should be done with the right strategy, right approach else that also leads to a drainage of money. I know of a organisation who has spent 1000's of pounds for automation of a product having immense trust in the person who heads the testing. He has very fancy ideas and comes with a framework built upon automated tool's framework.. works on it for 1+ years and then the project is discarded. No prototype was done.. No POC was done.. Right into developing of a framework.. Just becoz 1 person felt this is how it should be done and the result is empty hands. (Power Games, man!) Blame is passed onto offshore, contractors.. and it works..
I also have read about case studies where due to lack of testing and products moved to production w/o being tested has been a very costly affair to the company. The organisation had to pay millions of dollars because the customers were billed incorrectly and also of cases where the n/w has crashed the moment it went into production. As a Test Manager, I would say s/w testing is a must. But not the name sake type of testing. A planned, designed and well executed with reasonable timelines (reasonable because, testing even today gets the last bit of the pie) and it is sure going to make a difference
Clarification added September 20, 2007:
Cost -> If not don the right way
Investment -> If done systematically and in a planned way
Milind L
Solution Archtect
Best Answers in: Organizational Development (2), Quality Management and Standards (2), Regulation and Compliance (1), Ethics (1)
The status, I have seen in many companies, as I go on auditing and consulting, software testing is considered not as a cost but as a hurdle too. At many places people don't have efficient reviews, unit testing is absent, no knowledge about interface testing, integration testing. Everything boils down to system testing. There is no root cause analysis and process improvement.
If software testing defects are considered as process failures, one can improve the process to produce first time right. These are six sigma processes with 3.4 defects per million instances. That time we may call it as an invest. Otherwise it is find and fix the defecct.
Directly answering your question - it depends on the project :)
If you have the product, that is to be sold (or is being sold) to larger volume of customers - software testing is for sure the investment.
If you have very directed, single customer driven project - it might be seen as the cost. However it's not so clear - please remember, that solid tests will allow you to lower the future costs of guarantee service. So from this point of view - it is investment as well.
In general I suppose the testing costs should be basically part of the project price. And - if the whole project is seen as the investment (preparation of new company product) - the testing is investment as well. If the project is customer-based (so the price is calculated basing on the cost + margin), and the project preparation can be seen as the cost for the company - the testing is the cost as well.
Regarding the limitations you mentioned - I suppose tests should be priced (in the matter of resources) like a standard, additional project work. It should never be something, which "we will cover once we have the resources left" - it is never a float time - it's always some additional work that must be done during the project. How much it is - it depends on the project methodology and software development methods / software testing methods used (it will be completelly different on PMI/waterfall projects, then on Agile/XP projects). But the testing phase should always be a integral part of the project - and the matter whether it's a cost or an investment is based on what kind of project you are doing.
The smart companies will answer that testing is an investment. The not-so-smart companies will answer that it is a cost.
In my experience, most companies seem to fall in the "no-so-smart" category.
Let's rephrase things to put the original question in perspective:
"Quality software - is it a cost or an investment?"
When allocating resources, evaluate based on the results those allocations will produce, and be honest with yourself about exactly what those results will be...
More Answers (20)
Hi Venkat,
Software testing is a cost that could save you from unnecessary expenses forther on the road. Released bugs not only have extra cost to fix them, they also do hurt reputation greately.
Solutions to deal with hi Software Development costs:
1) implement test driven development
2) establish continuous integration in your environment and make sure that all small pieces are tested automatically per each commit (DI, IoC and other development patterns can help here).
3) add integration testing.
--
Rinat Abdullin
Venkata Ramana G
Senior Director, Global Financial Services and Insurance at Keane
Best Answers in: Starting Up (1)
Software testing is surely an Investment, whether it is for a product, enhancement or maintenance. It has to be done at various levels like unit testing, integration testing, performance testing, etc. It will avoid costly mistakes later which attracts penalties, liabilities, loss of good will.
Ofcourse: A stitch in time saves nine
Venkata Ramana Gundamaraju
gvramana18@rediffmail.com
Nitin Kishore G
SW-PM@ Marvell && Investor@ Siftin.com
Best Answers in: Job Search (1), Staffing and Recruiting (1), Advertising (1)
I think it's mostly a cost that you incur for not hiring good developers/designers. If a developer/designer do a good deal of "unit testing" than QA will take less effort and resources. Over time, effort should be made to automate some of the QA effort by dedicating a good developer to do this.
QA is absolutely necessary, but any issue that you find through QA should be tracked back to its origin to do real "quality assurance"
Krishna P
Process/Quality Coach, Mentor, Photography Enthusiast, Sporadic Blogger
Best Answers in: Customer Relationship Management (1), Market Research and Definition (1)
Strictly speaking, testing falls under "Cost of Quality" as a component attributing to Appraisal cost. CoQ is comprised of Appraisal Cost(Testing, Reviews), Prevention Cost(Causal Analysis, Training), Failure Cost(Rework, Lost Opportunity).
Early days of software development, it used to be just "code and deliver". But, as the industry progressed, we've graduated to "testing the product" before we deliver. So, plainly speaking, its a cost. But (oh... what a cliche !!), it is always recommended that it be perceived as an investment to improve quality. Learn from the results of testing and use them to improve the analysis and design activities. Many organizations have an objective of reducing the CoQ to the maximum possible. You obviously cannot do away with CoQ entirely but should look to minimize it. This can be achieved by strengthening the analysis and design phases so as to reduce the testing effort later on the lifecycle.
Hope this answers your question.
Jonathan G
Communications Software and Systems Verification Manager
Best Answers in: Software Development (1)
How much time, effort and money you *invest* in testing your software will be dependent on many factors (as you note in the question) but ultimately you have to answer the question "what is the *cost* of not testing XYZ?" It's then a simple ;) matter of balancing the investment against the cost!
How you choose to invest is also a big consideration, and again dependent on the specific circumstances that you operate in. That said, for my money, the most important investment in test that any organization can make is in the staff. Good test engineers are worth their weight in gold and when test equipment and automation tools are available on the open market it is the quality of your staff that differentiate you from the opposition.
BTW I can thoroughly recommend the following book to anyone interested in test management:
Title: Managing the Testing Process (2nd Edition)
Author: Rex Black
Publisher: John Wiley & Sons
ISBN: 0471223980
Links:
Sven S
System Administrator at University of Frankfurt/Main
Best Answers in: Career Management (1), Software Development (1)
As the "receiving end" of untested software and no expert on the costs involved for testing compared to non-testing and fixing afterwards, i only can add two things.
You can make money in many ways if you have to spend it on testing, restoring a tarnished reputation is next to impossible unless you are a global player.
For a small/medium company or a freelance developer, reputation is much more important than price in my opinion. I rather buy a more expensive product of a developer with an excellent reputation than a product from a developer who is known to have their customer beta-test their software.
Definitely an investment. In today's world, end users look for the quality of the application/product. It has become a *must to perform. Failure to do, will result (in most cases) in a loss of customers/clients, and many other known/unknown losses. Never underestimate testing. Timely and wise investment will definitely lead to a successful growth.
~Tushar Shah (tshah0415@yahoo.com)
John W suggests this expert on this topic:
Jaap, I am quite sure that you can add to this discusion?
i am not any software geekos like others,,, but its important to test your software before releasing,,,, it will be a total blessing for you .
Links:
Software Testing is an activity to check what you've actually developed. Until you haven't done it - you don't actually *know* what you've got, you can only guess. Of course, you may cut the corner and to light and informal test or just throw beta to your users. Some industries tolerate that (e.g. web startups), other - don't (think embedded software).
Since it's an inevitable part of being in software business, I decide to write it out as an investment. In financial term it is like buying option (options of different size) that lower your further risk.
Dennis K
Manager at Accenture
Best Answers in: Offshoring and Outsourcing (1), Computers and Software (1)
I would put that as an investment, because its always part of a larger IT deployment or change, which is justified by a business case.
But as far as IT investment is concerned its a cost. To ensure that the investments we made actually delivering the results we wanted.
So its a chicken and egg situation.... No SW goes without testing .
I would classify it not as a cost, nor an investment, but as insurance.
It isn't a pure outlay, nor does it have tangible returns. It does, however (when done right), prevent or mediate significant revenue loss due to defect.
I would say that there is no definite answer on the question.
First of all there is minimal software testing, which has to be done. How much? It is a function of multiple factors:
• Number of licenses, users, etc.;
• Requirement to the software stability and dependability;
• Your technical support model, cost, number of people, etc.
This part of the testing is an integral part of your development process and it is your cost.
Then there is another component. In some situations (not always) you can spend money providing maintenance and technical support or instead spend money on testing and QA: tools, testing/QA procedures, testers, actual testing (time, hardware, software).
That portion can be considered an investment: spend on testing and QA, save in a future on: avoiding loss of business, technical support, bug fixes, distribution of patches, etc.
Eileen B
IT Professional, Information Security Quality Assurance Operations & Administration / President, CMU SEI LI SPIN
Best Answers in: Using LinkedIn (53), Staffing and Recruiting (12), Career Management (12), Computers and Software (8), Quality Management and Standards (7), Software Development (7), Web Development (7), Ethics (6), Change Management (5), Professional Networking (5), Enterprise Software (5), Freelancing and Contracting (4), Job Search (4), Accounting (4), Government Policy (4), Internet Marketing (4), Organizational Development (4), Project Management (4), Education and Schools (3), Business Development (3), Supply Chain Management (3), Blogging (3), E-Commerce (3), Databases (3), Travel Tools (2), Certification and Licenses (2), Personnel Policies (2), Internationalization and Localization (2), Contracts (2), Employment and Labor Law (2), Advertising (2), Public Relations (2), Business Analytics (2), Corporate Governance (2), Inventory Management (2), Manufacturing (2), Personal Taxes (2), Professional Organizations (2), Biotech (2), Computer Networking (2), Commercial Real Estate (1), Customer Service (1), Facilities Management (1), Regulation and Compliance (1), Conference Venues (1), Corporate Taxes (1), Economics (1), Government Contracts (1), Government Services (1), International Law (1), Treaties, Agreements and Organizations (1), Criminal Law (1), Antitrust Law (1), Intellectual Property (1), Direct Marketing (1), Guerrilla Marketing (1), Labor Relations (1), Planning (1), Bond Markets (1), Hedge Funds (1), Market Research and Definition (1), Starting Up (1), Information Security (1), Information Storage (1), Telecommunications (1)
I think the best way to respond to this is to have you think about a few things:
1. What is the cost to your budget and bottom line for poor quality software that makes it to your customer?
2. How much does rework cost your budget and bottom line for poor quality software?
Clearly, your answers should tell you that quality is a cost effective investment.
Eileen
Kevin B
CIO / CTO, ITIL and IT Governance Consultant @kevinbehr on Twitter
Best Answers in: Organizational Development (3), Computers and Software (2), Business Analytics (1), Change Management (1), Project Management (1), Career Management (1), Information Storage (1)
If software is not tested how do we know it actually overcomes the constraint it was designed to?
THe cost of defect repair is lowest when the defect is detected earlier in the life cycle. Waiting until a user or situation finds the problem makes the costs of poor quality go through the roof. Not to mention legal, regulatory and loss of reputation issues that can arise.
Remember, that the only reason business uses technology (read software) is to diminish or remove an existing constraint. The only way software can do that is if it operates without interruption. Software onyl delivers value if it is running..when it has broken it now longer delivers continuous value.
When software breaks it not only stops diminishing constraints it actually creates new constraints..the very opposite of the reason it was created!
my .02
kb
★ Stu M
★ Veteran Project Manager ★ Snr QA Manager/TestLabs ★ Business Owner ★ Web Developer ★
QA should manage all Development teams!
Software testing is an absolute must...let it be an internal or an external release. A buggy product can kill a company. I am sure the IT world has many such examples.
One thing I would like to add is, testing should not only be towards detecting bugs, but also towards making the software user friendly. I know this falls under a different domain, but I really believe a user friendly software (even with a few bugs) can go places! Take the example of Windows (any version)....I love MS btw and I am a major Windows fan.
Sheridan L
Experienced IT Project Manager for Hire
Best Answers in: Event Marketing and Promotions (1), Career Management (1), Ethics (1), Computers and Software (1)
Software testing is an initial cost that, if done properly, has a direct impact on future viability and profitability. Being first to market may have a huge impact on the number of copies/contracts you sell in the first month. If you are first by sacrificing a strong testing regiment, you will not be able to sustain sales/interest of your product as bugs and limitations are discovered by early adopters / reviewers. No team is so good that they do not need a testing strategy.
Never mortgage your future for the sake of your present.
Would I call it a cost or investment? I’d say the answer is irrelevant. What your organization calls it is what’s most important. The make up of your organization and its goals is going to drive that.
First if your organization can look at a project and determine that catastrophic issues with deployment are a somewhat tolerable situation, you’re probably not going to consider it an investment. (In other words, production has your largest tester base.)
But if that’s not the case…
Many of the answers here have focused on the development staff, and there is obvious merit looking there. If your development staff is sub par you’ll probably really want your testing to take place.
However, if your organization has difficulty developing requirements documentation that is an accurate representation of what is truly needed; you may also discover that testing is very much an investment. This is because testing will not be so much about debugging, but also discovering all the ways in which clients can say: “That’s not what we need.” If project QA meetings often involve questions like: “Why is the software doing XYZ, when we need it to do ABC?” you probably fit in this category.
Anyone who says it is a cost, does not know what testing is all about. Testing should be part of a project. Testmanagement should be involved at the start of a project for reviewing requirements from testing point of view. Proper reviewed requirements leads to better and faster development.
With 10 years of testing experience i've seen enough projects fail or delayed just because of the underestimation of the role of testmanagement and the value of testing.
