Software testing optimization... what are current best and next practices in testing?
I want to understand how to optimize software quality by adopting best practices that span people, process, and tools. What are current best and next practices in testing and what key areas in software testing practices bring the most benefit to an organization looking for opportunities for improvement in testing?.
Good Answers (5)
Bryan C.
Global Engagement Manager, OnDemand Practice at BMC Software
Best Answers in: Software Development (2), Organizational Development (1), Project Management (1), Enterprise Software (1)
Much of this depends on the underlying technology but as has already been posted, leveraging a Test Driven Development (TDD) and Unit Test coverage is a good fit for more advanced technologies (Java/.Net). Continuous Integration which monitors code that is checked in to your Source Code Management tool and confirms that the entire build is still viable is also a good way to increase software quality but again this fits more contemporary technologies.
Another way to increase software quality is to improve the accuracy of software requirements. A growing movement in this area is called Executable Requirements which allows business users to specify pass/fail scenarios for their requirements that developers can write their code against (and which then act as tests that can be automated and executed to validate new code doesn't impact previously documented requirements).
Generally though, regardless of technology, I would emphasize Testing at the source, ensuring that business goals, requirements, design and implementation activities are all driven by quality parameters. I would also encourage incremental testing (verus waiting until everything is completed) and ensure good metrics to understand what has been tested and the number of defects discovered throughout the testing process.
I've put together some links to the material above as well as some articles I wrote on similar topics that might be of value including one called Ten Tough Questions To Ask When Developing a Project Test Plan.
Links:
Sagy Pundak M.
Experienced Software Development and Quality Executive
Best Answers in: Computers and Software (1), Software Development (1)
If you are looking for quality improvements then testing is likely to be the wrong place. You can't test quality into the product, Quality has to be built into the product. Testing can do a good job at measuring the quality of your product so it still plays a vital role in the process.
As far as testing practices, your best bang for the buck ("optimizing") I would suggest that you look at:
1) Inspections, code and documents (see link below or contact Tom Gilb).
2) Proper Unit Testing by the developers (include coverage and reporting).
3) Prototyping, especially if you can user feedback (this will improve you code and your tests).
4) Allocate time and resources for each test cycle and stick to it (don't shorten the time because the build is late, start a cycle before the previous one ended.
5) Design & code with testing in mind (how can this be verified/validated).
6) Automate your regression testing and run it on every build (even nightly if you can)
7) Move as much of the V&V activity up front (the earlier in the process you find it, the faster and cheaper it is to fix it)
8) Design your tests to verify the requirements/expectations/user needs not to verify that the code "works as coded".
9) Document analyze and track both test results and defects
10) Perform positive root cause analysis on escapes (bugs that made it to the field) and see what can be done to catch this class of bugs in the future.
Links:
Jessica M.
Product Director at NIIT China Ltd.
Best Answers in: Business Development (1), Biotech (1), E-Commerce (1), Enterprise Software (1), Computers and Software (1), Computer Networking (1), Databases (1), Information Security (1), Software Development (1), Web Development (1)
A search on "Software testing" on RequestFill.com reveals these result:
Effectively building and deploying qualify software is what every company focused on. Measurement at every stage is the only way by which one can track the quality of a product. Some kind of metrics is needed to prove that the quality of the product is good/bad. Practical and effective methods of testing the software should hence be adapted to arrive at these metrics. Measurement of software testing should start from the requirement stage to the product implementation stage to assure the quality of a product and each stage should be monitored effectively.
It might sound obvious, but one of the most important tasks of the software quality assurance life cycle is collection and analysis of test results. Lacking metrics, developers—and testers—would have no idea if they were making progress in removing software defects.
The software industry has witnessed a dramatic rise in the impact and effectiveness of software quality assurance recently. From its day of infancy, when a handful of software pioneers explored the first applications of quality assurance, to the development of software, software quality assurance has become integrated into ll phases of software development. At Virtusa, an industry-strength QA process is an integral part of every one of our engagements. Virtusa with its expertise in SQA has developed a unique point of view with its Software Quality Assurance approach, an innovative SQA methodology that extends beyond defect reduction.
Jessica
Links:
Andrew R.
Program Management Lead at Oceaneering
Best Answers in: Sales Techniques (1), Green Business (1), Green Products (1), Computers and Software (1)
I'm really pleased to see so many answers addressing the root cause of quality: the need to write defect-free code in the first place that performs the intended function. Great specs, reviews, inspections and the like are all great practices, but I think Evelyn needs a shorter-term answer to help while all those great long term practices are getting institutionalized. Here are some things to consider:
1. If you don't have one already, put a good defect tracking system in place. The only thing worse than discovering a defect is discovering it a second time because wasn't recorded or the record got lost. There are a lot of commercial packages, and you can also try bugzilla, which is free. At the very least, a spreadsheet with checkout/checkin version control can work for a small team. All of the packages are pretty good; the deciding factor will likely be which fits in best with your other tools and can be supported with your present support staff.
2. If you can, use a test management package. That way you'll be able to focus more on fixing your quality problems than on counting and recounting how many test cases have been written/executed/failed/etc., and trying to keep track of which defects match which test failures. I've used Mercury Quality Center and found it easy to setup and use. There are other packages too, but none that I can speak to first-hand.
3. Test automation is great, but be sure you first have the basics of manual testing. You'll learn a lot getting the manual tests into shape--don't spend the effort on automation until you're sure that your requirements and system are stable enough to make automation worthwhile.
4. Be sure to test not only that the software does what is intended, but that it doesn't do what is not intended. This kind of "negative testing" is essential, and can require creativity since the unintended consequences that software is to avoid are rarely specified in the requirements.
5. To test complex logic trees, consider using a "Design of Experiments" approach. In a complex system, you cannot test all combinations but you can pick tests that increase your likelihood of finding defects.
6. Consider performance testing. This can be difficult, since test environments rarely map exactly to production environments, but you want to get some assurance that your software won't signficantly slow the machine on which it is running--or bottleneck the network or other system components.
This is a start. My apologies if it's too elementary or not well suited to your specific needs.
Andrew
Next best practices coincide with the Agile Manifesto, in which People have precedence over the processes.
There should be no wall between Development, Testing, Domain Experts(Business Analysts) and Test Automation Expert. Everyone has to contribute to optimize the software quality because they are all responsible for the end product. We just implemented SCRUM methodology and getting immediate benefits out of it.
Every team member is responsible for the Quality of the piece of code they are working on. Business Analyst clearly specifies the acceptance crtiterias in collaboration with the team. The team works on achieving the tasks based on these functional requirements and acceptance criteria.
In the meanwhile Manual Testing works defining an overview of the possible test cases and share them with the team in a meeting. This meeting involves Manual Testing, Test Automation Expert and Development. The outcome of this meeting is the list of candidates (test cases) for each disciplines. Development will cover certain acceptance criteria while developing unit tests. Manual testing will focus on the UI part of the application and test automation will focus on automating the candidates for regression testing. Test Automation is also responsible for maintaining a suite of test scripts.
You can select any tool based on your requirement.
More Answers (17)
I would say the current best practice is test-driven-development (and unit tests), because:
- TDD/UT is process changing in the way the code has to be / is then written to be tested (which usually ends with code that's more maintainable)
- TDD/UT ensures that code does not break when refactoring (at least, as far as the depth of testing goes)
I guess the future of testing would involve a level of reflection and code analysis looking for problems (but that's a bit sci-fi for now ;) ) Also, automated tests for things that are difficult to test right now (like user interfaces)
Links:
Julian F.
Test Analyst at iLAB
Best Answers in: Quality Management and Standards (1), Computers and Software (1), Software Development (1)
The first thing to do is to use Inspections. In this technique, requirements documents are taken and inspected in detail. Are the requirements clear, do they contradict each other, do they match the source documentation (if there is any)? etc. Then the issues found are logged and the document author fixes the document or clarifies as necessary. This is a best practise as most errors are usually found in the documentation.
As mentioned by Thiago, Test driven development is a best practise as it focusses on "test early, test often".
Jose C.
Software Architect, Owner efeKctive
Best Answers in: Quality Management and Standards (1), Software Development (1)
Maybe take a look at a white-paper I wrote. It gives you a different perspective of where the bottlenecks are ( a good starting point)
I would say a good future step would be to automate, and speed-up, test creation and execution to move QA personnel upstream, like the auto industry.
Hope this helps...
José
Links:
My favorite way how to optimize testing is to automate it. Although it is not suitable way in every case (like interfaces change much and often or for testing GUIs) automation of SW testing can give really amazing results. Especially once you manage to not just automatically execute test but also automatically generate them.
Automate testing process is really worth in cases when you are developing long life software where you can expect long time of maintenance. Since build the infrastructure for test automation requires some effort. But every testing then is very cheap. But building proper infrastructure for test automation is not that difficult once you have some experience with it.
I can add to the other good answers (inspections, TDD, automatization...) the testing/process associated with performance... most of the protect managers considers that the performance is just an issue in production, or in the best case just a testing problem... teams with high level quality in terms of testing are doing a complete life cycle of performance covering all the project phases: requirements->targets, design-> budgets, coding->best practices, testing -> performance testing. This kind of process reaches a very big increase of the project quality.
Michal P.
Senior Software Designer, Technical Project Leader at CertiCon a.s.
Best Answers in: Software Development (1)
Unit testing is certainly very good practice to use in your case and I recommend it.
There are several advantages UT (TDD) approach. The tools needed for it are part of modern IDEs, therefore it is easily available. Running the tests is also very cheap. Checking consistency during refactoring was already mentioned above. A very important advantage is that making your code suitable for unit tests promotes good design and architecture.
There is also one pitfall of unit tests. That is the term "code coverage". Do not expect that your code is well tested if you have 100% code coverage. This number can only tell you if some part of your code is not tested. I think that it is good to design unit tests based on scenarios rather than code coverage. This means, that if you have 100% code coverage, you may well be half the way...
Joy M.
pumps up profits fast
Best Answers in: Using LinkedIn (6), Staffing and Recruiting (2), Public Relations (2), Manufacturing (2), Professional Networking (2), Small Business (2), Job Search (1), Mentoring (1), Occupational Training (1), Foreign Investment (1), Personnel Policies (1), Events Marketing (1), Business Development (1), Search Marketing (1), Writing and Editing (1), Corporate Governance (1), Planning (1), Philanthropy (1), Inventory Management (1), Supply Chain Management (1), Branding (1), Market Research and Definition (1), Engineering (1), Interface Design (1), Career Management (1), Communication and Public Speaking (1), Ethics (1), Databases (1)
Whether you're doing automated or manual testing, the best practice is always testing against a user requirements document that accurately reflects the end users' business needs.
First you need to look at the approach. Will it be top down or bottom up?
Is it one group, one application that you are trying to optimze? Or is it a large organization with multiple systems/applications?
So one at Logica we use a Structured Test Method, TestFrame. Secondly your test model/method may not be that bad. These are things to take into consideration. So for example I wouldn't tell someone to use TestFrame or TMap, but advise them that their testing be structured. Third, how do you manage your testing? Then last do yuo have a test policy and is it being followed?
More information here via Web Resources here:
Links:
Clarification added April 17, 2008:
Once you have analyzed these areas, does the test process need to be improved? If so look at the Test Maturity Model from the Illinois Institute of Technology, and possibly Test Process Improvement (TPI) from Sogeti and Tim Koomen as guides. Each improvement model has good templates.
John R.
Media Test and Tools.
Best Answers in: Computers and Software (11), Software Development (7), Web Development (7), Blogging (6), Enterprise Software (4), Education and Schools (3), Computer Networking (3), Biotech (2), Internationalization and Localization (1), Internet Marketing (1), Corporate Governance (1), Organizational Development (1), Planning (1), Starting Up (1), Wireless (1)
Evelyn: Have seen emphasis on customer needs, requirements definition, picking software that fits the problems, UML, design for test, built-in-self-test, adapting teams to problem resolution, hardware integration and drivers, refactoring and automation including code which outputs some of its own tests. Also see link for public domain sources. Thanks.
Links:
Even more critical then if you choose basic unit testing, functional testing, BDD, TDD, ect... is the feedback loops around those tests, and the effort it takes your developers to write those tests.
Old-school unit tests are going to win out over any new-school BDD/TDD if the unit tests get automatically run every time a file being worked on is saved, over the BDD/TDD tests that only get run before nightly builds. Don't eliminate the tests on the nightly build but make sure that those tighter feedback loops are in place.
Here is an example of such a tight testing feedback. http://benburkert.com/assets/2007/11/9/autometric.mov
Some good answers so far - I would add that to get the most benefit you should ensure you are doing risk based testing and making sure that the focus of your testing is on the highest business risk and likeliest technology risk.
To find out which areas these are would mean talking to the business users and the developers which in turn should lead to greater understanding
Charles S.
R&D Architect, Software Solutions and Testing at Research In Motion
Best Answers in: Wireless (2), Job Search (1), Mergers and Acquisitions (1)
As a long time tester and test book author, my answer is that testing will be optimized when software is constructed correctly so testing only needs to be done one time in order to establish that fact. Ideally with appropriate tools and practices, we would work in a utopia where the development tools and methods produce code that we don't have to test because we can depend on it being right.
Yes, that would practically put me and others in my profession out of business, but preaching this for the last 15 years has not brought it about in my company so I still feel secure divulging this secret.
Even inspections, which are a form of "pre-testing" can be eliminated if you can achieve correct construction. So itotal optimization will result in a $0 Cost of Quality (cost of labor, service and material expenses associated with establishing that the software works). Based on prior measures of the downstream cost of each fault, 1 dollar (or you local unit of currency) spent on tools, facility arrangements, more talented programmers, etc. towards producing correct code is worth 10 units or more spent on other downstream activities like testing, fixes, re-building code, etc.
If you *have* to test, then spend on stuff that gets the most testing done as early as possible in the development lifecycle instead of discovering a big pileup of surprises that can include everything from missing requirements to insecure code at the very last stage of the project. Amen.
Rather than focusing in improving process and tools (which may work well or does not work quite well at certain environment or class of project), you would get a better result by improving the testers’ and test lead’s skill in software testing. Some people such as Cem Kaner and James Bach spent lots of their time in improving the education for testers. Try googling BBST (Black Box Software Testing), this is one of the testing course prepared by Cem and James highly recommended to every one who is interested in improving their testing skill. The BBST training material is available for free to be reused under the creative commons license.
First, you'll have to realize that testing is only a detective measure in quality management. It would be cheaper in the end to avoid bad quality (i.e. lots of bugs) in the first place.
I'd consider testing as a way to gain insight into the product quality before taking it into production (or releasing to the market). The test processes that I see these days are really bad at getting a reliable insight: important defects are still missed and show up in production anyway. At the same time, testing is always considered too expensive.
The company I work for developed a test process for structured testing under the name of TMap (Test Management Approach). I myself use their test process improvement model (TPI) a lot to stepwise improve existing test processes.
Both TMap and TPI are published in books that are available at e.g. Amazon. Just check them up or drop me a line for more info.
Antonio S.
Marketing Manager
Best Answers in: Regulation and Compliance (1), Guerrilla Marketing (1), Business Development (1), Market Research and Definition (1), Ethics (1), Software Development (1), Using LinkedIn (1)
I would recommend to integrate defect collection for tracking with a mechanism to allow a reasonable prediction of future defects ratio. If this is done efficiently, the whole Q&A can provide an effective input to strategic planning such as non-conjunctural launch roadmap.
There are several techniques available in literature: for example, look for "defect pooling" and "defect seeding".
Dave S. suggests this expert on this topic:
Danny is a software testing expert; his web site is http://tejasconsulting.com
There are different stages to the overall testing of a solution such as unit testing, integration testing, performance testing, user acceptance testing etc. Just like development, being able to find the issues as soon as possible reduces the cost of fixing any issues found.
All the different tests should ultimately derive from a requirements document. It is important that the requirements are clear and that they are testable. My experience is that the use of use-cases to describe requirements have two main benefits:
Firstly, the users can see how the solution woulr interact with them from the requirements, therefore reducing the amount of rework required when the user first interact with the solution.
Secondly, tests can easily be derived and reference use-cases.
