Steps to create a testing productivity baseline
Clarification added June 16, 2008:
A Lot of projects are getting offshored primarily to be tested.Customers are very keen to understand testing productivity. Testing is a complex discipline to provide a single measure because of parameters to be applied like
a) Test Case Prepartion and Test Case Execution
b) Testing Phases like Functional, Component, Integration, Regression and etc..
c) Testing Approaches like Manual and Automated
c) Project types like Desktop, Client Server, Web, SOA etc..
Given the above set, how would you approach the creation of a testing baseline.Please share your thoughts with example.
Good Answers (2)
Nagesh B
SDLC support functions: Build, SCM, QA, SysAdm and Enterprise Data Centre management.
The productivity baseline for testing is tricky. The baseline of number of defects caught per X number of runs can indicate two things: Effectiveness of test OR failure in developement (could be in the area of design/requirement/code etc.).
Typically for a large or complex product, there are at least 3 types of test (i) a unit test that is run by the developer, (ii) integration testing usually the nightlies or the test harness, when integrated with different modules (iii) System testing or sometimes baseline testing or testing before product release; usually involves testing as a complete system (*all* modules as in the final destination). In some organizations, there may be a set percentage of defects caught at each stage e.g., if a product were to have 100 defects, the ratio could be 70:25:4:1 where 70 is the number of defects caught by unit testing and 25 by the integration test and 4 by System Test and 1 by the end receipient (customer).
These ratios and different types of testing may differ from organization to organization. The baseline can be tweaked further once you start hitting observing the performance of your organization and start comparing with past history or with other business units or the competitiors..
Bob M
Senior software developer
Best Answers in: Software Development (15), Ethics (11), Computers and Software (8), Government Policy (4), Staffing and Recruiting (2), Air Travel (1), Business Dining and Entertainment (1), Hotels (1), Travel Tools (1), Education and Schools (1), Mentoring (1), Exporting/Importing (1), Offshoring and Outsourcing (1), Employment and Labor Law (1), Business Development (1), Corporate Governance (1), Planning (1), Small Business (1), Starting Up (1), Information Security (1), Web Development (1)
One factor I don't see mentioned often is the quality of the defect reports. If a testing team does a great job of testing, and generates a lot of bug reports, but the bug reports are not sufficient to allow the developers to reproduce and fix the bugs, a lot of time can be wasted on clarification. So as part of the metric, I would include the number and/or percentage of reports returned for "more info" by the developers.
More Answers (3)
Jose C
Software Architect, Owner efeKctive
Best Answers in: Quality Management and Standards (1), Software Development (1)
I would suggest to skim a whitepaper I wrote about the financial and economic implications of quality assurance process. I think it would help a little bit in finding the bottlenecks, hence the appropriate productivity ratios.
Links:
Clarification added June 16, 2008:
Sorry for the typo...
There are various measures that can be incorporated especially on the testing side.
1. Requirements to Test cases traceability : This can give a measure of the coverage based on the requirements defined.
2. Number of test cases passed / failed during various test cycles : This can give a measure of how the application is doing at every test cycle. Each test cycle can contain incremental requirements being added or bug fixes or both.
3. Defects found per build : There should be a trend where the number of bugs found before the release should be near to zero.
4. Number of adhoc test cases created during test execution : This can indicate the risk due to either requirements change during test execution or insufficient test cases prepared during test planning phase.
5. Number of defects per requirement : This could give an indication of where the focus should be. Eg; If there is a requirement which was implemented by an inexperienced programmer there could be more defects in that area. If could also be due to misunderstood requirements, invalid defects logged or changing requirements.
The above are some of the measures or metrics that gives the testing effectiveness rather than just productivity. In my view, productivity measure is common between coding and testing. The estimation method could be different but essentially, productivity is just a measure of planned versus actual effort. There cannot be a predefined testing baseline where you predict the amount of effort required to do testing of a particular application. This can only be dependent on the type of application, new features added, stability of the existing system, existing test automation and experience of the team which counts.
John R
Media Test and Tools.
Best Answers in: Software Development (7), Web Development (5), Computers and Software (4), Education and Schools (2), Biotech (2), Blogging (2), Enterprise Software (2), Computer Networking (2), Internationalization and Localization (1), Corporate Governance (1), Planning (1), Wireless (1)
Md Abubucker: Agree that testing is complex. Have created a model of a Test Information Space as a way to organize efforts to capture data for analysis, presentation and use by design, manufacturing, and support. Phases may occur in series or parallel, factors may not be fully identified, and the output may be used in ways not specified up front. Also see links for example of test dashboard for progress indicator, troubleshooting and planning. Thanks.
