How effective is Code Coverage Analysis ?
This is a multi-part question:
1. How effective is Code Coverage Analysis ?
2. In your company is this activity done by developers or by QA ?
3. In your opinion who should do this activity ?
4. What tools do you use to instrument the code ?
5. Have there been instances where instrumenting the code changed the logic of the application (not performance) ?
6. Does your QA organization have a functional environment which includes debuggers and
a performance environment which is exactly like production ? Is there some process or
tool that ensures that instrumented code drops are only installed in the functional
environment and only production code drops are installed in the performance
environment ?
Good Answers (12)
Swamy, great questions!
1. Code coverage is effective when done properly. Measuring which lines of code have been executed is not very helpful, measuring which paths in a method have been executed is much better.
The question you have to ask yourself is - what happens if our code coverage is low? What is an acceptable level of code coverage?
2. Neither - coverage is most effective when it is part of your build process via Continuous Integration (using cruisecontrol or hudson for example)
3. Who ever runs the builds is best suited to perform coverage analysis. Developers should write the unit tests, QA should write Selenium tests, Build team does coverage analysis.
4. We use our own - CoView for JUnit creation, Comet for coverage via Ant. You can take a look at them at www.codign.com
5. Not in our case
6. Yes, our QA organization has test servers setup that are 100% that of production. Our continuous integration process sets this up for us automatically.
Geoff F
"Hands-on" Software Architect and Senior Developer
Best Answers in: Computers and Software (21), Software Development (18), Web Development (15), Enterprise Software (10), Wireless (6), Blogging (5), E-Commerce (5), Information Storage (5), Telecommunications (5), Offshoring and Outsourcing (3), Biotech (3), Starting Up (2), Computer Networking (2), Information Security (2), Occupational Training (1), Internet Marketing (1), Graphic Design (1), Search Marketing (1), Planning (1), Non-profit Management (1), Databases (1), Using LinkedIn (1)
1) Extremely effective, this assumes a good quality analyzer. It also assumes intelligent use as most analyzers will find more problems than are important to correct for typical businesses. There should be a "standard configuration".
2) BOTH it should be done by the developers and then spot checked periodically. More typically by an architect though.
3) Primarily developers, they are responsible for the quality of their code.
4) Visual Studio
5) Of course, if it is done badly anything can happen
6) No. They should do configuration management and periodic running of Unit tests. Yes, instrumented code should have switches to turn it on or off. This should include both conditional compilation and configuration switches as needed.
1) Code Coverage is helpful for the developer during tests execution. Usually most of it comes when the developer executes his Unit Test Driven plan (see TDD). The Code coverage helps:
i) primarly, to examine if unit tests exercise all source code
ii) to optimize parts of code that execute really often and affect performance
2) Usually this is done from the developer during the automated Unit test execution. QA could help on this when the developer provide a special build that he will later examine. Thus the developer will see that the Scenario Tests executed by QA exercise most (or even all) of the source code
3) The developer. QA person (usually) has no idea about the source code (even he might not be a programmer)
4) We have occasionally used Perl and Devel::Coverage ( http://www.softwareagility.gr/index.php?q=node/10)
5) Instrumenting the code shouldn't change the logic/requirements. Unit test will keep this conistent. It might affect performance, but the instumentation code will be removed later from the production code
6) Developers use the functional environment, maybe some QA people too. The performance environment is used by QA people for scenario,performance testing. Compiler directives take out the instrumentation code from the production one.
1. Very
2. Both
3. Both. Developers should do it to improve their unit test and code quality. QA should do it and provide reports to management.
4. For Java Clover and Cobertura (open source)
5. No. There have been instances where we have deliberately changed the logic once coverage reports were examined.
6. We have separate build scripts that instrument the code and do the analysis. These create their own artifacts and are run during a different process than our production builds.
Code coverage is a very effective tool for improving quality, but it has to be used intelligently. In particular, you need to clearly define what is supported and what is not. You do not want to waste time and effort "covering" things that aren't supported, or debugging these tests when they "fail" (which they often do, since the features you're testing aren't supported.)
Coverage analysis can be done well by either developers or a separate QA organization. There are two things to watch out for. First, no one should be looking at coverage of code they developed. It is too easy for someone to make a wrong assumption when developing, then make the same wrong assumption when doing coverage.
Second, coverage needs to be an important and rewarded part of the job for whomever is doing it. If your coverage analysis is done by a developer who is only being measured and rewarded for how many modules they can develop, the result is either low coverage numbers, or meaningless ones.
1. It depends on what you mean. It is highly effective at determining how much of your code is touched by testing so it is excellent for identifying portions of the code that aren't being properly tested. However, it does not tell you whether your tests are doing a good enough job, especially with highly data-driven code. It works best when applied to focused unit tests.
2. Both.
3. Both.
4. We code mostly in Curl and the standard Curl IDE includes both profiling and coverage measurement tools.
5. No. I would consider it a bug if coverage instrumentation did anything other than slow down the execution of the program, which should not break most correctly written programs.
6. Yes. In Curl, this is trivial.
Links:
Josh N
Progammer, Software Engineer, Developer, Hacker, Slayer of Tiamat, etc
Best Answers in: Web Development (1)
1)
It is very useful, but it certainly isn't a silver bullet. As has been mentioned, it does tell you which code hasn't been touched by tests. It does not indicate how well tested the code that is touched by tests is.
For example, imagine a method which takes a string. If it only gets called with valid values, you could easily get 100% coverage for it. But the second you pass it a null value, it blows up.
2)
When I've been on projects with a QA, this has typically been by the development side of things. QA didn't typically have anything to do with the automated developer-written tests.
3)
Developers. The reports typically go line by line over all the source code showing what lines are or aren't covered. This kind of detail might or might not be appropriate.
4)
It's going to depend on the development environment. We use Java, and have been using Clover.
5)
Not that I recall.
6)
Not for projects I've worked with a QA organization.
Hi Swamy
Very good questions and very good answers also.I only want to reinforce some ideas and in this way my answers to your questions are not in the very order you have enunciated.-
Code coverage analysis is a process for PROGRAMMERS. It's main mission is
Find areas of a program NOT exercised by a set of test cases (I.E. not tested by QA people). Is the kind of test we call STRUCTURAL because test program behavior against the apparent intention of the source code. This is VERY DIFERENT with functional testing which compares test program behavior against a requirements specification.
So this is an direct answer to your questions #2 and #3
If you can find those areas of your code not tested by test cases you can AFTER Create additional test cases to increase coverage and also identify redundant test cases that do not increase coverage.
Also Code Coverage analysis aids in determine quality of a prduct if you can obtain a QUANTITIVE MEASURE OF CODE COVERAGE (metrics) RELATED TO USE: Statement coverage, condition coverage, data flow coverage, loop coverage, etc, tec).-
If you achieve these you have a very useful tool. This answer your question #1. Caveat Emptor: The very esence of the test is prove the code written (NOT THE NOT WRITTEN !!). So functional test can discover other kind of errors (omission of functionality for example)
The task to do code coverage in a manual way is very hard and prone error. So is clever use a automated tool(atlassian clover and NCover). This is an answer to #4 and indirect answer to #5: IF YOU CHANGE WHAT YOU TEST WITH THE TEST ITSELF YOU ARE IN PROBLEMS (Is not the Heseinberg's uncertainty principle !!)
And last (but not least) #6. We develop and maintain software for different companies. Nobody has a PRODUCTION enviroment REPLICA . Is a matter of money.
If this answers are useful for you don't hesitate in contact me:
eduardob@m3sa.com
Regards
Excellent questions. Thought I would share my perspective. I code mainly in C++ and coverage analysis is an extremely effective tool. As a matter of fact that is probably the only way to improve s/w quality other than of course to use memory management tools like Valgrind etc.
2. This obviously can only be done by developers. We are trying to look into a complex C++ code and coming up with a test case that would exercise this. QA would not have any clue on this.
3. Developers. Asking QA to do this at the block or functional level is ok but if I am testing an if-else condition with multiple clauses I don't think it's going to go much far if QA folks are called in as only the developer who wrote that can decode what it does in most cases.
4. Quantify from IBM is the best
5. No, this should never EVER happen -- if it does tool bug and a serious one at that
6. When it comes to C++ coding, running tests built with optimized compiler settings in regression should be good enough. Often the debug and release executable outputs don't match. These are always worth looking into.
Regards,
Arpan
Ashish P
Engineerg Manager, ASF Committer, IEEE Senior Member
Best Answers in: Software Development (1)
Swamy, my thoughts on this
1. Code coverage is very effective for Developer, Project Manager. Here is how
Developer -
a) Knows which all code paths have been tested
b) Can analyze which portion need more test cases to be added
c) Publish some section could not be tested coz of complexity involved. Application Server Environments, at times, make 100% code coverage a little tricky.
Project Manager
a) Can view the report and plan for further refinement
b) Use this as source to see, if Code coverage is solving its purpose or has become just an activity. Careful analysis of Code coverage report is must. Can use the subsequent test results to strength the test suites
2. Its automated. Test cases are primarily written by Developers. Once done, anybody can execute the ant task and generate the report :-)
3. Its of prime importance to the Developer, hence he should own it. Execution role is not important. Anyone should be able to do it, if automated. Else Developer should generate Code coverage report as part of Unit Testing. QA can optionally have this activity during functional verification, to see if any dead code is present :-)
4. We sue emma and Cobertura for instrumentation of code
5. Nope, don't know of any instance wherein such a thing has happened. Instrumentation just add meta-data to the code, to generate coverage reports. They don't interfere with program logic
6. Our QA doesn't have access to these. Not sure how process or tool can be any help in this. Can follow a simple logic, to generate instrumented code only when needed.
It will be great to have this process in place
1. Have a test suite to test your code (typically JUnit on nUnit)
2. Have Ant as your build tool
3. Have different tasks for build and instrumentation
4. Integrate compile, instrument, test and report generation
5. Send a mail once done
6. Can use Maven, CruiseControl or Hudson for build Management
From Quality perspective, collect this data for a period of time and keep analyzing the matrices. They have lot to say about project health :-)
thanks
ashish
My modest answer would be a repeated one!
1. Code coverage analysis(CCA) to achieve quality is as good as the cases considered while coding and then testing. CCA can only find out "wasted efforts" (in unreachable codes) and effectiveness of "test cases". So CCA can be effective to ensure quality only if the test cases are exhaustive. Ofcourse it can't ensure quality through performance, brevity of code, memore usage, etc. So effectiveness of CCA and its association with quality is extremely limited.
The result of CCA needs to be analyzed carefully because say a high coverage can result even due to inexhaustive test cases.
3. In my opinion, CCA should not be done by the developer. Using a person who was not involved in the development is a simple way to avoid
negligence due to confidence in one's code and
negligence due to boredome which many developers feel as they would have exceuted most the test cases in the development/ UT process also.
Code Coverage is useful and required, the other answers vouch for it.
Come shortcomings
It is not a complete measure of test coverage, e.g. executing the same test case for multiple data sets, boundary cases.
It is possible that a developer deletes needed code to achive a higher coverage report (you need to look out for that).
It is also possible that testers stop thinking about needed test cases once coverage targets are achieved.
More Answers (1)
Mohammed Hussain K
Head - Content & Client Servicing at K WEBMAKER™
Best Answers in: Using LinkedIn (22), Web Development (3), Personnel Policies (2), Lead Generation (2), Project Management (2), Professional Networking (2), Education and Schools (1), Job Search (1), Mentoring (1), Occupational Training (1), Venture Capital and Private Equity (1), Staffing and Recruiting (1), Offshoring and Outsourcing (1), Business Development (1), Customer Relationship Management (1), Search Marketing (1), Writing and Editing (1), Non-profit Fundraising (1), Non-profit Management (1), Personal Investing (1), Positioning (1), Professional Books and Resources (1), Career Management (1), Computers and Software (1), Wireless (1)
N.A>
