Have an idea to improve LinkedIn? Please submit your suggestion below.
What are the useful Metrics for Code Quality ?
The quality of any software applications will depend mostly on it's code base and it's important to know what might be the key metrics that help us to evaluate the stability and quality of the code base.
Clarification added August 10, 2007:
Let me thank all of you for your time & answers. They are insightful. The question itself is very and didn't have a scope associated with it.
I am looking at the metrics that might help the developer to prove that his code is reasonably good enough & not at the whole application level. The metrics on testability, complexity, coverage, performance, scalability & maintainability might help.
From my experience, it's mostly the gut feeling that matters when some one ships the code to repository & interested in exploring the options that help to improve upon. Code Reviews will be there and it may not be feasible to cross check against all the implications that are there in the system.
Clarification added 10 months ago:
I would like to re-open this query to see more feedback. There has been some good explanations over the blogs on this subject.
Clarification added 10 months ago:
Links
1. <a href ="http://www.testearly.com/2007/09/26/code-qualitys-singular-metric/">Code quality’s singular metric</a>
2. <a href="http://www.developsense.com/2007/09/most-useful-metrics.html">The Most Useful Metrics</a>
3. <a href="http://softwareintegrityblog.com/blog/2007/09/26/the-right-metrics">The right metrics</a>
Good Answers (8)
Michael B
Participant in the Workshops on Teaching Software Testing
Best Answers in: Software Development (1)
Measurement is "the empirical, objective assignment of numbers, according to a rule derived from a model or theory, to attributes of objects or events with the intent of describing them." (Kaner and Bond, 2004)
So what are you trying to describe about the code?
Quality is "value to some person(s)" (Weinberg), to which James Bach and I add "who matter".
It's not clear to me that the quality of a software application depends mostly on the stability and quality of its code base. Lots of other factors could come into play. If the product is the only thing that does what it does, then the quality and stability of the code base don't matter much. If the product isn't available to its users, then the quality and stability of the code base doesn't matter at all to them--although it may matter a lot to the development organizations. A product could have dozens of bugs per KLOC, but if those bugs don't matter to someone that matters, then the metric doesn't matter either.
In my opinion, it's not worthwhile to talk about metrics until you've determined attributes of the code that you'd like to evaluate. Most of those attributes are bound to be subjective, and putting a number on them is a case of reification error. Here are some of the things that I would consider important in evaluating a code base:
1) Is it testable? (Does it include logging, scriptable interfaces, real-time monitoring capabilities?)
2) Is it supportable? (Does it contain helpful error messages? Can it guide the user, the technical support department, the testers, and the developers on what to do when it gets into trouble?)
3) Is it maintainable? (Is it clearly written? Nicely modular? Easily reviewable? Accessible to those who need access to it? Is it subject to version control? Is it accompanied by unit tests? Are there assertions built into debug builds of the code?)
4) Is it portable? (Can it be adapted easily to new platforms? Operating systems? Does it depend on OS-specific third-party libraries?)
5) Is it localizable? (Can it be adapted easily to some geographic or temporal region that is different from its current one?)
Those are just a few of the questions I would ask. Note that few to none of them are expressible as a number--but that all of them are highly relevant to the quality of the code--that is, its value to some person.
---Michael B.
It's very hard to suggest formal metrics, but let me try.
1) Code coverage by automated unit tests.
Any part of code should be covered by one or two automated test cases. Code coverage can be measured in automatic mode by different profiles.
2) Length of functions (methods) / units (classes). The one with 60 lines for method or 1k lines for unit (class) is bad one. Not counting in-code documentation.
Can be measured in automatic mode.
3) Not only "length" but also "width" of code - how many included constructions are used in method? For example, if { for{ while { if { do {...}}}}} is a signal of bad code.
Can be measured automatically.
4) Using of code patterns. It's hard to extract formal rules for using patterns, but may be this will help:
- any class should have interface, unless it's a utility class with static methods
- no fields should be public, unless it's a data-only structure.
- static methods should be only in utility classes
- etc. :)
Of course, the stability of code should be proved by tests, not by any formal metrics. But such formal metrics can help to find problematic pieces of code.
I'd return that question with:
1) What for?
Examine why your collecting the metrics. What are you trying to demonstrate by collecting the metrics? This may affect what you decide to measure.
2) Who wants to know?
Who are you displaying the results to? If its senior management, the detailed metrics might get a bit lost, if its for the developers, then detailed analysis may be acceptable.
Perhaps ask the metric audience what they want?
Good Luck!
Quality is in the eye of the end-user i.e. the person or entity that garners the benefit of the deployed system. By this definition, every quality metric must directly influcence the end-user's experience.
A high quality application will exhibit the following:
Adherance to requirements
Rapid response to change in the business space
Enthusiastic user acceptance
Robust behavior
Reasonable or better performance
Deliver on schedule and within budget
Taking them in turn
Adherance to requirements implies that the software does what its sponsors intend it to do. Note the users, *not* the developers, own the requirements. The developers own the specifications, which must conform to and support the requirements. This is extremely difficult to measure.
Rapid response to change in the business space -- a.k.a. agility. This encompasses maintainability. On the analysis/design level, this correlates strongly positively with strong separation by subject matter. On the coding level, this is supported by low cyclometric numbers within method flow of control graphs. (This is a fancy, mathematical way of saying "keep it simple.")
Enthusiastic user acceptance, a.k.a. 'Usability'. User Interface metrics -- particularly the principle of minimum surprise -- apply here. Focus groups and other usability evaluation methodologies help here.
Robust behavior, a.k.a. reliability or hardening. Rule 1: all publicly visible operations must be complete -- i.e., must exhibit reasonable and predictible behavior under all possible circumstances. Insist on a tripartate method structure containing:
1. Validation of preconditions
2. The method implementation
3. Validation of postconditions and invariants
Reasonable or better performance. First and formost, negotiate the ground rules, which should have the form p% of transactions complete within time t, where 0 < p < 100 and 0 < t. Next, design in the performance from the get-go, making sure that the system scales appropriately. It can be as bad to over-engineer performance as it is to under-engineer it. Make sure to load test throughly before deploying.
Deliver on schedule and within budget -- this sounds straightforward, but may be the most difficult to accomplish. First, the technical side has to manage user expectations and must push back against infeasible schedules and budgets. Secondly, the development group must measure progress continuously and alert the users to any deviation from plan. Open communication and establish trust from the beginning; sometimes this requires the technical side to bear bad news.
I hope all this helps.
"From my experience, it's mostly the gut feeling that matters when some one ships the code to repository"
In the process I use the code is ready to commit to the repository when the unit tests pass. That doesn't mean I'm done with it, just that it's ready to check in.
When doing code reviews there are three metrics I like to use to help me focus on possible problem areas:
1 Cyclomatic Complexity
2 NPath Complexity
3 Kirks Metric (Halstead's program volume / LOC)
The Personal Software Process (PSP) Process Quality Index (PQI) is relevant here since you are looking for the metrics that can be used by developer to monitor the quality of his/her work during the code development before checking in to the repository. Other metrics derived from the number of defects found by test team will need to wait until the test team performs the integration and system testing.
The following is the PSP PQI copy pasted from the link specified below:
1. Design/Code Time
The ratio of detailed design to coding time - when engineers do not take the time to produce a thorough design, they generally make more design errors. To reduce this risk, design time should equal at least 100% of coding time.
2. Code Review Time
The time spent in code review, compared with coding time - by doing a personal code review before they compile, engineers can find a large percentage of their defects. A thorough code review should take 50% or more of coding time.
3. Compile Defects/ KLOC
The defects per KLOC found in compile - even with good review times and rates, the review still could have missed a lot of defects. For quality products, compile defects should be less than 10 defects/KLOC.
4. Design Review Time
Detailed design review time, related to detailed design time - a thorough detailed design review should take 50% or more of the time spent in detailed design. Anything less generally indicates an inadequate review.
5. Unit Test Defects/ KLOC
The defects per KLOC found in unit test - the number of defects found in unit test is one of the best indicators of the number that will later be found. When the unit test defects/KLOC exceed 5, subsequent problems are likely.
Links:
Hi
Code quality is measured against what it is supposed to do and how it is done?
Most of the time requirements, architectures and design define how good the code is.
Metrics that can be used are
1) Reliability - No of defects identified in multiple rounds of testing or stress testing or burn in test and observe the stability.
2) Code coverage -- % of code covered using tools such as VIsual pure coverage etc.,
3) Design and code traceability -- % of conformance to design. No of design units covered to total design units
4) No of unit test cases per feature or method or class. Minimum 4 test cases per feature ( +ve, -ve and stress, system interaction)
5) % Conformance to standards -- use of fx cop or jtest and codify the rules
6) Number of methods/class, (6 avg), no of lines / method < 60). Same can be described for procedures, functions etc.,
7) % of code documentation. No of lines of documentation/code base( atleast 15% in documentation). Javadoc has more details
8) Exception handling --- Clean exits for application and system exception with proper throw
9) DFT -- Design and code for testability -- Need to provide necessary trace to pinpoint errors. This will be a clear metric for MTTR ( mean time to repair)
10) elements such as scalability, performance would have been covered as part of the design and ideally should not be covered in code metric.
11) no of issues ( bugs and peer reviews)/ kloc at unit level
12) defect leakage % . Number of defects identified from system testing/ total no of errors. This will be a key metric
Random thoughts and you need anything more can be expanded
cheers
cvn
To me, computers is still a science. There's a lot of engineering to happen. Here are some useful tips, in addition to the good ones from below:
1. Apply GRASP patterns. You may/not find an opportunity for a design pattern. GRASP patterns are almost always applicable to most of the modern languages.
2. Doing analysis like bugs per KLOC or lines changed per KLOC was a good practice in COBOL days. Many a times critical bugs are solved in C/C++ by changing one or two lines. From my experience, I developed a different methodology to measure this aspect. I use the ratio of effort(man days) spent on fixing the bug to the effort spent on development. This gives you a better understanding of the productivity. Use historical data to calculate this ratio and set targets accordingly for your team. That works best.
More Answers (12)
Previous answer is OK for OO code. In general you should define for the language you use a Coding Standard that covers : way of use of various structures ( e.g.limited numbers of 'if then if then....' ).
Define also the readibility of the code : proper identation, proper ratio of comments to line of code (1:1.5~ is reasonable).
Identification of changes ( who, when, why) .
Lack of unused code is positive.
Use an automatic code inspection tool as much asyou can . Use a peer to peer code inspection to gewt input from experienced person.
Try to define a mixed metric of the Automated tool ( number of errors /lines of code and add to it a well defined grade as a result of P to P.
You may come to a reasonabele mertic that might require a period of tunning.
Answere by previous posts are good, I would like to add
1. Number of unit test bugs identified and fixed per kloc. This tells how deep unit testing was done, The tools used for unit testing.
2. Number of QA bugs found against the criticality of the bugs.
3. The test reports and reviews of the artifacts referring which the coding was done.
4. Peers review logs.
5. I donot give preference to the number of test cases while prefer the quality of test cases again not missing any minute requirement.
Links:
Although I have different approach to measure quality of code – By conducting static test (Check List). For example – we have created various checklist point each language or platform, like we have identified few areas for Core Java or C++ Code standard review point (Design Patterns, Modularity, Configurability, Cross-platform portability, Memory Management, Exception Handling, Type casting (Variable, Classes, Method etc), Design Specification Traceability etc). We follow this approach as it is very much difficult to ensure code standard from any Dynamic Testing Approach (Regardless of Automation or Manual Testing). This approach is bit costly we need competent reviewer (Kind of Technical Architect level) but this worth as per my experience.
David B
Software Architect - Software Development Manager at Siemens IT Solutions and Services
Best Answers in: Software Development (24), Web Development (5), Computers and Software (4), Career Management (2), Databases (2), Education and Schools (1), Business Analytics (1), Project Management (1), Enterprise Software (1)
Good programming is a highly demanding activity, that stands closer to some form of art than anything else. Symmetry, orthogonality, rhythm and simplicity are examples of the non-technical aspects of good programming. The term "beauty" is not far away.
Measuring "beauty" is certainly not a small feat. So, the metrics you'll get from static analysis tools will give you some interesting indications, but nothing more. You can argue that that's largely enough. I wouldn't.
There isn't any better metrics than code review by your peers. Use the tools to quickly spot what might be the problematic areas in your code base, then ask the team to do a thorough code review of these parts and provide constructive criticisms.
This is also a good opportunity to learn from each other, and to make the team more cohesive.
If you are playing in the Microsoft .NET world, I cannot tell you how much NDepend is a wonderful tool for getting information on your code.
Links:
Clarification added August 8, 2007:
Interesting link about the importance of code review by peers:
http://smartbear.com/codecollab-white-paper.php
Don't forget about the people!
Your ability to change the product, fix bugs or turn it around completely largely lies in hands of developers. If they are professionals and they are excited, you actually have chances of doing so. Better yet, get them excited about the quality as well - and they will find new ways to keep the quality up.
Metrics mentioned above do make sense. People is just another dimension.
Go for function points rather than KLOC. Function points are difficult to measure, need an expert, but then KLOC method is not accurate,
because the measurements depend on the programmer's programming
style and the programming language that you are coding in.
Here are some metrics that you can look at :
Unit testing defect density : No. of unit testing defects/No. of function
points
(Same can be applied to unit integration testing)
2. No. of requirements defects found
3. No. of functional spec. defects found.
4. No. of defects found in code review/No. of function points
in that function.
Note that cost of fixing a bug increases as the product development
cycle proceeds to later stages. So, it's important to keep track
of WHEN a bug is found. The sooner the bug found, the less the
cost, energy and time to fix it. This is an important consideration.
There are several metrics used for gauging software quality. First, I’ll echo the comments of others by saying you should clearly identify what you want to learn from the metrics. Second, keep it simple; don’t collect a huge volume of metrics at first. Begin with some straightforward measurements, and then add others as you gain knowledge about the system. This said, some basic software quality metrics follow here.
• Number of defects – Break this down into no. of issues found during development, found during QA, and found following release, etc. Finding and fixing a defect is always better sooner than later.
• Rate of defect discovery – I.e. defects found in the past day/week/month. Does QA continue to find new defects? Is QA still adding new test cases (preferably automated test cases)?
• Test coverage – Test cases (preferably automated) per module/class/KLOC/etc.
• Defect density – Total defects per module/class/KLOC/etc.
Hope this helps.
Walt S
Sr. Systems Engineer and Designer
Best Answers in: Software Development (6), Personnel Policies (2), Accounting (1), Incorporation (1), E-Commerce (1), Computers and Software (1), Web Development (1), Using LinkedIn (1)
Whatever metrics you use, consider not making them too public internally, or if you do, do not provide incentives for them -- it becomes too tempting to game the system rather than doing the right thing.
To measure overall code quality, we have an independent QA shop measure how many defects have been found, how many are outstanding, and how many have been fixed. By watching these numbers, it's possible to see when one reaches the knee of the curve. That's the point when more things are getting addressed than broken.
Well here is my take on this topic, I don't see why a developer would want to know if his code is "reasonably good". From a quality perspective I would say: Start with "Testable Requirements", if the requirement can be tested, then this is a pass/fail for the developer to meet.
If the inputs and outputs are clearly defined, then the full range of test cases can be run against the code.
Now this is fine if you are implementing new functionality, but obviously does not help if you are performing maintenance work, unless you can retrofit testable requirements into the existing code base.
Code coverage and other quality metrics really should not be the developers responsibility, the QA dept should have an approach that will determine what needs to be tested, but in smaller shops this does not happen, as the developer becomes the tester and the QA manager etc.
Overall my response has to be: Hire a QA team to improve the overall quality, they will put the necessary mechanism in place to ensure the developer delivers quality code.
Eileen B
IT Professional, Information Security Quality Assurance Operations & Administration / President, CMU SEI LI SPIN
Best Answers in: Using LinkedIn (53), Staffing and Recruiting (12), Career Management (10), Quality Management and Standards (7), Computers and Software (7), Software Development (7), Ethics (6), Change Management (5), Enterprise Software (5), Web Development (5), Freelancing and Contracting (4), Job Search (4), Accounting (4), Government Policy (4), Internet Marketing (4), Organizational Development (4), Project Management (4), Professional Networking (4), Education and Schools (3), Business Development (3), Supply Chain Management (3), Blogging (3), E-Commerce (3), Databases (3), Travel Tools (2), Certification and Licenses (2), Personnel Policies (2), Internationalization and Localization (2), Contracts (2), Employment and Labor Law (2), Advertising (2), Public Relations (2), Business Analytics (2), Corporate Governance (2), Inventory Management (2), Manufacturing (2), Personal Taxes (2), Biotech (2), Computer Networking (2), Commercial Real Estate (1), Facilities Management (1), Regulation and Compliance (1), Mentoring (1), Corporate Taxes (1), Economics (1), Government Contracts (1), Government Services (1), International Law (1), Treaties, Agreements and Organizations (1), Criminal Law (1), Antitrust Law (1), Direct Marketing (1), Guerrilla Marketing (1), Labor Relations (1), Planning (1), Bond Markets (1), Hedge Funds (1), Market Research and Definition (1), Professional Organizations (1), Starting Up (1), Information Security (1), Information Storage (1), Telecommunications (1)
Hi Venkat,
Big topic, thanks! Code quality is directly aligned with fitness for use. Life critical software deserves rigorous testing and metrics more so than gaming software for emample. So, my response is - it depends.
Eileen
A metric which I consider applicable to any paradigm (oop, procedural etc)
and efficient to prove at least that a design and an effort for molularization
took place is called cyclomatic complexity.
It is a concrete measurement and besides the above it definately gives an
indication of how it would be for another programmer/developer/designer/architect to follow/take-over the application,
which is definately a metric of quality in our era.
Links:
Kevin H
Executive Consultant at CGI
Best Answers in: Product Design (1), Computers and Software (1), Telecommunications (1), Software Development (1)
for me complexity is the key - the real killer in any system is complexity. A highly complex system becomes brittle quickly, which leads to further errors down the road. Complexity is something code inspections don't pick out or are adpated for - so find a good tool in the paradigm you're working in that drives to understan the complexity of your code. Once you're there, and you establish boundary conditions for allowable levels of complexity, then you can use other methods, such as code inspections, to reduce the complexity. But the key predictor for poor quality systems is complexity. This is especially true the larger your software system.
I agree with the thoughts that quality is in the eye of the beholder, and it's actually the end user experience that is essential to get right. Combine that thought with metrics related to function points and defect density per function point, and you're on the right track. Measuring defects per requirement, functional spec defects vs. requirements, and code coverage are all good indicators of working or defective systems. But few people have the methodology or patience to run validation against functional specs or do requirements traceability. These are worthwhile acitivities if you want to get predictive in your software development effectiveness.
In the end you have leading indicators and trailing indicators. A test methodology provides trailing indicators that hamper overall producitivity and effectiveness. Leading indicators, such as function point analysis, requirements coverage/traceability, and even code complexity are more meaningful, productive and cost effective metrics to track. But you need to change your overall approach to make them work, and in the end, you need a system large enough to invest in the time, training and tools to make them work.