CI Tools and Best Practices in the Cloud

Continuous Integration

Subscribe to Continuous Integration: eMailAlertsEmail Alerts newslettersWeekly Newsletters
Get Continuous Integration: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn

Continuous Integration Authors: Stackify Blog, Aruna Ravichandran, Plutora Blog, Dalibor Siroky, PagerDuty Blog

Related Topics: Continuous Integration, Static Analysis

Continuous Integration: Article

How Good Is Good Enough?

Code quality and the point of diminishing returns

Intellectually everyone understands that improving code quality is a good thing. After all, we know bad quality when we see it. (Anyone old enough can cast his or her mind back to the late '80s and Microsoft Word for Windows 1.0.) But we also know that there comes a point where there's a diminishing return on our investment in code quality. How much work would you put into tracking down a bug that's only ever been reported once by a user running on OS/2 Warp?

The problem with code quality initiatives is that we really don't know how much effort to put into them. We have never truly answered the question: how much quality is enough?

Why Code Quality Is Important
The Standish Group famously reports on the software industry's inability to deliver successful projects on a regular basis. In 2004, it reported that just 29% of software projects were considered a "success."

Projects fail for a number of reasons, and as far as the Standish survey is concerned, that means either the projects were completely canceled, or that they were "challenged" because they significantly overran time and/or cost.

There are many reasons that projects are canceled or overrun, and almost certainly the most common is that the software just didn't do what it was supposed to - it didn't conform to requirements. That's a hard problem to fix - and it's an even harder one to measure.

Code quality, on the other hand, which is probably the second largest contributor to project failure, is much easier to measure, and therefore easier to manage. Yet, many development teams waste time on rework cycles fixing bugs that crop up in QA or production that could have been caught and fixed upstream, with relatively little effort.

A big part of the cause of that problem is that during the coding phase, oftentimes the only measurement we have available to track progress is whether we hit the deadline or not. Even if we monitor defect rates (it may shock you to learn that not all development teams do), we just don't have any meaningful metrics by which to manage the process until it's too late.

Agile methods help. Test-driven development, where unit tests are written with, or even before the code, and continuous integration and test cycles flush out many bugs in development before they get to QA.

But fundamentally, managing only to deadlines causes an accumulation of "technical debt" - a term coined by Ward Cunningham to describe the effect of all those decisions to use a quick and dirty approach to solving a problem, with the idea that you'd come back and fix it later. Just like financial debt, technical debt isn't necessarily a bad thing, especially when an important deadline is looming. But like financial debt, if you have too much of it, it can get out of control and becomes very hard to pay back.

How Much Quality Is "Enough"?
Quality is a tradeoff between cost and risk. If you ask most (inter-nal or external) customers what level of quality they'd like to see in their software applications, they'll likely tell you "perfect" - i.e., zero defects. Of course, that's something of a trick question, because you didn't tell them how much "perfect" would cost.

After a 1996 Fast Company article on the Lockheed Martin group that builds and maintains software for the space shuttle program, that software is often cited as the most expensive code on the planet, line for line. I'm not sure anyone really knows the cost per line of the space shuttle software (it's been estimated at $1,000 per line), but we do know that (as of 1996) it took roughly 260 developers to maintain 420,000 lines of code, which comes out at about 1,600 lines per person. That's expensive - but the approach seems to work: according to the article, the previous three versions of the software had only a single defect detected per release.

Of course, a typical business application isn't controlling a space shuttle (which, even in 1996 dollars, cost $4 billion a pop, not to mention the lives of a half-dozen astronauts being at stake), and many development organizations struggle with the cost/risk equation. How much quality is enough for your organization? For your application? How can we even start to answer that question?

Ideally, we should be able to come up with a mathematical solution. In probability terms, the expected cost of failure would be expressed as the cost of something bad happening multiplied by the probability that the bad thing will happen. So, as you spend more on increasing the quality of the software, the probability of failure comes down, reducing the expected cost of failure.

This approach becomes problematic in practice, because it's hard to measure the incremental cost of increasing quality and the probability of failure. But, unless we (as an industry) start to measure these things, we'll remain in the Dark Ages.

What Gets Measured Gets Managed!
So, what to measure? A good place to start is the cost of fixing a defect. Once you know the cost of fixing a defect, and you start to measure defect rates, you can assess the return on the investment spent on quality measures (which should be driving defect rates down).

The cost to fix a defect depends on many factors. Conventional wisdom states that the earlier in the software lifecycle a defect is found and fixed, the lower the cost. Of course that's true, although depending on the application, it probably costs less to fix an application in production now than it did even a few years ago. (How many times have you opened up an application only to have it ask if you want to automatically download and install the latest and greatest version?)

There are other quality-related metrics that can help get a handle on how well code is being written. Most development organizations have a set of coding standards. How many organizations measure how well the standards are being applied? Coding standards don't just tell you where to put your curly braces. If applied consistently, they can have a real, measurable impact on code quality and maintainability.

There are a number of static code analyzers available that help development teams implement their coding standards. And static code analysis can go further than just checking adherence to standards - it can find bugs that would otherwise slip through the compiler undetected (consider the old chestnut of incorrect use of the "=" operator in a conditional statement).

Unit test activity is another early indicator of software quality. Although it's not perfect, code coverage (the percentage of executable lines of code exercised by your unit test suite) is a good measure of the effectiveness of your unit tests. You have to keep in mind of course that code coverage isn't the same as path coverage (in other words, 100% code coverage doesn't mean that your code is bug free!). But it's still fair to say that, up to a point at least, the higher your code coverage percentage, the better chance you have of squeezing those bugs out.

This is a great example of a place to look closely at return on investment. You should set a realistic goal for code coverage. A development team could spend many hours trying to get the coverage rate from 98% to 100% - with very little return in terms of the number of defects trapped in development. Many of the better-known open source projects show coverage rates in the 50% to 70% range, and I know for a fact that many commercial projects sit at about half that number.

The important point here, to roll out a well-worn adage, is that you can't manage what you don't measure. If you're not measuring defect rates at different stages of the lifecycle, start now. Take a look at some of the tools available for static analysis and code coverage and, more importantly, start to track those quality metrics over time. Only when you have sufficient data to analyze can you start to make some decisions on how far to go with your code quality initiatives, and finally be in a position to answer the question: how good is good enough?

More Stories By Nigel Cheshire

Nigel Cheshire is CEO of Enerjy Software, a division of Teamstudio Inc. He oversees product strategy and has been driving the company's growth since he founded it in 1996. Prior to founding Teamstudio, Inc., Nigel was co-founder and principal of Ives & Company, a CRM solutions consultancy. He holds a Bachelor of Science degree in computer science from the University of Teesside, England.

Comments (1) View Comments

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.

Most Recent Comments
JohnnyCantCode 12/15/06 08:32:24 AM EST

These snippet's have several issues that your product does not catch or comment on.
(goto overview for developers and then click to enlarge the image)

(1) First access of public variables within a method, ie without context. No parameters for values used within the method body. Use of underscore within a variable name, ambiguous meanings applied to special characters. Finally regardless of "integer division in floating point context" as your tool notes, it isn't clear at all that the number of bounced checks divided by a transaction count(of any nature) would provide a bounced percentage that is meaningful in any context, so basically naming conventions are thrown out the window.

public float getBouncedPercentage()
return bouncedChecks_ / transactionCount_;


(2) One, never ever place the value testing for first in a statement. First it's simply incorrect to test if null is equal to an object, the test is whether the object is equal to null. I see this in young so-called programmers you haven't yet formed proper styles in any language. Second, never bracket a one statement if, this is simply confusing to the reader and requires wasted effort to decipher. For example, if (1 == javaCoderLevel) should be if (javaCoderLevel == 1), the first is just nonsense.

if (null == ci)
return false;

The proper code is(note the false extends slightly beyond the null, although may not be present in this text editor). It's immediately readable and understood without wasting space and brain cycles.

if (ci == null)
return false;

(3) However, this is the worst code snippet. What exactly is being returned? Excessive use of parenthesis to break a simple comparison. Again, the obvious lack of variable context. Avoiding an if statement for expediency will later on cause the poor sap who has to work with this code to re-write it.

return (ci.customerNum_ == customerNum) && (ci.balance_ == balance);

The proper statement (assuming proper logic in the first place and these are all primitives) is the following. Note the intent is immediate, the proper use of white space makes it easily readable and the BUSINESS purpose is immediately understood. I tend to break down multi-level and/or's into tiered stmts so as not to extend beyond the IDE view pane and for readability. In this case for a simple 2 point comparison it makes no difference.

if ( ci.customerNum_ == customerNum && ci.balance_ == balance )
return true;
return false;

I think fools that actually believe that you can legislate quality simply have never created a "successful" product. The example of code on your website is extremely poor coding practices period. I think the IT ranks should be decimated and returned back to real technologists not Java or .Net weenies who can only code, albeit code very poorly. Your tool ain't going to change any of that, and projects will continue to fail regardless because quality cannot be manufactured, it takes real talent, not reams of documentation and management. Note to be fair to you I'm not attacking you personally, I'm just tired of all these people who simply want to make a lot of money in IT forcing their ideas onto idiot managers who purchase tools and force "best Practices" on us. This has been going on for many years, all the so-called "innovative" ideas and IT "visionaries". Well guess what, the products of today don't manage businesses any better than they did on mainframes 30 years ago and that's what it's all about, not the latest rad fad BS. The only thing they do is take more time to build, more time to re-engineer, more time and money spent on server farms and wasted CPU cycles and more "engineers" that can't code and definitely cannot convert a design document into a working product.