CI Tools and Best Practices in the Cloud

Continuous Integration

Subscribe to Continuous Integration: eMailAlertsEmail Alerts newslettersWeekly Newsletters
Get Continuous Integration: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn

Continuous Integration Authors: Stackify Blog, Aruna Ravichandran, Plutora Blog, Dalibor Siroky, PagerDuty Blog

Related Topics: Java EE Journal, Apache Web Server Journal, Continuous Integration

J2EE Journal: Article

Taking Continuous Integration to the Next Level

Taking Continuous Integration to the Next Level

Sometimes, the easiest and most rewarding part of development is the actual coding. Managers and developers often dismiss the building, deploying, testing, and metrics-gathering aspects of the software life cycle.

Continuous integration is the concept of automating the build cycle so that code is built and tested many times during the day. This allows developers to integrate their code into the build daily or hourly, thus lowering the chances of integration problems.

In this article, I introduce an Ant-based build and integration system called BigBrother that leverages XP (Extreme Programming) principles. BigBrother combines several open-source tools into a "continuous integration" system. I also offer a real-world scenario in which BigBrother was used to successfully manage a continuous build process.

The concept of continuous integration has been around for a long time and one of its biggest proponents is Extreme Programming. XP is a disciplined approach to software development that stresses communication, simplicity, feedback, and iterative design. One of the main coding practices of XP is to integrate often. The benefits of integrating code are numerous:

  • Developers are guaranteed to be working with the latest version of software.
  • Compatibility problems are more likely to be detected and avoided early.
  • Integration cycles that used to take days or weeks are reduced to minutes.
  • Communication between developers increases.
  • Integration bugs are found much sooner than later.

    Automating the build-and-test cycle definitely brings many benefits to any software development life cycle, but this only scratches the surface of what can be done by combining some of the open-source tools available today.

    Paradise Found
    With the advent of Java and open-source projects, creating and automating a continuously integrated environment has become simple, yet extremely powerful. These open-source projects include components for compiling and deploying code, executing automated unit tests, computing and displaying unit test code coverage, calculating various code metrics, as well as many other tools for making a project successful. All of these individual components can then be wrapped and automated with the help of yet other open-source projects.

    Using each of these tools by themselves is helpful in the development of software, but the real power is when you combine them into a single continuous integration environment that compiles, tests, and reports metrics on a code base every time a file is checked in. Imagine a development environment with the following features:

  • Project Web site with up-to-the-minute build information and code metrics.
  • Development team is notified of broken builds immediately and accountability is placed on the last person(s) to check in code.
  • Unit tests for the entire project are run automatically after every code change. Results of these tests are reported on the project Web site.
  • Code metrics are run after every code change, showing such things as Javadoc coverage, cyclomatic complexity of each method, and lines of code.
  • Code coverage report displaying how much of the code was tested by the automated unit tests.
  • Provide team leads with the ability to monitor coding and testing efforts on an up-to-the-minute basis.
  • Both Web-based container and EJB container code can be fully and automatically unit tested.
  • Reporting on code metrics can be done on both the client and server tiers.
  • XP-style feedback - every build is a snapshot of how much of the project has been completed and how much has been unit tested.

    This might seem like a lot to ask for on every Java project you undertake, but using Apache's Ant as the glue makes it easy to tie various open-source projects together such as CruiseControl, Clover, JavaNCSS, JUnit, and StrutsTestCase for JUnit. I've combined all these components together into a system affectionately called BigBrother. BigBrother is simply an Ant build process that pulls together all the open-source tools listed above into a single integrated environment. A development team using this environment is forced to monitor all the code they check in and to make sure it compiles, doesn't break any unit tests, and is well testedŠbecause BigBrother is watching.

    The Tools in the Toolbox
    Before I explain how BigBrother works, I need to explain the various components used to build the continuous integration environment. Apache's Ant is the cornerstone of this architecture. Ant is a Java-based framework that has revolutionized the build process. Ant has many built-in tasks that can be used for everything from compiling and packaging code, to starting and stopping application servers. Ant can also be easily extended using Java classes to provide customized functionality. Ant is the perfect tool for tying many disparate components together because of its flexible, task-based architecture.

    CruiseControl, open sourced by ThoughtWorks, is the second largest piece in the BigBrother puzzle. This pluggable open-source framework allows for a continuous build process and includes a Web interface to view the details of the current and previous builds. It has built-in version control polling for code changes and e-mail notifications for builds that are not successful.

    Clover is a code coverage tool for Java and the only commercial software in the BigBrother architecture. While Clover is not open source, it's very inexpensive given the benefits derived from its use. Code coverage is used to determine which sections of code are not being executed by the existing set of unit tests. Using a code coverage tool exposes exactly how well developers are testing their code down to the method, statement, and branch level. It also helps determine a quantitative measure of code coverage. This measure has been theorized as an indirect correlation to the quality of code delivered.

    JUnit is a Java framework for creating repeatable automated unit test cases. Some features include test fixtures for sharing common data, test suites for organizing tests, and methods for testing expected results. JUnit is great for testing most Java classes, but there are times when you'll need something more robust. There are many extensions for JUnit, ranging from the ability to unit test Web-based code to the ability to unit test J2ME code.

    JavaNCSS is a source measurement suite for Java. It provides helpful metrics to gauge how well code is being programmed and commented. Some of the metrics are cyclomatic complexity per method (program complexity number); noncommenting source statements; number of formal Javadoc comments per class and method; and the number of packages, classes, and functions.

    Combining all these components into one build environment allows any development shop, large or small, to have a world-class enterprise build solution.

    BigBrother 24/7
    Now that all of the different tools have been defined, let's explore how they fit into the overall picture.

    Figure 1 depicts the components that make up BigBrother. The top rectangle signifies the BigBrother Ant build file. The Ant file can be seen as both the wrapper and the driver for the continuous integration cycle. This build file has tasks that control the different components that have been plugged in. The BigBrother build environment can control multiple subprojects at once. The interface design pattern is implemented to execute a subproject's specific Ant target.

    The bottom rectangle symbolizes the subproject's build file. BigBrother can concentrate on monitoring the automated build cycle while leaving all the nongeneric work, like compiling, deploying, starting, and stopping the different application servers, and unit testing code, to the subproject.

    There exists a contract between BigBrother and each of its subprojects. Each subproject implements an Ant target that BigBrother has been told to call ahead of time. This specific target will perform all the necessary work needed to compile, deploy, and test the subproject. BigBrother is blind to the functionality in its subprojects, caring only about executing the correct commands. Using this design, you need to write only one "master" BigBrother build file, as it's reusable across all the subprojects.

    As shown in Figure 2, there's a rigid and repeated structure to all BigBrother software builds. After configuring CruiseControl to poll every n seconds and starting the CruiseControl Java Virtual Machine (JVM) using Ant, the project build cycle is literally on "cruise control." If modifications have been found in the code base, a fresh copy of the entire branch is retrieved out of the version control system; the code is compiled, deployed, and then unit tested on the appropriate application servers. Code coverage and metrics reports are then generated from the code branch and the results are published to a Web site accessible by developers and project managers. If at any time in the process the build fails, all the developers on the project are sent an e-mail containing the location of the Web site for details about why it failed. There's also an option to spam developers if the build stays broken. Needless to say, this is not a popular option.

    Figure 3 shows that the logical architecture is made up of a dedicated machine to perform the continuous integration tasks. This machine has everything it needs to build, deploy, and test an entire J2EE environment. It consists of a WebLogic application server used to deploy the server-side code, Tomcat Web container used to deploy the client-side code, and another Tomcat Web container used to publish the results after the build process is complete. Multiple instances of BigBrother can run on the same dedicated machine, thereby continuously integrating multiple projects. It also takes advantage of having only one instance of WebLogic running multiple domains and one instance of Tomcat running different ports for its unit testing.

    The Rules
    Certain rules must be put into place as part of the software development life cycle when using BigBrother. The first rule of BigBrother is do not check in code that does not compile. The second rule is do not check in code that does not compile. The third rule is that new code checked into version control cannot break any of the existing unit tests. The fourth rule is that any new code checked in must have working unit tests. The last rule is that any new code checked in must have at least a 90% code coverage rating after running its unit test(s). The rule "traffic cop" in this case is BigBrother. If any of these rules are broken, BigBrother will be the first to know and will "punish" accordingly (usually a friendly e-mail reminder of the offense). Adhering to these rules will greatly improve the chances of a zero-length integration life cycle and a much shorter quality assurance (QA) life cycle.

    The Real World
    If, after reading all of the above, you're still not a believer in having a continuous integration environment, here's a real-world case study to ponder. Before BigBrother, my company's software builds were in a state of total disarray. If you were lucky enough to check out a branch of code that compiled, you still weren't guaranteed that the code actually worked since there were no unit tests. Developers had to wait hours for code to be fixed so that the build would compile.

    Murphy's Law states that a developer would check code in on Friday and break the build without anyone knowing it. Many weekends were lost due to bad builds. Nobody knew how many lines of code there were in a certain project or the complexity of the methods and classes. Nobody knew how much of the code had Javadoc in it. A striking figure was that only 65% of the code moved to QA actually worked. Development life cycles were greatly extended because of code integration issues.

    After implementing BigBrother across the board on all of my company's development projects, a dramatic shift occurred. The builds are seldom broken; and if a build is broken, it's fixed in a matter of minutes. Developers now take pride in their work and a competitive spirit has arisen to see who can have the most code coverage out of their unit tests. Every functional piece of code checked in now has a corresponding unit test. The result that managers love is now, on average, 97% of the code moved to QA is bug-free.

    Integrate Often
    A continuous integration environment literally turns software code into an open book. (see Figure 4). Detailed information is provided on who has been doing their unit testing, who has been checking in code that compiles, and who has been creating simple yet functional code. It also allows communication channels to open up more quickly between developers, and generates more of a team-oriented approach rather than the lone developer. Generated reports do not lie; they paint a picture of the current status of a project. The investment time up front to automate the build and test process is so slight and the rewards so beneficial that no development shop should be without a continuous build environment.


  • Continuous Integration:
  • Extreme Programming:
  • Cruise Control:
  • Apache Ant:
  • Clover:
  • JUnit:
  • StrutsTestCase for JUnit:
  • JavaNCSS:
  • Struts:
  • More Stories By Franz Garsombke

    Franz Garsombke has been developing and architecting enterprise software solutions in Colorado for the last eleven years and is currently employed at Rally Software. Franz is a huge proponent of open source frameworks and is passionate about developing and delivering simple, quality, pragmatic applications. He is proud to be the co-founder of a Java Bean mapping framework ( and can be reached at or on his mountain bike.

    Comments (7) View Comments

    Share your thoughts on this story.

    Add your comment
    You must be signed in to add a comment. Sign-in | Register

    In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.

    Most Recent Comments
    Andrew Boyd 06/14/03 01:14:00 PM EDT

    First off just let me say thanks for the demo code and good article. I'm psyched to try it on my next project that should be starting in about a week!

    Do you have any procedural guide lines that us use that might be some help?

    Thanks again

    Doug Serres 05/02/03 11:32:00 AM EDT

    for making the demo available.

    -- Doug

    Glenn Snyder 04/23/03 03:19:00 PM EDT
    Franz Garsombke 04/22/03 03:43:00 PM EDT

    I have tried to pull all the pieces together to provide a loose demo of the capabilities. Email with any questions. Haven't had much time to document but I think most of it is self-explanatory.



    Michael Billings 04/21/03 08:30:00 AM EDT

    I would love to expose my organization to continuous integration and getting a kick start to implementing the process would be a great help.

    Please provide the ANT script to get started.

    denis krizanovic 04/13/03 07:51:00 PM EDT

    subject header says it all really.

    David Hibbs 04/04/03 08:49:00 AM EST

    Hey, that's not nice... publishing a teaser like this and then not providing a jump start on the actual hard part--the scripts! Certainly there's not so much proprietary information in an ANT script that you can't save us all a little bit of time in developing our own versions of BigBrother...!