Blue Prints‎ > ‎

Generic Coverage Report Plugin


A generic Code Coverage Report plugin for Jenkins, that supports various formats: NCover, Bullseye, Cobertura, Clover, EMMA,...

User Scenario

We have used different coverage tools from many sources and for different languages: Cobertura, Clover, NCover, Bullseye Coverage.

Our best practice setup- when it comes to code coverage - is typically like this:
  • Write tons of unit tests for your code
  • Have your Continuous Integration server build and execute the unit tests and let the coverage tool monitor the unit tests
  • Let the Continuous Integration server display a report of the coverage metrics
  • On you Continuous Integration server setup subscriptions on different thresholds for various types of measurements (classes, conditionals, etc)
With a setup like this you get a continuous coverage measurement, and you get a warning when coverage becomes suspiciously low (which basically means that it's time to write more or better unit tests).

One setup that works exactly like this out-of-the-box is when you are coding in Java, using JUnit for your unit tests, Cobertura for your coverage, and Jenkins together with the Cobertura plugin for your CI server.


At several assignments we used Bullseye Coverage with CppUnit in C++. Using a small bullshtml utility we converted the bullseye XML coverage report to Clover, and we could then display it using the Clover plugin for Jenkins. The approach is simple and works well (enough). But from our perspective the way the Cobertura plugin handles things in Jenkins is nicer than the Clover plugin and we're looking at the exact same issue with NCover.

So basically we would like one common DoTheRightThing coverage plugin for Jenkins, which could work equally well with various formats of Coverage XML reports: Cobertura, Clover, NCover, Bullseye.

This approach is already seen with various other plugins for Jenkins that handles reports that come in a variety of formats, you don't to write a new plugin for each format, you just need one plugin that reads them all. This is already done for UnitTests, Compiler warnings, Style violations.

But when it comes to Coverage reports, there are still as many different Jenkins plugins as there are tools and some are neater than others.

Design Proposal

Taking offset in the Cobertura plugin, simply because that's the one we like the best, we would expand then plugin with an option to tell which format the coverage report is in (Bullseye, Emma, NCover, Clover, Cobertura...), and then write new parsers for these formats one-at-a-time. The parsers should translate the "native" format to an common internal generic coverage report structure, which should be the one the reporting is actually based on.

To get a quick start this structure could simply be the DOM of the Cobertura XML format, then the plugin is almost already done. It's possible that in the future there will be so large differences between the individual formats, that some may turn out to be untranslateable (hmmm - new word ;-) to Cobertura, and that will then be the cue to come up with a generic format.

Estimate

The main challenge in this assignment is on data analysis, coming up with a genric XML format for coverage metrics and then becoming familiar with the Jenkins CI object model and community.

We estimate that students that are unfamiliar with Coverage tools and Jenkins CI would spend 2-400 hrs on an assignment like this.

A professional implementation done by Praqma is estimated to 50 hrs.

References


Contact

Call Lars on +45 30 87 25 30 or mail to him on lak@praqma.net if you are interested in this Blue Print.