RCOV C0 line coverage more generous than EMMA’s C1 line coverage

11 07 2007

Coverage tests in Ruby (with rcov) are less strict than in Java (with EMMA), so watch out – 100% coverage is easy to attain but not as meaningful.

For Java code coverage, I like EMMA. Clover looks really nice and all, but seriously, $250 a seat? Yeesh. Talk about poor product pricing – I will never buy that product, nor seriously consider buying it, because of the outrageous price. I mean, a single license of IntelliJ is $1 cheaper. That’s like charging $300 for a better Ant. C’mon.

Anyway, EMMA is very conservative about coverage estimates. Consider this single line of Java code:

int x = 1; if (x > 1){throw new RuntimeException();}

When you run it, the assignment will execute, as will the comparison, but the block containing throw will never be reached. So, the line is not fully covered. EMMA looks at bytecodes and maps those back to line numbers, and will mark this line as partially covered.

It’s maddening sometimes, but it’s correct; it’s your problem to figure out how to force all those darned IOExceptions that you know can’t ever happen, or to give up and let some apocalyptic error handling code not be covered. I used to shoot for 85-90% coverage, and just live with some uncovered wacky code when I couldn’t find a reasonable way to trigger hideous errors.

(There’s a little voice in my head that says, “Come on, you slacker, you should make it possible to dynamically break the database configuration and remount the root filesystem as read-only during the test suite so that the error handlers can all be 100% covered!” But that little voice never ships anything, he just sits around and writes more and more paranoid code — no Ariane 5 disaster on my watch! — and gets asymptotically closer to 100% coverage.)

Okay, now the Ruby version:

x = 1; raise RuntimeError if x > 1

Same story, prettier code (cuz it’s Ruby; duh, of course it’s prettier): assignment gets executed, condition is executed, no exception is raised. Rcov says it’s 100% covered.

The difference is C0 coverage (Rcov) vs. C1 coverage (EMMA). EMMA only counts a line as 100% covered if all of the bytecodes compiled from it were executed. Rcov counts a line as 100% covered if execution visited that line at all.

(EMMA uses “basic blocks” instead of bytecodes, so in obscure failure cases the reported line coverage figure is even lower, but that doesn’t affect the conditions for what 100% line coverage requires.)

I’m not sure if it’s feasible to get C1 coverage on the regular Ruby VM, but it would be nice to have. In the meantime, just be aware that 100% C0 coverage is easy to attain with rcov, but doesn’t mean you’re done testing everything.

Of course, 100% C1 coverage doesn’t mean you’re done testing everything either, but it’s closer to meaning that than 100% C0 coverage is.

In fact, it’s good to over-test code that’s already covered once. On any given project in the real world, there is code that will succeed with some values and fail with others. (print x/y, for example.) Just testing with one set of values isn’t enough. I like to create an array of inputs and expected outputs and use Array.each to iterate over them and pass them into a block that calls the code under test and then compares the actual outputs to the expected outputs with assert_equal. It means that I’m following DRY (rather than cutting and pasting several lines of test code and then editing a teeny bit of each pasted chunk… eww) but still testing many different ways.

But, even with a C0 coverage tool, at least you know what hasn’t been touched at all, and you can use your big smart programmer brain to think of a few ways to torture that code. Just remember that you probably should be aiming for an unmeasurable way-over-100% imaginary coverage target, beating the heck out of your fanciest code with many, many different inputs and then moving on to the next chunk of uncovered code only when the last chunk is thoroughly proven to not suck.

Good testing!


Actions

Informations

3 responses to “RCOV C0 line coverage more generous than EMMA’s C1 line coverage”

31 07 2007
John (22:30:15) :

My last comments formatting didn’t work so well. Maybe this will be better.

===

Hi Jamie,

Perhaps you know about this already, but if not…

You wrote:

====
There’s a little voice in my head that says, “Come on, you slacker, you should make it possible to dynamically break the database configuration and remount the root filesystem as read-only during the test suite so that the error handlers can all be 100% covered!”
====

One thing that I’ve done to increase test coverage is to reopen classes, and redefine methods that allow me to fail certain operations (for instance saving). Here’s an example:

class PictureStream
cattr_accessor :fail_saves
@@fail_saves = false

alias_method :unaliased_save, :save
def save(param = nil)
return unaliased_save unless @@fail_saves
false
end
end

This code goes inside one of my tests (test/units/whatever_test.rb), and when I have a test for failing to save, the code does this:

PictureStream.fail_saves = true
… do the operation …
… check that it failed …

When I run rcov on these tests, it tells me that the test did indeed cover the case where a save failed, allowing me at least some sense that things are working right.

Perhaps you’d seen this before, and I know it won’t help to produce a read only root filesystem, but I found it really useful to push up the amount of code that was covered by my tests.

4 08 2007
Jim Deville (19:21:40) :

I just wanted to point out that even the creator of rcov points this out. The page you link to goes on to explain C0, C1, and C2 coverage. I’m wondering why you didn’t point that out?

4 08 2007
Jamie Flournoy (22:20:42) :

>I’m wondering why you didn’t point that out?
I linked right to the very thing you’re saying I failed to point out.

Here’s what you’re apparently not grasping: the EMMA web site doesn’t talk about rcov, and the rcov web site doesn’t talk about EMMA. Likewise with Clover. The web sites for EMMA and Clover don’t talk about their functionality in terms of C0 vs. C1 vs. C2, but rcov does.

This post *compares* EMMA and rcov, and uses the *same terminology* (C0/C1) on both, with an identical code example for 2 different programming languages. I also talk about reasonable expectations for coverage % based on personal experience.

Leave a comment

You can use these tags : <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>