People often talk about the percentage of code coverage they have from unit tests, whether it be actual coverage or the goal they or their project have set. Having a high coverage percentage is often seen as a quality, but code coverage is not really a metric that gives much value in itself. In this post I want to investigate the different types of code coverage that exists and address some of the problems with code coverage.

Types of Code Coverage

There are various forms of code coverage, not just the one we seem to always talk about. A short introduction to some of the existing coverage types:

Statement coverage – How many percent of statements / lines have been covered?

Branch coverage – How many percent of branches (control structures) have been covered? In the case of an if statement, has the expression evaluated to both true and false. There are variations of branch coverage that talk about more detailed decision coverage, like covering all permutations of true and false in an expression that contain stuff like AND and OR.

Path coverage – How many percent of all possible paths have been covered? This may not sound too different from the two above, but it is actually much more complicated. An example could be a function with two simple if statements (A and B) after each other. To obtain full path coverage, you would have to get all four permutations of true and false in the two if statements (A B - !A B - A !B - !A !B) whereas in statement coverage you would only have to exercise the if statements in isolation. Mind you that this is the simple case – if statements with multiple sub expressions, nested ifs and loops are much worse.

The coverage type we usually talk about is actually statement coverage – but as you can see from just looking at branch and path coverage, much is left to be desired. Coverage types like path coverage are also hard to measure, as many loops will have a near-infinite number of paths.

Another thing that is very hard to measure with code coverage is multi-threaded behavior. It is definitely not caught in our statement coverage and if getting full path coverage is hard, consider getting full path coverage with multiple threads.

One Problem – Test Quality

The problem with using code coverage on its own is that code coverage tells you nothing about the quality of the tests that cover the code. Code coverage tells you how much of your code has actually been executed, but it tells you nothing about the asserts that were made in the tests – that is, it says nothing about the correctness of the code.

The perfect example of this is writing a test with zero asserts (state or interaction). This test will produce a certain percentage of coverage, while ensuring nothing except that the code can successfully execute. While such sanity tests can be useful in some tricky cases with exceptions, this is often a worthless test – it has no quality whatsoever, it doesn’t verify any intent of the programmer who wrote the code it is testing.

The thing that is easy to sell about code coverage is that is is reasonably easy to measure. It is easy to set a percentage, a goal and then try to obtain this goal. 100% code coverage is a great goal – it might be unrealistic in most situations, but it is a good thing to strive towards. But if the tests suck or the programmers who write the tests become lazy the code coverage will be nothing but misleading.

The problem with test quality is that measuring it is really really hard – at least programmatically. How can you even begin to measure how much this code matches the intent of the programmer. Usually the number of asserts that make sense for a given test match only a very low number of the values in the system. And even if we could measure test quality using a program, we might as well do away with the tests and just measure programmer intent versus the real program.

The solution, to me, is discipline and good engineering principles. Code coverage can be really valuable to a team that treat their test code like production code, sharing ownership of the code and doing regular inspections / code reviews to ensure that the test quality is high.

The Inverse Property

One of the things I like best about code coverage is it’s quality as an inverse property – that is, a tool that can specifically tell me which parts of my code that I have not tested. This is a clear signal that you need to be more careful when touching this code.

This is also one of the reasons that I actually like to remove tests that either are very low quality or haven’t kept up to date with the intent of the code they are testing. The ideal solution is to rewrite the tests to match the intent of the code / desired quality, but this is not always realistic. To me, such tests are more harmful than no tests at all – they give a false sense of security and will only confuse if anyone look at them. At least if there is no tests my coverage tool can tell me there is a problem in this part of the system.

And while we are at it, it is actually amusing that some people treat test code like it is nowhere near production code and then the same people seem to think that it is blasphemy to delete even a single test. Treat your test code like your normal code, delete / rewrite it if it is obsolete or doesn’t make sense. Maintaining code that doesn’t add value to the system makes no sense.

Conclusion

Code coverage is a useful metric, but often not in isolation. If you use it then be aware of the implications of the way you’re using it, increasing test quality is much more valuable to me than reaching a specific coverage percentage, although it is good to have some sort of goal. Write tests to verify intent, not to increase the coverage percent – use coverage to find the intents not covered.

Now, I've written the first 4 (7 really) parts of my ReSharper Series - I was a bit curious if anyone is actually reading it - and if anyone is getting value from it? Personally I am learning quite a bit more than I already knew about ReSharper. So a few questions:

• So, are the topics too basic? Too advanced? Too long posts?
• Would it be better with screencasts? I feel it can be kind of hard to show the speed that you can obtain with ReSharper using simple screen shots.
• Are there anything in particular you would like to hear about? Hear more about something that has already been mentioned?
• Other suggestions?

I am going away on ski vacation next week and then to Copenhagen for a few days after that (for the MDIP meeting), so there's probably not going to be any more posts until I get back in about 2 weeks.

Hope to get a few useful comments (or mails) on this post.

Welcome to the 4th part of my ReSharper Series. Today we are going to look at one of my new favorite ways to use ReSharper - to move code around. We haven't dipped into refactoring really yet, and today isn't really going to be that much about it - this is moving code on a lower syntactic level - shortcuts to save that copy / paste. In addition, we are going to have a little bit of code navigation that plays well with this feature.

Furthermore, since many people use the VS binding mode in ReSharper, I am going to try and supply both binding sets in the future. I might even go back and change the previous posts at some point - today all the bindings I will show work for both binding modes though.

Navigating Methods

The first small feature is navigation between members. Basically what it gives you is a way to quickly navigate between methods in your class. If you are in a method like so:

Pressing Alt+Arrow Down will navigate to the next method, while using Alt+Arrow Up will send you to the method signature, like so:

When at a method signature already, you can jump up and down between method signatures in the same way.

This comes in handy when moving methods, since you need to be at the method signature to do so.

Moving Code

The shortcuts for moving code involve a lot of keys, but they are pretty easy to remember. You can move code over 2 axis, up/down and left/right. This is controlled using the arrow keys. To enable the move functionality, you need to hold down Alt, Ctrl and Shift.

Moving Methods

Let us look at the first example. With our cursor placed on the method signature of the Withdraw method, we hold down Alt, Ctrl and Shift to enable movement. This will make the block of code we are moving turn a light cyan (with my color scheme at least):

Hitting the move up shortcut (Alt+Ctrl+Shift+Arrow Up) sends our method above the Deposit method like so:

We can also move methods down using Alt+Ctrl+Shift+Arrow Down. Quite a bit easier than copy pasting it - and useful for reordering methods quickly if combined with the method navigation above.

Moving Arguments

Now we can move quite a bit more than methods. Say we have a method call where we wanted to move the arguments. We can do so using Alt+Ctrl+Shift+Arrow Left and Alt+Ctrl+Shift+Arrow Right. Again, as we hold down Alt, Ctrl and Shift, the block we are about to move is highlighted:

And sending it left is easy:

This also works for actual method signatures, just be aware that this doesn't actually refactor your method and change all the call sites for the method (although ReSharper can do this in it's refactor menu - look for the Change Signature refactoring).

Moving Statements

When we are dealing with statements inside a method, it can some times be useful to reorder lines of code or move code in and out of control structures, so when we hold down Ctrl, Alt and Shift here it actually suggests that we can use all 4 directions.

Moving up and down lets us maintain our level of scope, in this case, pressing Alt+Ctrl+Shift+Arrow Down, we would send the method call into the else branch:

Repeating this would send it below the negativeText call. When we move left and right, we move in and out of scopes, thus if we press Alt+Ctrl+Shift+Arrow Left, we yank the statement all the way out of the entire if statement:

Again, if we move up and down here, we maintain our level of scope and thus do not re-enter the if statement unless we move the statement right.

Reordering Expressions

The last short example is reordering expressions, this makes it easy to reorder expressions and move parts of them around. Like here:

Moving Balance right would actually swap the two values.

Summary

What I have shown is some of the possibilities for moving code, but as always, play around with it - you can move quite a bit more - like fields and properties in your classes - possibly a lot more.

This is the 3. part of my ReSharper series. Today we are going to have a look at how ReSharper helps you complete your code / show options while programming. I know I didn't know about some of these for a long time, maybe there's something new for you as well?

Standard Symbol Completion

This is the basic intellisense that you are used to and the one that is automatically provided for you whenever you are typing. It contains everything that matches the prefix that you have written in the scope (class, object, etc) that is relevant - just like you are used to from normal Visual Studio intellisense.

The keyboard shortcuts for enabling this type of completion is either Ctrl+Space or Alt+Right Arrow, with Ctrl+Space being the usual one. Now as I mentioned this is enabled when you are typing, but say you have returned to a line of code you were writing and it doesn't pop up on its own, this is the shortcut to be hitting.

Import Completion

Remember in part 1 when I mentioned an easier way to import namespaces than typing out the full name and doing a quick fix (Alt+Enter)? Well this is it. With this auto-completion mode, you can as ReSharper to complete the names of types that are not imported yet, thus not requiring to type the full type name and getting the automatic import for free.

Say I wanted to use a LinkedList in my CheckBalance method shown above, the standard auto-completion isn't very helpful since I have not imported the collections namespace:

This doesn't really help me much. However, if you enable symbol import by pressing Alt+Shift+Space [IntelliJ: Ctrl+Alt+Space], this is what you get:

ReSharper will complete the common prefix and suggest type names in other namespaces. When using generic types I usually hit < at this point and type out the generic constraint. While writing this post I noticed that it was behaving oddly when completing types with multiple generic types, but it might just be me. Anyway it is a really helpful in addition to the basic intellisense that we usually use.

Smart Completion

To be honest I hadn't looked at this completion mode until recently - but ReSharper actually also has a completion mode that tried to be clever about the type context of the expression you are currently writing. Since ReSharper knows the scope of the code you're writing and it has an idea about the current expression, what smart completion does it to filter the completion list to only show symbols that have the relevant type.

I find that I mostly use smart completion when I am writing method calls, especially if I am either messing in some code that I don't know that well (since the completion list is often much smaller) or if I know that there is only one possible completion. In the latter case, ReSharper will do the full complete when enabling the completion and proceed to the next parameter instantly.

If I have a method that takes a boolean for example, enabling smart completion will only suggest symbols that actually have the relevant type, like so:

The shortcut to use smart completion is Ctrl+Alt+Space [IntelliJ: Ctrl+Shift+Space]. I find that it is somewhat situational for me, but it is useful - I use the other two more though.

Hint: Out of personal opinion, I suggest that people using Visual Studio bindings remap these keys so that Import Completion become Alt+Ctrl+Space like in IntelliJ - I use it much more often and it's so much easier to hit this combination. Smart Completion could then be Alt+Shift+Space.

Camel Humps

Søren mentioned camel humps when I was talking about basic navigation in the last post - and this also holds true for all the auto-completions shown above. If you type upper case letters while completing, it will match camel humps in the list. For example, if I wanted to use a LinkedResourceCollection as seen in the symbol completion example, here's what I could do:

I find that this is mostly useful when you have types with long names that you use often.

Case Sensitivity

Another thing you can do if you want more control over what is filtered in the completion lists is to enable case sensitive prefix matching in the ReSharper settings under IntelliSense -> Completion Behavior.

This will force ReSharper to match the case as you are typing and it will only present symbols that actually match the case - this could potentially be useful if you don't prefix your instance fields (personally I prefer prefixing with underscore).

Without case sensitivity:

With case sensitivity:

I haven't played around with case sensitivity enabled so much, generally I just accept having a longer result list and not being forced to case my identifiers.

Summary

In this post I gave an introduction to the auto-completion modes of ReSharper, try them out and see what works for you, I especially recommend the import symbol completion if you are not already using it.

What a great start on 2009, I only hope the rest of the year will be as eventful as this month. The highlights were:

• Blogger of the month on danish MSDN Flash newsletter [danish].
• Awarded the MDIP (Microsoft Designated Information Provider) title [danish] by Daniel.
• Was invited to join the core group of Aarhus .NET User Group [danish].

I am truly humbled by all these things, it's just great to know that someone out there thinks that my ramblings and crazy technical obsession is valuable.

Anyway, back to the technical posts.