An Exercise in Managing the Measurements

I’m a firm believer in the phrase:

“What gets measured gets managed.”

When it comes to unit testing code coverage is something we can use to measure how much of our code is under test. By no means does this mean that the code that is under test is actually covered by valid tests that mean anything to the product at hand – it just means at some point during the test execution a path of test code passes through the given line of code. However, it is still important to know what assemblies and code paths are not covered during the testing process so teams can analyze areas that might need to be covered a bit more.

When I set up a CC.NET server I always try to convince the client that NCover (a Code Coverage Tool) is a requirement just as much as the NUnit is. As long as I can set it up rather quickly the client is happy to oblige because from a management perspective they can now see via the CC.NET Dashboard what code is covered and what code is not covered.

That is  … if you understand how NCover works.

 

The Problem

NCover will create a profile for any assembly that is loaded into memory at runtime of the test suite. Lets think about that….

“… any assembly loaded into memory at runtime …  ”

There’s a problem here. What if one of my assemblies have NO tests whatsoever? The end result is that the assembly _does_not_ show up on the CC.NET report. But… to NCover’s defense this is expected. How can NCover know _exactly_ what assemblies to profile? It can’t. The best it can do is to load the one’s it see’s during testing and profile those.

Unfortunately if your project has 10 assemblies and only 2 of them are under test you coverage percentage might be 75% – because those two assemblies might have a decent test suite. Since the other 8 assemblies were not loaded during the execution process they do not get profiled. Hmmm. problem…

So how do we get NCover to profile _all_ the assemblies that _we_ want?

 

Hacking NCover

I’ll be the first to admit – this is a hack. But… it works.

In order to get NCover to profile the assemblies I want covered I created a class file called “NCoverHelper.cs” with one test inside of it, shown below.

[Test] 
public void This_will_load_all_Foo_assemblies_for_ncover_auto_discovery() 
{ 
    // HACK: This is a SUPER NCover hack. Read below for more info. 
    // 
    // This test is purely here to force NCover to LOAD the assembly into memory so that 
    // we can profile the ENTIRE stack of DLL's that we want. If we do not do this, NCover will 
    // only load those assemblies that are "touched" by tests. Therefore, if you have an 
    // assembly that DOES NOT have a test suite it would NOT show up. This test FORCES it 
    // to show up in the coverage report. 
    // 
    var files = Directory.GetFiles(AppDomain.CurrentDomain.BaseDirectory, "Foo.*",SearchOption.AllDirectories); 
    var dlls = files.Where(x => Path.GetExtension(x) == ".dll" && x.Contains(@"\build\bin\")); 
    // Uncomment line below to see all files the app is loading. 
    //File.WriteAllText(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "dlls.txt"), String.Join(Environment.NewLine, dlls.ToArray())); 
    dlls.ToList().ForEach(x => Assembly.LoadFile(x)); 
}

 

How it works:

The code looks in the currently executing directory, then searches for all “Foo*” files. This would return Foo.DataAccess.dll, Foo.Business.dll, Foo.Repository.dll, you get the point. The code searches all folders under the currently executing root folder. In the end, it could possibly find 10 of the same Foo.DataAccess.dll if it was referenced throughout the project.

After the dll’s are found, I filter them based upon a location. In the example above, I filter them based upon the path “\build\bin\” which is where my build is dropped.

After I have all the Foo dll’s I need, I load them via Assembly.Load(…). Yes, its kind of slow. BUT… it works.

After loading the assemblies, NCover will then profile them and find that some of them are never tested. Therefore, the report that is returned from NCover is a 0.0% coverage for any assembly that has no tests.

Please note – I’m sure there is a better LINQ query to get the info I need, but this one works for me.

 

Conclusion

A hack? Yes. Does it work? Yes.

In the end we are able to load our assemblies via runtime of the tests which enables NCover to profile them. The end result is that we can now see our true code coverage for our entire project, not just the projects which were “touched” by a unit test.