After initial excitement about xUnit and how cool your tests become with AutoDataAttribute. My itch to convert all my tests into xUnit have died down. And I’m not excited as much. XUnit is certainly not the silver bullet.

I do agree with certain things that are in xUnit, like not providing [SetUp] and [TearDown], rather have constructor and destructor, but can’t agree with other things. And the authors of xUnit have not been listening to their community which is unfortunate. I’ll list the annoyances I discovered in the order I faced them.

1. Unable to provide a message on failed assertion

In NUnit you could say Assert.IsFalse(true, "Yep, true is not false"). In every assertion you could’ve added a piece of text to the failed assertion. I have used that in the past. Not in many tests, but in few, enough to notice when I decided to convert these tests to xUnit. In xUnit you can’t do that. Apart from Assert.True() and Assert.False() where you can provide a message.

This feature have been removed by authors and it is not coming back. The reason for that is “If you need messages because you can’t understand the test code otherwise, then that’s the problem (and not the lack of a message parameter)“(comments from Brad Wilson). I do agree with the part of smelly test. But this is not the only reason to use the messages – there are cases when this message represents the error.

See my test that uses reflection to go through all EF models to verify a presence of empty constructor. In that test I build a list of classes that fail the condition and in the end if this list is not empty, I throw assertion exception with string of comma-separated types that fail condition. This is not your conventional unit test, but still a valid test. With NUnit the assertion looks like this:

var finalMessage = String.Join(", ", errors);
Assert.IsEmpty(errors, finalMessage);

As part of the message I get on a test fail, I get a list of classes that fail my test. There is no other way to pass a message to developer about failed test.

If I move this to xUnit, assertion would look like this:

var finalMessage = String.Join(", ", errors);
Assert.False(errors.Any(), finalMessage)

Now this is a code smell. It is not immediately clear what we are asserting. Maybe I’m just picky, but I’m not very happy about these lines. I can predict that Brad Wilson would suggest to write an extension for [InlineData] attribute that provides list of Types to be tested. Yep, this sounds like a reasonable idea. But I could not find any documentation on how to do that.

2. Lack of documentation.

The previous point slowly leads to this one. I could not find any apprehensive document/site that has documented features of xUnit. Documentation for xUnit looks like this at the moment:

2014-02-15 19_58_58-xUnit.net - Unit testing framework for C# and .NET (a successor to NUnit) - Docu

There is a bunch of outdated articles linked from the home page. There is another bunch of blog posts which you need to find first. But there is nothing central anywhere. Unlike NUnit amazing documentation. In this sense NUnit beats many projects, even commercial ones, so can’t really compare.

To find out some specifics of operations I needed to troll through xUnit source code, only to find out that I’m looking on source code v2, where I was using 1.9.x. I was determined enough to get the source and find parts I needed. But less experienced developers would not do this and will struggle.

3. Unable to have “Manual execution only” tests

You can ignore NUnit tests with [Ignore] attribute. I would like to have a reason as a mandatory parameter in this attribute, but this is secondary. Ignored tests will be skipped in test runners, but you can specifically run this one ignored test manually. This is useful when you write exploratory tests – where you just trying things out and they are not really a test, but more like a piece of code in your system which you can execute separately from the whole thing. Or if these tests are integration tests interacting with external API, where you need to manually undo effects of the executed test. And I’m not the only one who uses this practice.

In xUnit you ignore tests like this: [Fact(Skip="run only manually")]. Only you can’t run them at all! Not even manually. And people want that! Jimmy Bogard restores test database with a manual execution of ignored test.. He came up with idea where skipped tests do run only when debugger is attached. Not a bad idea, also other people done the same.

Running these tests in debugger kinda work, but looks like a cludge to me. Why not allow for manual execution and do not multiply the hacks?

4. Unable to filter out tests by categories

NUnit has [Category()] attribute that marks the test(s) with some category. Usually these categories mark tests as “fast”, “slow”, “database”, “integration”, “smoke”, etc. So when you run these tests you can only include (or exclude) tests suitable for the environment. xUnit has Trait attribute which is a pair of strings. To create a category with Trait you’ll have to do this: [Trait("Category", "database")]. The key-value structure gives a bit more flexibility with categories. But I can’t come up with a scenario where I’d use other than “Category” for the trait key. Also, code examples shipped with xUnit 1.9.2 do have [Category("")] attribute which is inherited from Trait and placing the key to be "Category". But in xUnit v2 (which is in alpha just now) Trait attribute is sealed, so you can’t inherit it any more.

We do run our tests in TeamCity and way to execute xUnit is through MSBuild script. The problem is here: I can’t filter out tests by their Trait attribute when executing through MSBuild. And this is not an oversight. This is intentional. The idea behind this is “…place different test types into different assemblies, rather than use traits for the filtering.“. Excellent idea, I say! Let’s have a bunch of test assemblies to make Visual Studio even slower. A few months back I’ve merged a million (OK, there was 6) of our test assemblies into one project to speed up VS and reduce maintenance. Now let’s revert this and create a few extra test assemblies to filter out by test types.

See this scenario: In one of my projects I have integration tests that need database. Also I have integration tests that use Azure Emulator. On the build server in one step I execute all non-integrtion tests first, then if none fails, in the next step I re-create database and further build step executes database tests. See this video for reasons behind it. For these build steps I first need to filter out database tests, then to include only database tests.

Because I use hosted build server I can’t run Azure emulator on the build server and all my tests using storage emulator will fail without it, so I need to filter them out.

According to xUnit authors, these tests should live in 2 separate assemblies. I have about 10 db-tests and about 20 for Azure emulator tests. This is 2 extra assemblies with very small number of tests. Good practice? I don’t think so! Only encourages to make a mess – I’ve been there, I did not like it. Every separate test project in your solution doubles maintenance burden.

What about people who separate tests by being “fast” and “slow”. And execute fast first, slow later on their build server. Or ignore slow tests on CI build, run them only on nightly builds. There is no clear distinction between the tests, unlike in my example. And within the same class you can have slow tests next to fast ones, all testing the same SUT. How do you propose to work this one out? Throw tests from one assembly to another when they become slow? Now here is some serious mess waiting to be happen.

And if you can’t filter by Traits on build server, what is the point of them? Gui runners usually allow you to choose what tests you’d like to run. And on very rare occasion I filter out tests by their category in GUI.

I know, you can filter out by traits with console runner. I could not make it to work -(. Also this sounds a bit hypocritical to me: authors allow to filter out traits in console runner, but not in MSBuild runner, because MSBuild for automatic test execution.

Conclusion

While I enjoyed writing tests with AutoData attribute from Autofixture, I can’t really say xUnit solved my issues with test execution. There is a big possibility that I don’t understand a lot of concepts behind this framework, but there is no good place to go for an explanation. And if somebody has answers to my moans, please feel free to speak out in comments! I’d love to hear you proving me wrong. Because I hope I’m wrong here.

So far xUnit been a disappointment in my experience with a lot of hype around it. I’ll keep it for the cases where I’ll benefit from AutoDataAttribute, but all other tests will be based on NUnit.

UPDATE: After initial excitement about xUnit, I needed to deal with some specifics. And now I don’t advise to move ALL your tests to xUnit. I’d recommend go into that slow and read through my rant here, before you commit to converting.

Nunit is a great testing framework. Only it is a little old now and new testing approaches are no longer fit into Nunit concepts. I’m talking about using Autofixture for generating test data. So I decided to see how xUnit will pan out for our projects and should we convert all our tests into xUnit or not.

Benefits of xUnit over NUnit are simple for me: AutoData attribute from Autofixture does not play nicely with NUnit due to architectural wirings. There are ways to have Autodata for Nunit, but this does not work nicely with Resharper 8.1 and NCrunch.

Also I decided to move away from Moq mocking framework and give a go to NSubstitute. I love Moq framework – it is simple, flexible enough and easy to learn. But if you churn out tests at any rate per hour, you are sick of typing Mock<IServiceName>() every time. And if you use Autofixture, then you have to type fixture.Create<Mock<IService>>() – a bit of mouthful. Amount of times I’ve forgotten to put Mock<> as part of generic type is infinite. This simply gets annoying.

NSubstitute allows you to create mocks without specifically saying Mock<>. And this shortens your tests and reduces scope for error.

I’ve made an experiment and compared exactly the same test in NUnit/Moq vs. xUnit/NSubstitute. Nunit test was 17 lines of code with setup, xUnit with Data Theory was 8. See this gist for old style. And check out the shortness of new approach. The tests are from production and both test exactly the same thing. Only second one takes almost half the size and time to write.

Convert Your Projects

In this blog-post I’ll talk through of adding xUnit to a project. I’m writing this as I’m converting one of my projects from NUnit/Moq to xUnit/NSubstitute.

First of all you’ll need new Nuget packages:

  • AutoFixture.Xunit
  • AutoFixture.AutoNSubstitute

These 2 will install for you xUnit, xUnit.Extensions, NSubstitute as dependencies. Then you’ll need to update NSubstitute to a latest, because as a dependency it does not give you a latest version.

Then you’ll need to create AutoData attribute for xUnit tests. Because I have a lot of tests (in this project I have 250+ tests) on NUnit with Moq, I still need to cater for old tests, because I’m not planning to convert them to xUnit at once. So I’ll need to make sure xUnit and NUnit tests work along side each other. First you’ll need to create classes for xUnit to work with Autofixture:

using NSubstitute;
using Ploeh.AutoFixture;
using Ploeh.AutoFixture.AutoNSubstitute;
using Ploeh.AutoFixture.Xunit;

// This one is to be used with DataTheory attributes    
public class AutoDomainDataAttribute : AutoDataAttribute
{
    public AutoDomainDataAttribute()
        : base(new Fixture().Customize(new XunitCustomisation()))
    {
    }
}


public class XunitCustomisation : ICustomization
{
    public void Customize(IFixture fixture)
    {
        fixture.Customize(new DomainCustomisation());

        fixture.Customize(new AutoNSubstituteCustomization());
    }
}

I separate Domain Customisations from mocking, because I also have this:

using Ploeh.AutoFixture;
using Ploeh.AutoFixture.AutoMoq;


public class DomainMoqCustomization : ICustomization
{
    public void Customize(IFixture fixture)
    {
        fixture.Customize(new DomainCustomisation());

        // use Moq for stuff that has interfaces
        fixture.Customize(new AutoMoqCustomization());

    }
}

This allows to have same Domain customisations, but with different mocking frameworks. Code reuse! And DomainCustomisation class looks like this:

public class DomainCustomisation : ICustomization
{
    public void Customize(IFixture fixture)
    {
        // ignore circular dependencies
        fixture.Behaviors.Remove(new ThrowingRecursionBehavior());
        fixture.Behaviors.Add(new OmitOnRecursionBehavior());

        // if object has Start and End properties, start will be assigned as RandomPastDate() and 
        // end will be assigned as RandomFutureDate()
        fixture.Customizations.Add(new StartEndDateSpecimenBuilder());

        // ignore all Virtual properties on complex objects. 
        //See here http://stackoverflow.com/a/15707393/809357
        fixture.Customize(new IgnoreVirtualMembersCustomisation());

        fixture.Inject<ILoggingService>(new TraceLogger());
    }
}

So old tests still use fixture = new Fixture().Customize(new DomainMoqCustomization()). Where new ones will involve xUnit and NSubstitute.

Now you are ready to write your first test in xUnit. I’ve picked a simple test from nUnit and converted from this:

[SetUp]
public void SetUp()
{
    fixture = new Fixture().Customize(new DomainMoqCustomization());
}

[Test]
public void MakeJobFromCommand_Always_MatchesFields()
{
    //Arrange
    var sut = fixture.Create<CreateJobCommandHandler>();
    var command = fixture.Create<CreateJobCommand>();

    // Act
    var result = sut.MakeJobFromCommand(command);

    // Assert
    AssertionHelpers.PropertiesAreEqual(command, result, "RoleOther", "RoleId", "AdminComments");
}

To this:

[Theory, AutoDomainData]
public void MakeJobFromCommand_Always_MatchesFields(CreateJobCommandHandler sut, CreateJobCommand command)
{
    // Act
    var result = sut.MakeJobFromCommand(command);

    // Assert
    AssertionHelpers.PropertiesAreEqual(command, result, "RoleOther", "RoleId", "AdminComments");
}

This basically removes all of the Arrange stage. Making the test half the size.

Please be careful with reference of Theory attribute. Both NUnit and xUnit have this attribute, so many times you need to specifically mention xUnit.Theory for things to work.

Binding Redirect

You may have noticed that I’ve advised to update NSubstitute to a latest version. And if you run the test above, you’ll get an exception:

System.IO.FileLoadException: Could not load file or assembly 'NSubstitute, Version=1.4.3.0, Culture=neutral, PublicKeyToken=92dd2e9066daa5ca' or one of its dependencies.

That is because Autofixture is build against older version of NSubstitute. To fix this problem open nuget console (in VS View -> Other Windows -> Package Manager Console) and run this command:

Add-BindingRedirect -ProjectName MyApp.Tests

Where MyApp.Tests is a project name for your test project. Or you can select Default project from the top of the console. This should modify your app.config file and add appropriate binding redirection for runtime that looks like this:

  <dependentAssembly>
    <assemblyIdentity name="NSubstitute" publicKeyToken="92dd2e9066daa5ca" culture="neutral" />
    <bindingRedirect oldVersion="0.0.0.0-1.7.1.0" newVersion="1.7.1.0" />
  </dependentAssembly>

IDE test runners

I’m using NCrunch for running my tests in Visual Studio. NCrunch has no problems with xUnit – supports out of the box.

If you are using Resharper 8.x there is an add-on for running and recognising xUnit tests. Go to Reshrper menu -> Extensions Manager; Find and install extension “xUnit.Net Test Support”. Restart VS and you’ll good to run your new tests in R#. I’m not sure how to do this in R#7, you’ll have to google yourself -)

Visual Studio also has a plugin to run xUnit in inbuild test runner. Go to Tools -> Extensions and Updates; Search and install for “xUnit Runner”. Again, restart VS to make it work. Or download it from Codeplex

Build Server

Next step is to get xUnit tests to be executed on your Build Server. We are using TeamCity, so I’ll talk about it just now. My other project is built on hosted TFS service http://tfs.visualstudio.com/ and make xUnit run on that beast will be a topic of my next blog post.

xUnit has inbuilt support for TeamCity – you don’t have to do anything special, only create .msbuild file to execute your tests and xUnit will check the environment and will output the result in format that TeamCity can understand.

To make xUnit run through msbuild you’ll need test-runner assembiles. And you already have the assemblies! Did you install xunit nuget package? Well, everything you need is already inside the package. Head to /packages/xUnit/**/ folder and find these files:

  • xunit.dll
  • xunit.runner.msbuild.dll
  • xunit.runner.tdnet.dll
  • xunit.runner.utility.dll

I like to have assemblies used for build process separately and have .build folder in solution. So I copied these files to .build/xunit. You might want to use the files from packages folder, but I don’t trust Nuget on this part.

My build script is on MsBuild, so in that file I’ve added another target that looks like this:

<UsingTask AssemblyFile="$(MSBuildProjectDirectory)\.build\xUnit\xunit.runner.msbuild.dll" TaskName="Xunit.Runner.MSBuild.xunit"/>
<Target Name="xUnitTests">
    <xunit Assembly="$(MSBuildProjectDirectory)\Tests\MyApp.Tests\bin\Debug\MyApp.Tests.dll" />
</Target>

You’ll need to point Assembly property from xunit element to your assembly(s) with tests. And that should do it. Now check in, don’t forget to check-in xunit assemblies in .build folder and changes to your build script.

After you’ve checked in, new build script is available on Build Server and you can add another step to your TeamCity build configuration:

xUnit TeamCity build step configuration

If everything went fine, you should see your xUnit tests running in TeamCity.

For my unit tests I’m using an awesome tool Autofixture as data generator and as an Automocking container. But by default for creation of objects it uses the constructor with least number of arguments. Mark Seemann talks about this in his blog. Also he provides code on how to make Autofixture work the opposite way. But the code example in that blog post is outdated and no longer valid in the latest version of the framework.

So every time I need this (and this is not often) I have too search for the solution over-again. And just to save myself this research next time, I’ll put the solution here:

var fixture = new Fixture();

fixture.Customizations.Add(new MethodInvoker(new GreedyConstructorQuery()));

In the past month I have been using brilliant tool from Mark Seemann called Autofixture. The tool is aiming unit tests and shall help you create objects with some data filled in.

Usually for unit tests you need to create your objects manually and give them some kind of data inside of the objects, so you don’t get NullReferenceExceptions. And that sometimes takes a while and not always pleasant:

        var person = new Person
        {
            FirstName = "Dilbert",
            MiddleName = "Wall-E",
            Surname = "Smith",
            KnownAs = "Pointy Haired Boss",
            DateOfBirth = new DateTime(1975, 12, 31),
            Gender = Gender.Male,
            MaritalStatus = MaritalStatus.Single,
        };

You can use builder pattern and re-use the builder every time you need an instance of Person.

var person = new PersonBuilder().WithSurname("Martin").Build();

That is more DRY and much more readable. Also to the test reader it is much more clear what are your intentions in the test and what is important for the test.

The problem comes when you have hundreds of models that all have to have builder classes. But AutoFixture relieves you from writing builders:

var fixture = new Fixture();
var person = fixture.Build<Person>().With(p => p.Surname, "Martin").Create<Person>();

Intent is clear, it is readable and you don’t have to write any builders. Autofixture will fill in all the properties and public fields for you. Strings by default are filled in with GUIDs, numbers are random. All the properties that are classes are also created for you and fill in with “junk” data. Autofixture walks down your model chain and creates objects for you, so you won’t see a NullReferenceException coming from your model ever again. This is just magic!

One thing that Autofixture allowed me to do is to replace Integration Tests with Unit Tests. I just did not need to go to database to test my service classes. Combination of Autofixture and Moq is just wonderful. Moq allowed me to mock out repositories and Autofixture return a descent set of data from mocks.

Say I have a repository like this:

public class PersonRepository : IPersonRepository
{
    private readonly MyContext context;
    public PersonRepository(MyContext context)
    {
        this.context = context;
    }

    public IEnumerable<Person> All
    {
        get { return context.People.AsEnumerable(); }
    }
    public Person Find(int id)
    {
        return context.People.Find(id);
    }
    // other methods
}

In my tests I can fake out repository by Moq manually:

var repository = new Mock<IPersonRepository>();

Or I can use Autofixture MoqCustomisation, where Autofixture replaces objects with fake objects where it can:

var repository = fixture.Create<Mock<IPersonRepository>>();

This just creates an instance of the faked out repository. If you are working with just a repository, this is good enough. But usually you want to test a service class that uses a repository as a dependency:

public class MyService
{
    private readonly IPersonRepository personRepository;

    public MyService(IPersonRepository personRepository)
    {
        this.personRepository = personRepository;
    }

    public Person FindById(int id)
    {
        return personRepository.Find(id);
    }

    public IEnumerable<Person> AllPeople()
    {
        return personRepository.All;
    }
}

For that case you need to create an instance of the MyService class and give it faked repository:

var repository = new Mock<IPersonRepository>();
var sut = new MyService(repository.Object);

Problem with this approach is maintainability. What if you have 5 dependencies in your service class? You’ll have to create fakes for all the dependencies.
What if you refactor your class and kill(or add) some of the dependencies? In this case your tests won’t compile until you add another dependency into the constructor.

Autofixture can work as automocking container and create all the mocks for you automatically:

var fixture = new Fixture().Customize(new AutoMoqCustomization());
var sut = fixture.Crate<MyService>();

Magic! This way your service got all the dependencies as fakes. Just magic, I’m telling you!

Next step in unit tests is to verify actions and usually you want to verify against your dependencies. With the snippet above you don’t get access to any of the fake objects. Now Freeze<>() comes to help from Autofixture.

var fixture = new Fixture().Customize(new AutoMoqCustomization());
var repository = fixture.Freeze<Mock<IPersonRepository>>();
var sut = fixture.Crate<MyService>();

Freeze<>() basically creates an instance of an object and keeps it in memory. And every time Autofixture needs another object of this type, it uses pre-created instance of the object. This way you get a handle of the faked object that is injected into your service. And can verify against it when you need it. Although be aware that you must freeze the object before you create your systems under test, otherwise injected fake won’t be the same as your handle.

Next step is to actually do some tests:

    [Test]
    public void FindById_GivenId_PersonObjectIsReturned()
    {
        //Arrange
        var fixture = new Fixture().Customize(new AutoMoqCustomization());
        var repostory = fixture.Freeze<Mock<IPersonRepository>>();
        repostory.Setup(r => r.Find(It.IsAny<int>())).Returns(fixture.Create<Person>());

        var sut = fixture.Create<MyService>();

        // Act
        var result = sut.FindById(fixture.Create<int>());

        // Assert
        Assert.NotNull(result);
    }

Here we set up an instance of fake repository, give it a setup, saying on Find() with any parameter of type int, return a person object that is created by Autofixture. Then we execute the method under test and verify that something was returned from it.

And another example:

    [Test]
    public void AllPeople_ReturnsCollectionOfPeople()
    {
        //Arrange
        var fixture = new Fixture().Customize(new AutoMoqCustomization());
        var repostory = fixture.Freeze<Mock<IPersonRepository>>();
        repostory.Setup(r => r.All).Returns(fixture.CreateMany<Person>());

        var sut = fixture.Create<MyService>();

        // Act
        var result = sut.AllPeople();

        // Assert
        Assert.IsNotEmpty(result);
    }

Here we set up repository to return a collection of People objects when asked for AllPeople(). Notice how I’m using fixture.CreateMany<People>() to create the collection of dummy data. By default Autofixture will give you 3 objects in the collection, but you can specify any number you like as a parameter: fixture.CreateMany<People>(42). Test assertion don’t really assert anything here, but I’m not really testing anything – just writing examples.

This is all for today. Next time I’ll talk about performance hit on your tests from Autofixture and how to deal with that.