MSTest vs. NUnit with Visual Studio 2010 & TDD

By | June 21, 2010

All unit testing frameworks share the same core features: test declaration, test execution, and pass/fail notices based on assertions. The shop I work at has been using NUnit for over two years now, with great success. We considered switching to MBUnit a year and a half ago, primarily because of Gallio’s compelling nature, but didn’t pull the trigger because of NUnit’s fluent interface and MBUnit’s lack thereof. Yes, we could have spun up our own fluent interface for MBUnit, but we preferred spending time solving business problems instead.

All that said, now that we’ve finally secured our Visual Studio 2010 licenses, we decided to give another look at MSTest to see how it would stack up given our day to day usage. We have three primary criteria when it comes to a unit testing tool:

  1. Be amenable to test driven development
  2. Stay out of the way
  3. Allow for readable tests sans comments

This post will compare MSTest to NUnit in Visual Studio 2010 to see how the two frameworks stack up from the perspective of those three criteria. It’s a fight to the end!

29718

The Problem Space

For our examination, we will spin up a framework upon which we can build game board objects. Think Tic Tac Toe or Connect Four. Our boards will have a number of rows and columns, will have two or more players, and each position on the board will be occupied by at most one player. We will code to the below interface layout.

interfacesWe’ll write up an implementation that represents a Tic Tac Toe board.  Since we’re aspiring to TDD, we need to wire up our tests first. To do that, we need to prepare our environment.

Environment Prep – MSTest

All you have to do to add a test project to your solution is to, well, add a test project to your solution. Right-click on the solution node and click Add –> New Project.

addMSTestProject1

Then expand the Visual C# node in the list of installed templates and select the Test option. There should only be one project type that is subsequently displayed in the middle portion, named Test Project. Select that and give your new test project a descriptive name. In our case, we’ll call it GameBoardTests.

addMSTestProject2

Then click OK and let Visual Studio do its thing for a few seconds. Once it’s done, you’ll see a brand new test project added to your solution. Visual Studio was even nice enough to add a new unit test class file for you called Unit Test1.cs. We’ll look at the code in this file when we actually start writing tests.

Environment Prep – NUnit

To get started with NUnit, you first have to go out and download the latest version. We mosey on down to the NUnit home page and click on the Download menu option at the top. As of this writing we see that the latest version is 2.5.5, so we download the msi and install it, taking note of the install location.

After that, we have to add another test project to contain our NUnit unit tests. We could have added these to the MSTest project that Visual Studio created for us, but I want to keep our comparison as much apples to apples as possible. For our NUnit tests, we’ll add a regular old class library project called NUnitGameBoardTests so that the distinction is painfully clear. This should give us a solution similar to the below.

bothTestProjects

Environment Prep Winner – MSTest

MSTest’s tight integration with Visual Studio pays off here. It’s virtually brainless to get going with it, whereas it takes a non-trivial amount of extra work to get NUnit ready to go.

Our First Test – MSTest

Our first logical test will ensure that when we create a Tic Tac Toe board, we have three rows and three columns. This will take the form of two physical tests, one to test the rows, and another to test the columns. Let’s pop open the source to the unit test that MSTest gave to us for free and see if we can actually use it.

using System;
using System.Text;
using System.Collections.Generic;
using System.Linq;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace GameBoardTests
{
    [TestClass]
    public class UnitTest1
    {
        [TestMethod]
        public void TestMethod1()
        {
        }
    }
}

Looks like your standard unit test shell. We’ll go ahead and rename this class and file TicTacToeBoardTests and write our first test, namely to test whether or not our board has three rows.

[TestMethod]
public void BoardHasThreeRowsTest()
{
    TicTacToeBoard b = new TicTacToeBoard();
    Assert.AreEqual(b.Rows, 3);
}

Simple stuff, though I am definitely missing NUnit’s fluent interface which we’ll see below. Obviously I can’t yet run our test as I haven’t defined the TicTacToeBoard class yet. Let’s go ahead and do that, along with the stub code that we need to compile after we declare our new class as implementing the IBoard interface (yes, let the TDD purists gnash some teeth).

public class TicTacToeBoard : IBoard
{
    public int Rows
    {
        get
        {
            throw new NotImplementedException();
        }
        set
        {
            throw new NotImplementedException();
        }
    }

    public int Columns
    {
        get
        {
            throw new NotImplementedException();
        }
        set
        {
            throw new NotImplementedException();
        }
    }

    public ICollection<IPlayer> Players
    {
        get
        {
            throw new NotImplementedException();
        }
        set
        {
            throw new NotImplementedException();
        }
    }

    public bool MakeMove(int x, int y, IPlayer mover)
    {
        throw new NotImplementedException();
    }

    public IPlayer Winner
    {
        get { throw new NotImplementedException(); }
    }
}

We need to add a reference to the MSTest test project for our GameBoards project, import the GameBoards namespace, and then finally we are off to the compilation races.

To run our test, all we have to do is right-click anywhere in the test body and select Run Tests.

msTestRunTest

You’ll note that I have another similar menu option called “Run Test(s)” in my context menu; this is for the awesome TestDriven.NET tool that no .NET developer should be without. That said, we won’t be using it during this demonstration, as we are testing out-of-the-box implementations only.

After running my tests, I get exactly what I expected: failure.

msTestResultWindow

Double-clicking on the error row takes me to the code that threw the error; this is a place where the tight integration with Visual Studio pays off. So we have basic test authoring, execution, and result checking covered, let’s get our tests passing. We’ll go ahead and wire up the row and column logic into the constructor of the tic tac toe board and create backing stores for the Row and Column properties of our class.

public TicTacToeBoard()
{
    Rows = 3;
    Columns = 3;
}

private int _rows;
public int Rows
{
    get
    {
        return _rows;
    }
    set
    {
        _rows = value;
    }
}

private int _columns;
public int Columns
{
    get
    {
        return _columns;
    }
    set
    {
        _columns = value;
    }
}

With this code in place, our test now passes.

msTestTestPassed

Given that the test that checks if we have three columns is pretty much the same thing, we’ll skip any detail on that here.

Our First Test – NUnit

With the work to get our tests passing performed up above, we can simply concentrate on wiring up NUnit to our project in this section. The first step is to add a reference to NUnit from our NUnit test project. You’ll find the needed reference in the Extensions group, as shown below.

nunitReference

With the NUnit reference added, we’ll rename our default class file to match the one that is in the MSTest test project. Then we add a reference to our GameBoards project as well, and pop open the test class file.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

using GameBoards;

namespace NUnitGameBoardTests
{
    public class TicTacToeBoardTests
    {
    }
}

As can be expected, there’s nothing here since this is just the empty class shell that Visual Studio provided for us when it created the class library project. Indicating that this class file is a unit test is done in the same way that MSTest does it, namely by adoring the class and subsequent methods with test attributes. We’ll import the NUnit.Framework namespace and write our test as below.

using NUnit.Framework;

using GameBoards;

namespace NUnitGameBoardTests
{
    [TestFixture]
    public class TicTacToeBoardTests
    {
        [Test]
        public void BoardHasThreeRowsTest()
        {
            TicTacToeBoard b = new TicTacToeBoard();
            Assert.That(b.Rows, Is.EqualTo(3));
        }
    }
}

Now the next step is to run it, as before. Right-clicking on the test doesn’t produce the nice little green arrow with the Run Test option anymore, however. I could use my fancy TestDriven.NET tool to run the test, but as I said above, that would be cheating given that TestDriven.NET is a third party tool that requires a purchase. Thankfully NUnit comes with its own test runner that we can use to run our tests. To use it, we have to browse to NUnit’s installation directory and launch the nunit.exe application.

nunitTestRunner

This gives us the NUnit test runner in all of its glory.

nunitTestRunner2

Beautiful, yet worthless at this point. To get this to run our tests, we have to load up the DLL that is created when our NUnit test project is built. To do that, we’ll have to create an NUnit test project into which we can load any test DLLs we want to inspect and run. Creating a new project is straightforward, we’ll skip over that step. To add your test DLLs to the newly created NUnit test project, we have to click on Project –> Add Assembly and then browse to the bin\Debug folder where our NUnitGameBoardTests.dll file is found. After we select that, our NUnit interface changes a bit.

nunitTestRunnerTestsLoaded

It’s a pretty straightforward interface. To run our test, we can either double click its name or just click the Run button from any of our test’s parent nodes in the left treeview.

nunitTestsPassed

Finally we have an indication that our tests passed as MSTest did. Note that one of the consequences of having to load our test DLL like this is that we have to rebuild it for the NUnit test runner to detect any changes in the test suite.

Our First Test Winner – MSTest

MSTest wins this round, again due to its tight integration with Visual Studio. With NUnit, we’re now saddled with an extra program that we have to run, and we have to build our test project before running our tests, a step I’m sure many developers will forget to do every now and then, enough so to cause consternation. While this determination isn’t based solely on the test content as our subsequent test comparisons will be, we have to take these additional dependencies into account somewhere, so we’ll just go ahead and do it here.

Our Second Test – MSTest

Our second test is to ensure that our tic tac toe board comes with exactly 2 players, and appears below.

[TestMethod]
public void BoardHasTwoPlayersTest()
{
    TicTacToeBoard b = new TicTacToeBoard();
    Assert.AreEqual(b.Players.Count, 2);
}

Simple enough. We go ahead and augment the constructor to create two players for us, and wire up the corresponding property to a backing store. We go ahead and create a Player object that implements the Player interface with a default implementation so we can make the compiler happy.

First our Player class.

public class Player : IPlayer
{
    public string Name
    {
        get
        {
            throw new NotImplementedException();
        }
        set
        {
            throw new NotImplementedException();
        }
    }
}

Our new Players property.

private ICollection<IPlayer> _players;
public ICollection<IPlayer> Players
{
    get
    {
        if (_players == null)
        {
            _players = new List<IPlayer>();
        }
        return _players;
    }
    set
    {
        _players = value;
    }
}

And finally our new constructor.

public TicTacToeBoard()
{
    Rows = 3;
    Columns = 3;

    Players.Add(new Player());
    Players.Add(new Player());
}

The test now passes, and we have verified that we have exactly two players after instantiating our tic tac toe board.

Our Second Test – NUnit

Again, since we went through the pains of actually getting our test to pass while working with MSTest, this section will be quick and painless. The NUnit test is below.

[Test]
public void BoardHasTwoPlayersTest()
{
    TicTacToeBoard b = new TicTacToeBoard();
    Assert.That(b.Players.Count, Is.EqualTo(2));
}

Our Second Test Winner – Draw

Nothing crazy went on in our second test, and since we had our environments set up already, the effort to get these working was bordering on trivial.

Our Third Test – MSTest

Our third test will ensure that we have two players with the right names, namely one as “Nought” and the other as “Cross.” The MSTest test is below.

[TestMethod]
public void BoardHasRightPlayersTest()
{
    TicTacToeBoard b = new TicTacToeBoard();
    Assert.AreEqual(b.Players.Count, 2);

    Assert.IsNotNull(b.Players.Where(x => x.Name.Equals("Cross")));
    Assert.IsNotNull(b.Players.Where(x => x.Name.Equals("Nought")));
}

We now have to change our Player class to provide a constructor which gives the client code an opportunity to pass in the player name, and subsequently wire up the Name property to a backing store. We then have to adjust our TicTacToeBoard constructor to use this constructor to pass in the desired player names. The new Player class is below.

public class Player : IPlayer
{
    public Player(string name)
    {
        Name = name;
    }

    private string _name;
    public string Name
    {
        get
        {
            return _name;
        }
        set
        {
            _name = value;
        }
    }
}

And the updated constructor is below.

public TicTacToeBoard()
{
    Rows = 3;
    Columns = 3;

    Players.Add(new Player("Nought"));
    Players.Add(new Player("Cross"));
}

Finally our test passes.

Our Third Test – NUnit

The NUnit version of the test is below.

[Test]
public void BoardHasRightPlayersTest()
{
    TicTacToeBoard b = new TicTacToeBoard();
    Assert.That(b.Players.Count, Is.EqualTo(2));
    Assert.That(b.Players, Has.Some.With.Property("Name").EqualTo("Nought"));
    Assert.That(b.Players, Has.Some.With.Property("Name").EqualTo("Cross"));
}

Again, since we’ve done the work to augment our classes to make our test scenarios pass, this test passes as soon as its written.

Our Third Test Winner – NUnit

The NUnit version just reads better to me. My bias was clear above, but the fact that we have to crack open lambdas for a test this straightforward doesn’t bode well for when we get into more complicated scenarios down the road. NUnit’s fluent interface really starts paying off here, and things only get better in the next case.

Our Final Test – MSTest

Our final test will check for the case wher ewe try to make a move that is outside of the bounds of the board. The test is below.

[TestMethod]
[ExpectedException(typeof(ApplicationException))]
public void OutOfBoundsMoveThrowsExceptionTest()
{
    TicTacToeBoard b = new TicTacToeBoard();
    b.MakeMove(4, 3, null);
}

Note the use of the attribute to indicate that the test method will be throwing an exception. To get this working, we haveto implement the MakeMove method on our TicTacToeBoard class, as below.

public bool MakeMove(int x, int y, IPlayer mover)
{
    if (x < 1
        || x > Rows
        || y < 1
        || y > Columns)
    {
        throw new ApplicationException("Move out of bounds");
    }

    return false;
}

Our Final Test – NUnit

Here’s where we really start to see the power of NUnit come into play as compared to MSTest.

[Test]
public void OutOfBoundsMoveThrowsExceptionTest()
{
    TicTacToeBoard b = new TicTacToeBoard();
    Assert.That(() => b.MakeMove(4, 3, null),
        Throws.Exception.TypeOf<ApplicationException>()
        .With.Message.Contains("Out of bounds"));
}

First off, no attributes; the meaning of the test is captured in the code. Next, we can check for the message that the exception throws back at us in addition to the exception type, which can be useful when we’re checking both for program correctness and display correctness.

Our Final Test Winner – NUnit

The clarity of the NUnit version in our final test beats the MSTest syntax hands down. Yes, you can extend MSTest such that you could write similar-looking tests when checking for exceptions, but again, we are reviewing out of the box implementations.

And The Winner Is…

NUnit, hands down. By the numbers above it’s actually a draw, given that both frameworks took two categories and tied on the other. But I feel that the overhead of an external application to run your tests and some additional up-front setup that will get rote-like after a few times is a small price to pay for more powerful assertions and a more legible and readable test style. Throw in TestDriven.NET, and it’s really no contest. I’m sure MSTest’s attraction could be upped with some third party add-ins as well, but when you’re having to load add-ins to increase readability vs. buying a tool to help you run your tests more efficiently, well…

So our shop will be staying with NUnit + TD.NET for a while. Let me know about yours in the comments.