The devil of regression testing


"No sympathy for the devil; keep that in mind. Buy the ticket, take the ride...and if it occasionally gets a little heavier than what you had in mind, well...maybe chalk it off to forced conscious expansion"

 Hunter S. Thompson (Fear and Loathing in Las Vegas)

On our mailing list where we discuss all sorts of important development issues and organise pub trips, a mate of mine came up with an interesting problem to do with testing - It went a bit like this ..

“I've just been talking to someone who I has attempted agile without any QA or automated tests at all. Unsurprisingly they've seen an increase in regression issues and are now looking to retrofit done automated testing.”

As the resident tester it got me thinking. What is the cost of regression testing in an agile project ? - well the cost in my opinion is actually 0, zip, nadda, snake eyes !!

Right about now everyone who has faced the cost of regression testing is thinking I am smoking crack? or maybe hill billy drunk on the 4th of july? - I’m not and here is why.

Lets say you have a project you estimated at 30 points - whatever - you figure as you are going along you can do roughly 5 points a week including getting it manually QA’d in a test environment and showed with great fan fare to your product owner  - Happy days!! - you will be out of there cashing the check in 6 weeks... 

or will you.. 

What was your ACTUAL velocity??

Easy I hear you all say - 5 points!! 

The problem is, it wasn’t really - The 5 points a week was stuffed with tech debit, Now I’m not talking the stupid tech debit i’ve cited before where you just sweep all the hard stuff into the tech debit backlog to temporarily inflate your velocity (and we all know how thats going to end) I’m talking about serious tech debit that wont bite you for a month or so. The question is actually what did you DO for that 5 points? 

Write code that solved the problem?

Or write code that solved the problem with unit tests, integration tests, platform tests, automated UI tests that prove every build and deployment you haven’t broken anything??

In most cases its the former  - And here is where logically (though not financially I’m sorry to say) it doesn’t cost. What is actually happening is your velocity gets corrected. You weren’t doing 5 points, you were actually doing maybe 3 but incurring the tech debit of poorly tested code allowed the team to inflate their velocity to an unrealistic 5. The over all correction theoretically will be 4 weeks in this case - but anyone who has tried to go back and retrofit tests to poorly tested code will know its harder than writing them at the time. so in this case it will take longer. 

I guess if you can guarantee some other poor stooge will have to deal with it later you *could* give them crappily tested code to work with but once the smell has pervaded the building you wont be working back there any time soon and your rep wont be the greatest.


July 5 2011

NBuilder is fantastic for generating object graphs


A library which I have not seen used often enough is NBuilder. On my current project we needed to create a WCF service for consumption by a 3rd party. Since the structure of the web service was agreed beforehand, we wanted to publish an early version for them to connect to.

Instead of manually hand cranking the test data that the API was to return, we were able to use NBuilder to do 90% of the work, with overrides to handle more complex cases.

You can also use it in your unit tests wrapped by a builder pattern to create common object graphs for use by your tests.

We created some common interfaces for our builders to get a default setup of an object, which we could then use the provided builder methods to override.

public interface IBuilder<T>
    ISingleObjectBuilder<T> GetDefault();

public interface IManyBuilder<T>
    IListBuilder<T> GetDefault();

Next we implemented default builders to get the common configured setup. NBuilder has generators for many common types, and iterates through the object graph populating dummy data.

public class CustomerBasketBuilder : IBuilder<Basket>
    public ISingleObjectBuilder<Basket> GetDefault()
        var basketItems = new CustomerBasketItemsBuilder().GetDefault().Build();

        var basket = Builder<Basket>
            .With(basket1 => basket1.BasketItems = basketItems)
            .With(basket1 => basket1.ItemCount = basketItems.Count)
            .With(basket1 => basket1.CustomerId = new PrimitiveBuilder().GetCustomerId())
            .With(basket1 => basket1.TotalBasketValue = basketItems.Sum(item => item.BasePrice));

        return basket;


Whenever we use the With method here, we are setting up an override over the default generators. I manually call the BasketItemsBuilder to get the default setup for a list of items, and then ask NBuilder to Build the object graph using a combination of my overrides with the default generators.

public class CustomerBasketItemsBuilder : IManyBuilder<BasketItem>
    public IListBuilder<BasketItem> GetDefault()
        var generator = new UniqueRandomGenerator();
        var basketItems = Builder<BasketItem>
                    .With(item => item.ItemId = generator.Next(100000, 999999))
                    .With(item => item.Colour = Pick<string>.RandomItemFrom(new[] { "Red", "Green", "Blue" }))
                    .With(item => item.ItemType = BasketItemType.Product)
                    .With(item => item.ValidOperations = new[]
        return basketItems;


UniqueRandomGenerator is one of the OOTB generators, which is normally called implicitly, however we needed a defined range in this case. The Pick methods are also useful for one liner random item choosing. I strongly recommend going through the NBuilder codebase to see what’s available. There’s some real gems in there which unfortunately are not documented.

Here’s a refactored sample from one of our tests:

Before After
image image


It turns out refactoring unit tests is really satisfying, and reduces stress, I highly recommend it.

Be The Headlights Not The Handbrake


Some people will tell you that slow is good – but I’m here to tell you that fast is better. I’ve always believed this, in spite of the trouble it’s caused me. Being shot out of a cannon will always be better than being squeezed out of a tube. That is why God made fast motorcycles, Bubba…"

— Hunter S. Thompson (Kingdom of Fear: Loathsome Secrets of a Star-crossed Child in the Final Days of the American Century)

I haven't been at ThoughtWorks long but its been a happy couple of weeks - There is good conversations on a wide number of lists about all sorts of crazy stuff.

One that really grabbed my interest was a whole fiery chain about test automation and much opinion. Again someone raised the point of view that testers are like tigers (In the future survival stakes rather than the scary stakes) and will be extinct soon. This is something I have been yearning after for years - Imagine a world where the development practices employed by development teams are so quality driven we didn't need to test at all after they were done :-) 

I actually think many of the projects I have worked on were like this - It didn't make me redundant - In fact I have never been so busy! I spent a lot more of my time on defining acceptance criteria, Defining automation Scenarios, writing automation fixtures, Working with the devs on their unit tests, Generally finding ways to prevent the issues from occurring in the first place. In some ways the vision of the room full of testers busily finding fault with a release is redundant. But I think that has been going the way of the dodo fast with the uptake of agile.

However it seems that people are not sure what to do with us (QA) these days, Its as if the quality of a tester is ranked by the number of bugs they find. I would argue that if you are doing your job right the answer should be none!

On a big bang project with epic specs and reduced testing time I remember getting into trouble when I found bugs. "You keep raising issues like this we'll never go live!" my standard defence was always "I didn't put the bug there - I just found it" These days I don't believe that is sufficient. I think any bug I find is a failing of a process I am an integral part of. It is a failure of mine to get the right acceptance criteria, Or a failing of the communication channels between myself and the devs to ensure we are building the right thing, Or a failing in the automation scenarios we designed that allows some of the common edge cases to be passed incorrectly. These are just some examples but where you compare this to the old Us and Them (test and dev) mentality that used to exist I much prefer driving the quality of the process and application from the front where I can make an impact rather than from the end where all I can really do is find fault.

So does all this mean I never look at an application in a test environment anymore? - Not a bit of it - It does however mean I try to never spend hours going through a costly tedious boring and above all expensive regression test suite. I spend my time exploring the application and assessing the strange edges where it may fail - Its much more fun and a lot faster to identify critical issues. and I think I started this post with a quote that says fast is better.

Don't squeeze your application out of a tube 


November 24 2010

Bob the builder

I recently started a new project which is very focused on having good test coverage which is great goal to have.  However, looking through their tests I started to notice a lot of setup code which was very similar and looked a lot like the following:

   1: [TestMethod]
   2: public void WhenTheAddressRepositoryIsAskedToAPersistAddressesItShouldCorrectlyPersistThoseAddresses()
   3: {
   4:     // Arrange
   5:     var addressesToPersist = new List<Address>();
   7:     var address = new Address
   8:                           {
   9:                               AddressLine1 = Guid.NewGuid().ToString(),
  10:                               AddressLine2 = Guid.NewGuid().ToString(),
  11:                               AddressLine3 = Guid.NewGuid().ToString(),
  12:                               AddressLine4 = Guid.NewGuid().ToString(),
  13:                               City = Guid.NewGuid().ToString(),
  14:                               County = Guid.NewGuid().ToString(),
  15:                               CountryCode = Guid.NewGuid().ToString(),
  16:                               Postcode = Guid.NewGuid().ToString()
  17:                           };
  18:     addressesToPersist.Add(address);
  20:     var address1 = new Address
  21:                            {
  22:                                AddressLine1 = Guid.NewGuid().ToString(),
  23:                                AddressLine2 = Guid.NewGuid().ToString(),
  24:                                AddressLine3 = Guid.NewGuid().ToString(),
  25:                                AddressLine4 = Guid.NewGuid().ToString(),
  26:                                City = Guid.NewGuid().ToString(),
  27:                                County = Guid.NewGuid().ToString(),
  28:                                CountryCode = Guid.NewGuid().ToString(),
  29:                                Postcode = Guid.NewGuid().ToString()
  30:                            };
  31:     addressesToPersist.Add(address1);
  33:     var addressRepository = new AddressRepository();
  35:     // Act
  36:     addressRepository.Persist(addressesToPersist);
  37:     var actualPersistedAddresses = addressRepository.GetAll();
  39:     // Assert
  40:     Assert.IsNotNull(actualPersistedAddresses);
  41:     Assert.AreEqual(2, actualPersistedAddresses.Count);
  42: }

Now the above code isn’t all that bad in a single test but what happens when you start to write more tests?  Well you could copy and paste the code for creating addresses from test to test which will certainly work but you end up with a lot of very similar code which you have to maintain going forward and update whenever a new address property is added or removed.  All this sounds like a lot of work plus it goes against the DRY principle.  DRY for those of you who don’t know means “Don’t Repeat Yourself”.  The principle encourages that duplication within an application is removed as every line of  code written has to be maintained and each line of code can cause bugs. More importantly the principle is about having a single source of knowledge within a system and then using that source to generate instances of that knowledge. In this case the instances of knowledge would be the address objects which are being created within each unit test.  The question then is how do we create something which we can use as the source of that knowledge?

Enter the builder

That single source of knowledge within the project I’m working on is provided by using the object builder pattern. Using this pattern we can have a single and consistent approach to creating the entities within tests and the ability to easily extend those entities within our domain and test data. We can also chain the builders together to create complex test data very quickly.

So how do you create a builder? Well below is my implementation of an address builder to get you going.

   1: public class AddressBuilder : BuilderBase<AddressBuilder, Address>
   2: {
   3:     private int id;
   4:     private string addressLine1 = GetUniqueValueFor();
   5:     private string addressLine2 = GetUniqueValueFor();
   6:     private string addressLine3 = GetUniqueValueFor();
   7:     private string addressLine4 = GetUniqueValueFor();
   8:     private string city = GetUniqueValueFor();
   9:     private string countryCode = "GB";
  10:     private string postCode = "SE1 9EU";
  11:     private string county = "London";
  13:     public AddressBuilder WithId(int id)
  14:     {
  15: = id;
  16:         return this;
  17:     }
  19:     public AddressBuilder WithAddressLine1(string addressLine1)
  20:     {
  21:         this.addressLine1 = addressLine1;
  22:         return this;
  23:     }
  25:     public AddressBuilder WithAddressLine2(string addressLine2)
  26:     {
  27:         this.addressLine2 = addressLine2;
  28:         return this;
  29:     }
  31:     public AddressBuilder WithAddressLine3(string addressLine3)
  32:     {
  33:         this.addressLine3 = addressLine3;
  34:         return this;
  35:     }
  37:     public AddressBuilder WithAddressLine4(string addressLine4)
  38:     {
  39:         this.addressLine4 = addressLine4;
  40:         return this;
  41:     }
  43:     public AddressBuilder WithCity(string city)
  44:     {
  45: = city;
  46:         return this;
  47:     }
  49:     public AddressBuilder WithCountry(string countryCode)
  50:     {
  51:         this.countryCode = countryCode;
  52:         return this;
  53:     }
  55:     public AddressBuilder WithPostCode(string postCode)
  56:     {
  57:         this.postCode = postCode;
  58:         return this;
  59:     }
  61:     public AddressBuilder WithCounty(string county)
  62:     {
  63:         this.county = county;
  64:         return this;
  65:     }
  67:     public override Address Build()
  68:     {
  69:         var address = new Address
  70:                           {
  71:                               Id =,
  72:                               AddressLine1 = this.addressLine1,
  73:                               AddressLine2 = this.addressLine2,
  74:                               AddressLine3 = this.addressLine3,
  75:                               AddressLine4 = this.addressLine4,
  76:                               City =,
  77:                               CountryCode = this.countryCode,
  78:                               County = this.county,
  79:                               Postcode = this.postCode
  80:                           };
  82:         return address;
  83:     }
  84: }

As you can see the code behind the builder is really very simple with default values assigned to the properties which can then be explicitly set using the “With” methods on the builder. When I’m using builders I also like to add a base for them which provides default behaviour.

   1: public abstract class BuilderBase<TBuilder, TBuild> where TBuilder : BuilderBase<TBuilder, TBuild>
   2: {
   3:     /// <summary>
   4:     /// Builds this instance.
   5:     /// </summary>
   6:     /// <returns></returns>
   7:     public abstract TBuild Build();
   9:     /// <summary>
  10:     /// Builds the many.
  11:     /// </summary>
  12:     /// <param name="number">The number.</param>
  13:     /// <returns></returns>
  14:     public List<TBuild> BuildMany(int number)
  15:     {
  16:         var result = new List<TBuild>();
  18:         for (int i = 0; i < number; i++)
  19:         {
  20:             var builtItem = this.Build();
  21:             result.Add(builtItem);
  22:         }
  24:         return result;
  25:     }
  27:     /// <summary>
  28:     /// Gets the unique value for.
  29:     /// </summary>
  30:     /// <returns></returns>
  31:     protected static string GetUniqueValueFor()
  32:     {
  33:         return Guid.NewGuid().ToString();
  34:     }
  35: }

So now we have the address builder this put it all together to create our new and improved unit test.

   1: [TestMethod]
   2: public void WhenTheAddressRepositoryIsAskedToAPersistAddressesItShouldCorrectlyPersistThoseAddresses()
   3: {
   4:     // Arrange
   5:     var addressesToPersist = new AddressBuilder().BuildMany(2);
   6:     var addressRepository = new AddressRepository();
   8:     // Act
   9:     addressRepository.Persist(addressesToPersist);
  10:     var actualPersistedAddresses = addressRepository.GetAll();
  12:     // Assert
  13:     Assert.IsNotNull(actualPersistedAddresses);
  14:     Assert.AreEqual(2, actualPersistedAddresses.Count);
  15: }

As you can see the duplicate code for creating each address is now gone and we have a nice simple method for creating as many address as we want using the “BuildMany” method provided by the base builder. Well that’s it for now. The complete source code for this example can be found at

Older Posts