Mutation Testing

As a tester, sometimes I lose sleep wondering if my tests are actually good enough.

Sure I might cover happy path and some appropriate negative cases. I may be writing tests according to what kinds of bugs I want to catch.

The tests might even all pass. Great news, right?

Well… hm. I mean… hm.

Thaaaaaaaat’s hard to say. I can’t bring myself to say that.

I mean, I didn’t run every possible combination of data. There’s no way to do that and get done in a decent amount of time.

How else can you tell if your tests are good enough to catch problems in the code? In essence: How do you test your tests?

That’s where mutation testing comes in.

What’s Mutation Testing?

Mutation testing is a method of seeing how robust your test suite is.

It’s primarily used with unit tests, which is great, since unit tests usually run much faster than heavier-weight tests.

Howzit Work?

The way it works is, it makes minor changes to your code (at runtime! don’t worry), and runs your unit tests against the changed code.

If your tests are robust enough, they should fail. Some, anyway.

A piece of changed code is called a mutant. If any of your tests fail, it’s called killing the mutant.

The types of changes that a mutator can do are ones that will change the behavior of your code.

So it can do things like:

  • Change whether a method is public or private
  • Change the order of given method arguments
  • Switch equality operators–<= instead of >=, or == instead of !=
  • Remove the “super” operator if used
  • Many other things

There’s actually a great PDF here describing some mutations that can be done for Java programs. There’s a table on page 11.

Mutations like this will cause you to think about what tests you have.

If your tests don’t detect whether a private method is made public, then are you checking to see if inheritance is being used properly?

If your tests can’t tell when the order of arguments has changed, are you even testing that method?

A lot of crazy can happen if equality operators are switched around–some of your tests really should fail if this happens.

I’m Bringin’ Rb(3.15)2Rk* Back!

So I’ve noticed a lot of places don’t even talk about mutation testing. To date, I’ve heard of ONE place that mentioned it.

And it was during an interview–incredibly exciting. Was really surprised because I was going to bring it up but they beat me to it. Awesome awesome.

Mutation testing’s actually been around since the early 70s. But because technology wasn’t as good back then, the ROI wasn’t as high. As such, mutation testing has been relegated to more an academic exercise.

Now though we have technology coming out the wazoo so I think it’s time to bring it back.

There are lots of mutation testing frameworks available for different languages. I’ve heard of PIT for Java, and Heckle for Ruby. There’s one for Python too, somewhere.

I’m also huge on using unit tests for almost everything–in fact, I subscribe to the theory that every test can be boiled down to a unit test (within reason of course. UI tests can’t be caught at the unit level).

With mutation testing in place, if your testing team is focused on keeping the mutator happy, a ton more unit tests will be created, and your quality should go up without sacrificing a ton of time on complex, long-running tests.

What do you think, worth a shot?

Thanks for reading!


*(That’s a genetic mutation nomenclature there, was trying to be funny. I bet one guy laughed.)


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s