Rebuttal: “My Software is Free of Defects!”

(Original article here)

Is There Such a Thing as Defect-free Software?

The opening points in the article say that software has to be exhaustively tested before it can be considered defect-free. The closing question asks if it’s even possible to consider software free of defects if you can’t possibly run all possible tests.

So if it is not possible to execute all tests to ensure that all defects was founded, there is such thing as Software Free of Defect?

The Scope

The premise is, in order for software to be completely vetted and found to be defect-free, every possible test case has to be run.

So to be clear: at least every possible combination of inputs.

Even so: Is it possible to consider a piece of software defect-free?

Yep, I Think So

Here’s why: It’s not required to run every possible test because code isn’t written in such a way that every possible test needs to be run. 

An example is given in the article which will help explain:

“Consider a small piece of software where one can enter a password, specified to contain up to three characters, with no consecutive repeating entries. Using only western alphabetic capital letters and completing all three characters, there are 26 × 26 × 26 input permutations (not all of which will be valid).”

So there are 17,576 possible 3-character combinations of letters A-Z. Other requirements are: there can’t be any repeated consecutive characters (such as “AAB” but “ABA” would be ok), the character string has to be 3 characters long–not 4, not 2, and no lowercase characters. No digits, no other kinds of characters.

The Exhaustive Testing Problem

There is a mentality I’ve seen toward testing that, in order to have the best possible software, you have to test every possible combination of inputs.

And it’s understandable that people who say that sort of thing out loud sometimes chuckle to themselves.

It’s just not feasible.

There are three ways this mentality is dangerous:

  • It leads testers to do a ton more work than is required to qualify software as ok-to-release,
  • It creates a crutch for people by setting up this standard, when code knowledge is much better to have instead,
  • It creates an unattainable standard–it really shouldn’t be tried for anyway.

There’s no reason to not white box test a piece of code. You should have (or be given) the ability to take a look at the dev’s code, and assess if there are ways to structure your test.

It’s not cheating to look at the code. You still have the requirements to test against, and make sure that the code does what it advertises, but you can also see what kind of input validation there is, and test against that.

I also think that you should know what kind of bugs you’re looking for, and search particularly for those, rather than attacking code with data and seeing what shakes out.

We can also safely assume that code’s not written such that you would ever need to test every combination.

Here’s an example of what I mean. Referring to the example above, the only way you would have to test every possible combination, would be if the code looked something like this:

switch(input) {
   case "AAA": return error; // AA is not allowed
   case "AAB": return error; // AA is not allowed
   case "ABA": return ok;
   case "ABB": return error; // BB is not allowed
   case "ZZZ": return error; // ZZ is not allowed
   default: return error; // got all possible 3-char combinations, anything else is an error

The thing is, no professional developer writes code like this. So I think as a professional tester, you shouldn’t expect to see code like this.

Instead, what you’d likely end up seeing is something like this:

input = getInput();
input.setPassword(); // if you made it here, yay!

Assume each validation will throw a particular error if the validation fails (i.e.: a non-3-character string will say something about the input needing to be 3 characters).

If your input survives the validation gauntlet then you have successfully entered a password per the requirements in the example.

The question is: do you need to test every possible combination of characters? And I’d still say no.

Each validation method is likely written in a way that it doesn’t care what you feed it–it’s looking for particular characteristics about the input data, not anything about specific pieces of data.

So for validateLength(3), you could give it “AAA” or “CORRECTHORSEBATTERYSTAPLE“, and it would still get validated.

For validateAllCapitalLetters() it’s unlikely this is a homebrewed method. Many languages give you some built-in validation of this type–C’s isupper() method is a great example.

The only complex looking function here would be validateNoConsecutiveLetters(). You may have to test this a little harder, but still won’t have to test every combination.

What Tests Do You Need?

For this example I’d suggest the following tests:

  • Null input–program definitely shouldn’t crash, so make sure it can be handled gracefully. Particularly the length validation–what happens if it gets an empty string? Does it puke or does it tell you you need 3 characters?
  • “AA1A”–this is for seeing if the order of validation ever changes. It’s 4 characters long, contains consecutive letters and also a number. Right now, the length check would be the first to fail, but if you ever got a different error, that would indicate the code changed. Why did the validation get switched around? Might be good to set up a tripwire for that instance.
  • “AB” and “ABCD”– should fail the length check. Also sometimes people mistakenly use <= or >= instead of !=, so check both 2- and 4-character strings. The length should be 3 and only 3.
  • “ABCDACBAASDFAPSOUFPOWEURNQWOCRUROPQUNRPOQWNROPQWNR…”–put in a decently-sized string to see if there’s possibly a stack overflow defect in your code (which normally you only see if you’re testing a C or C++ program, and most compilers will throw an error or a warning if the code’s using a function that could cause that kind of defect.)
  • “ABC”–simple happy path test. This should work just fine.
  • “AB1”–non-alphabetic character in here, should fail for that reason only.
  • “aba”–non-uppercase characters in here.
  • “ABA”–if there’s something goofy with the duplicate character validation (like it’s counting characters total instead of seeing if they’re next to each other), that’s a problem. This string should work according to the example.
  • Consider integration tests. How are these passwords getting stored? Something’s getting done with them–are they being stored in a database? Are they being encrypted? Can they be decrypted in the same way?

So 17,576 combinations down to… 9? 10? Much more manageable.

Other Tricks

There’s a whole science devoted to paring down a massive set of data into smaller sets of tests. The first trick that comes to mind is all-pairs testing, which surmises that most defects come from a pair of inputs–not a single input of a single type.

Rather than test all combinations then, you test all possible pairs of values. This greatly decreases the number of test cases.

(aside: there’s a great online tool called Hexawise that I plan to do a review on sometime, which helps with this approach–I’ve seen millions of test cases get smooshed down to, like, 40. Noice.)

Also, there’s nothing stopping you or the devs from writing unit tests for these. This example is more of a unit test example anyway–there’s one input, and the password probably gets stored in a database somewhere. Unit tests are also much faster to build and execute than system level tests, so if you can pull off your test at the unit level, I’d say go for it.

Defect-free code is possible! 

With the right strategy, you can have defect-free code. I don’t think it’s impossible–it’s just a matter of knowing the code you’re testing, and having the appropriate tests set up.

Easier said than done, I know, but definitely not impossible.

Happy testing!

– Fritz


2 thoughts on “Rebuttal: “My Software is Free of Defects!”

  1. I suggest that you start by reading “Principles of Program Design” Michael Jackson, Academic Press 1975, which explains exactly why you CANNOT prove a program is correct by testing. You cannot fully test even a single statement. That was 40 years ago, and the lessons don’t seem to have been fully learned. His strategy was to ensure the program design was correct, which minimises the risk of error. But there will still ALWAYS be errors unless a program is formally proved to be correct – and this has only been possible for small programs.

    Liked by 1 person

    1. Hi Neil,

      Thanks for replying. That book looks interesting and it’s at a low enough price I may just pick it up. Or I can hope someone will buy it for me 🙂 Is the whole book on that topic? Is it just a chapter or two devoted to that topic?

      There’s something about the idea that software has to be formally proven out, or all possible input combinations have to be tested, that’s just not sitting right with me.

      I’m getting a lot of replies that are pretty similar, so I will be posting a second part to this post within a week. For this post I focused more on application, which didn’t end up making my point as well as I’d hoped. So next time I want to focus just on that statement, and see where it goes.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s