For manual testing, it’s important to have a record of what’s being tested.
We need to track our body of work–how to actually test the things–and be able to share that knowledge among other team members.
This body of work is made up of test cases.
For test case format, I’ve seen:
- General descriptions of how things should work,
- Step by step instructions, and
- Adhoc testing with NO step by step instructions. Wild!
On the justification of test case formats, I’ve heard:
- We should focus on the functionality, not the specifics of how to test it,
- Tests should be understood by anybody coming in off the street,
- Exploratory is the best way to find bugs, and we shouldn’t be limited by test cases,
respectively.
The variable that’s common to each approach is detail. How detailed should our manual test cases be?
The Devil’s in the Details
There’s a maintenance cost with writing anything down. The more you write, the more there is to maintain. And there’s a different cost with not writing anything down.
Being too detailed can actually be… well, kinda painful. Where’s the balance? Probably somewhere in the middle.
Example
Let’s say you have a website where you can create a user. After creating the user, the next screen lets you talk a bit about yourself–education, hobbies, profession, etc.
As a detail-oriented manual tester, one of your happy path test cases might look like this:
Testcase: Create a new user profile
Go to http://www.mytestsite.org
Click the “New User” link at the top of the screen
On the Create New User screen, enter a username
Enter a password
Enter the password again in the confirmation box
Click “submit”
On the “Tell Us About Yourself” page, select the “Software Tester” profession
Select “Doctorate” from the Education dropdown
Check the “Full Contact Origami” checkbox in the Hobbies section
Click the Submit button
This is one test. Any other test with this level of detail, that goes through the process of creating a user–positive or negative case–will include some info that we’ve already written out above.
Now imagine that we have 20 different points of functionality that we want to test, that require us to create (or at least try to create) a new user.
What would happen if it was decided that now we have to make the user enter their email address as part of creating a profile? Now every test has to be updated to have a line that says:
Enter a valid email address into the Email field
You’d also have to make similar changes if any of the following things happened:
- Change to the url
- The “New User” link text changes–maybe it now says “Register” instead of “New User”,
- The title of the Create New User screen changes,
- The “Tell Us About Yourself” page title changes, or
- the Profession dropdown goes away because meh it’s too personal, bad UX,
- Any other “choice” section changes names or goes away (i.e.: education, hobbies)
- The chosen value mentioned in any of the “pick this option” test steps is no longer available…
I think you get the idea. Any of these would require a change to the test case, and changes like this do happen often.
The Pain Point
A tenet I operate by is, “Every line of code has to be maintained.”
But guess what: even text has to be maintained too, or it’s stale.
Test cases written in this amount of detail will incur a huge maintenance hit after awhile, and you’ll find yourself spending a ton of time keeping the test cases fresh, rather than writing new ones or running the ones you have.
Although they mean well, test cases written like this are actually more harm than good. They force a maintenance point that doesn’t scale with the number of test cases created.
It forces us to either hire additional people, or unfortunately start cutting corners to meet deadlines. Both options are expensive.
This scalability problem will never go away unless something changes.
Is There a Better Way?
Yep I believe so. There’s almost always a better way to do anything.
We’re probably not hiring just anybody to do testing. And whoever we get, is probably smart enough that if you show them how to do something, they’ll remember it.
Instead of that much detail, let’s describe it in a way that still gets the point across, without requiring us to dip back into it for maintenance as often:
Create a new user profile with a profession, education and hobby.
Go to the site
Create a new user with a random profession, education and hobby
Submit the new user information
Actually we probably didn’t even need the test steps on this. The title describes it pretty well.
But see what we did there? Last time we talked about how to test user creation. This time though we’re talking about what is being tested.
When This Much Detail Is OK
I will say, there’s a time when a step-by-step level of detail is ok, and that is for when you’re using these tests to drive your automation.
This detail is great for automation tools like Cucumber, where it has to handle both the “what” and the “how” of running the test.
It’s ok to have this level if it’s actually doing work for you, with automation. The maintenance then would be helpful, because it helps you later by automating those detailed test steps on your behalf.
Positive Effects of Less Detail
There are additional benefits to managing test cases this way:
- Instead of writing a new test, just for slightly different data, consider using some kind of table to hold the data combinations.
- Generic tests require some pairing and collaboration to get new people ramped up. Very helpful for onboarding and setting the correct first impression.
- Also helps with holistic understanding of the system. Step-by-step instructions don’t explain why the system behaves the way it does, what value the product provides, or where the revenue stream can be impacted.
- Step-by-step instructions also don’t allow for much exploratory testing, which is where the really fun stuff happens.
Are you pinned down on test case maintenance? Maybe it’s time for a change.
With my consultancy, Arch DevOps LLC, I solve problems like above. Particularly, I solve the problem of arming manual testers with the tools and knowledge to automate their tests.
Would you like to find out more?
testzius 