Should You Build a Test Framework?

Through the years, I’ve been able to work at companies that either have test automation frameworks, or not. There have been good ones and bad ones.

It’s interesting, because a lot of factors come into play–culture, team skill level, needs of the company.

Very challenging puzzle.

Today, I’d like to talk about what the pros and cons are of writing (or not writing) a framework.

How About a Definition: What Is a Test Framework?

A framework is anything that abstracts away the low-level details, or frees up your time to think about higher-level things.

A method is the smallest example of a framework. If written right, you don’t have to think about the code the method contains.

So if you think about it… you’re actually using frameworks all the time. They’re all around you. They’re everywhere. 

woo boogedy boogedy.

A test framework in particular will abstract the low-level details of test automation. The more “atomic” things that you have to think about, such as interacting with webpage elements, or sending and receiving webservice payloads–these things, the framework should take care of, by not forcing you to have to think about the ugly.

The higher-level things that you want to think about instead, have more to do with focusing on the thing you’re testing.

Should You Write One? 

Maybe. I’ll admit, I’m on the fence on this. But let’s take a look at the pros and the cons:

The Pros of Writing a Test Framework

They make it easier to write tests: If you’ve ever had the, uh… privilege, of writing a complex automated test from scratch, you’ll understand why having a framework will help you.

The learning curve is shorter: There is great value in making certain jobs easier for other people. Other people in the team/department/company can get more done.

It increases the talent pool: It also increases the talent pool for hiring additional people–if the job isn’t as hard because you’re using a framework, there are more people to potentially pick from when they may be needed suddenly.

The Cons of Writing a Test Framework

The underlying technology can be hidden pretty easily. The tech stack that drives the framework can get buried by abstraction pretty quickly. It’s nice not having to think about the nuts and bolts, but sometimes you have to.

It’s all on you for support. One advantage to just using 3rd party tools is that the onus of support isn’t all on you. And there are a lot of options out there, so if one chunk doesn’t work, you can probably swap it out for another.

Marketing them in a resume is hard. It’s important to have marketable skills being developed by people on your team. It’s difficult to convey to a new company that you developed skills on a piece of home-rolled software. Yes they may end up leaving for better opportunities, but if your company is known for having Cool Things(tm) to work on, then you’ll attract talent.

The Pros of NOT Writing a Test Framework

The underlying technology is closer to the surface: When you can “see what you’re doing” it’s sometimes easier to understand the flow of logic and data in your test code. That’s a helpful thing.

You have more control: If you’ve written the code, it’s quicker to swap out code or make fixes, than trying to wrangle 3rd party tools.

The Cons of NOT Writing a Test Framework

It may be slower to write a test: if you’ve decided to not write a framework, and realize that your velocity isn’t as quick, it’s nice firepower to have. This depends mainly on how complex your work is–are you trying to test a lot of complex logic? Maybe not writing a framework is a con in this case.

If you aren’t going to write a framework

That’s cool. It’s a wide open, brave new world, and you will have complete control over all the operations you want to do, along with the freedom to drop or adopt any piece of technology you want.

have fun storming the castle
“Have fun storming the castle!” … “Think it’ll work?” “It would take a miracle.”

I’m not trying to sound pert, or trying to say you’re on your own.

Well, actually I am saying that. You’re on your own!

And that’s not a bad thing!

The kind of flexibility that being a free-range framework-free organic gluten-free automation tester offers, allows you to really dig in and try out a bunch of different technology stacks.

You’ll get to see what chunks of tech play nice with each other, and which don’t.

Simply do what works and don’t do what doesn’t work and you’ll be fine.

If you are going to write a framework

Consider combining the pros and anti-cons together. There are probably more pros and cons on either side than what I listed above, but let’s see what we get when we lay everything out:

  • Tests should be quick and easy to write,
  • The learning curve of the framework should be short,
  • Many people should be able to operate it–either in the department or company, or with new hires,
  • It should not bury the underlying technology–if you have to get under the hood for a fix, it shouldn’t take a long time to figure it out,
  • It should be easy to support,
  • It should include technology that people would be proud to market in their resume (and that technology should be accessible per the point made two points earlier),
  • The user should have as much or as little control as they want. If they just want to snap out code, or want to hack the framework, it should be made to be easy.

Holy grail? Not exactly.

At a previous job I had good success writing a framework in Ruby. All the team members–including the ones who weren’t programmers–were able to write code into it. The ones of us that could code were able to hack up the framework, while not disturbing what other team members were doing.

One approach I took was to use dynamic automation, to take care of defining locator methods at runtime. So instead of having to write up a bunch of click/select/enter methods, all that had to be done was specify the xpaths for each element you cared about, and the methods would be generated for you.

The methods were also named in a way that you didn’t have to go digging around the code to find what method to use–if you were looking at the webpage and you saw a Submit button, then the click function would be called “click_submit_button”. Always.

Oh, I forgot–if you plan to write a framework, PLEASE stay consistent. It’s really herky-jerky (and slows people down) when the flow and flavor of the framework code jumps around and is hard to follow. People shouldn’t have to think about your code if they don’t want to. Free them up to think about what they’re trying to test instead.

I guess I’m not on the fence anymore about frameworks!

If you end up writing one, most of all I hope you have fun and enjoy the challenge.

– Fritz

Hey, if you like this blog, consider following by clicking the Follow button. You’ll get an email when a new post is hot off the Press.

Feel free to share with your friends, groups or professional network, by clicking the appropriate button. 


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s