Tests Getting Slower?

sloooooooooooooow doooooooooooooown

Why is it that our automation starts off fast and get slow after awhile? It’s likely because of sleeps.

Here’s a scenario:

  • Write a test.
  • It works.
  • A little later, it doesn’t work.
  • Why not?
  • Because something didn’t show up or respond quick enough.
  • It’s not really a bug, so let’s put a sleep in there. We’ll even be smart and check periodically up to a timeout, instead of just sticking a static N second sleep in there.
  • Now it passes. Much better.

A short time later…

  • I’m a new guy, running some tests.
  • One of them fails.
  • Why?
  • Because something didn’t show up or respond quick enough.
  • Oh, but hey, there’s already a sleep there, let’s just bump it up a little bit.
  • Now it passes. Much better.

In the near future…

  • Hey we need to write a new test.
  • There’s a test over here that’s really close to what we want.
  • (you know, the one with the sleep)
  • Copy/paste/modify (but leave the sleep as-is)
  • Test passes. Very cool.

Now the sleep is propagated in multiple places.

This Happens All. The. Time.

I think of a “sleep” like how I think of “goto” in code. They’re not bad themselves, they’re just easy to misuse and cause problems for everybody. We can see how quickly quick test suites can start to slow down.

But sometimes, we really do need to wait for something to happen, before we continue. How can we combat this problem?

Set the Terms

Figure out if your system has a Service-Level Agreement (SLA) for how quickly things should respond.

Whether that’s external (driven by the client) or internal (driven by team discipline), get agreement from everybody to say “[x] operation should take not more than [y] seconds”.

Smart Waiting

Please avoid putting static sleeps in your tests of N seconds. I assume that most of the time your tests will pass, and whatever condition you’re wanting to have happen within that amount of time will have happened.

If you do a hard wait of 30 seconds for something that shows up in 5, you’ve wasted 25 seconds. Multiply that by possibly hundreds of tests, and you’ve wasted a buttload of time.

Instead, wrap the condition check inside a loop, and wait a really small amount of time–a second, or maybe a fraction of a second–and keep checking until either the thing you want to have happen happens, or a timeout has elapsed. If the latter occurs, throw an error.

Code Hygiene

Use some global variables to hold timespans for waiting, instead of hardcoding numbers into even smart sleep loops.

The reason for this is: it’s much easier to make a wholesale change to the times in all places, when you need to. This is great for when your system becomes more performant, and you want to make sure it stays that way.

I recommend naming variables after what they’re waiting on, such as waitForPageLoad, waitForWhateverOperationToComplete, etc. This helps when tests fail, because you’ll usually get a stack containing that variable name in there, and it may be more obvious that that operation took too long to finish.

Bottleneck Visibility

When we just increase time, we can be masking performance bugs. We’re probably not aware of what the time was originally, we only see what it is now and what’s going to be once the change is made.

But what if it was originally a 5 second wait, and it’s gradually crept up to 25 seconds? Making it 30 wouldn’t be terrible, relative to 25, but if it was originally 5 then we’ve got a huge bottleneck somewhere that’s making an operation take 6x longer than it did originally.


There are lots of patterns and anti-patterns that bungle up the works with automation. This is one of them. Do you have others? Let me help you find them. 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s