Beyond BDD with Cucumber

I’m a huge fan of Behavior-Driven Development.

Although many people use tools like Cucumber for tests, technically that’s not what Cucumber does. Cucumber doesn’t test. All it does is call chunks of code based on what phrases are used in a Feature file.

And that human-language-to-computer-language-with-regex engine is really helpful for increasing the audience of contributors–as in, you don’t have to be a developer to write Feature files–but it’s also helpful for other tasks that are related to QA, without actually doing testing.

Today I’ve got two examples of ways to think beyond BDD, using Cucumber. For brevity, I’ll skip the install/basics of Cucumber and assume you have some experience with it already.

Here we go:

Pairwise Test Generation

When we’re faced with testing a system with a large set of possible inputs, we might be tempted to test every possible combination to get full coverage.

But what if that would result in thousands or millions of test cases? Does every test case really add that much value? Probably not.

There are some ways to whittle down the possible combinations, and one way is to do Pairwise Testing.

Pairwise testing is based on the fact that most software bugs don’t come from the system failing because of just Value A, or just Value B, but something about values A and B together. So instead of testing every possible combination of inputs, we test the combinations that have all the possible pairs of values. If you’re interested in how this works, I’ll be putting a post together about it soon.

Cucumber can allow us to specify what the parameters are, and even some invalid pairs, and then generate the test data for us. Here’s how:

combo.feature

Feature: Generate Combinatoric Tests via Cucumber

Scenario: First Combinatoric Tests
Given the "state" can be "MO, IL, AR, IA, KY"
And the "company" can be "unisys, compucom, ficticomp"
And the "product" can be "shipper, dropper, sendem"
And "unisys" is not in "MO"
And "unisys" is not in "IA"
And "compucom" is not in "AR"
And "compucom" is not in "KY"
And "shipper" is not in "ficticomp"
When the tests are generated
Then the set of pairwise tests are...

steps.rb

$parameters = []
$combinations = []
$invalids = []
$pairs = []
$tests = []

def get_all_combinations(array)
 return array[0] if array.size == 1
 first = array.shift
 return first.product( get_all_combinations(array) ).map {|x| x.flatten.join("@@").split("@@") }
end

Given(/^the "([^"]*)" can be "([^"]*)"$/) do |arg1, arg2|
 $parameters.push(arg2.split(", "))
end
 
Given(/^"([^"]*)" is not .* "([^"]*)"$/) do |arg1, arg2|
 $invalids.push([arg1, arg2])
end

When(/^the tests are generated$/) do
 # get all the possible combinations
 $combinations = get_all_combinations($parameters)
 filtered_combinations = []
 # filter out the ones with invalid pairs
 $combinations.each do |c|
  invalid_found = false
  $invalids.each do |i|
   invalid_found = true if (c.include?(i[0]) and c.include?(i[1]))
  end

  next if invalid_found
  # if you made it this far, this is a valid combination--save it
  filtered_combinations.push(c)
 end
 # set the global list of combinations to the filtered out ones
 $combinations = filtered_combinations

 # get each pair out of each combination
 $combinations.each do |c| 
  c.combination(2).to_a.each do |p|
   $pairs.push(p)
  end
 end

 $pairs.uniq!

 # get one test for each pair and stick it into an array
 $pairs.each do |p|
  $combinations.each do |c|
   if c.include?(p[0]) and c.include?(p[1])
    $tests.push(c)
    break
   end
  end
 end

 $tests.sort!.uniq!
end

Then(/^the set of pairwise tests are\.\.\.$/) do
 $tests.each do |t|
  puts t.join(" ")
 end
end

Result:

 AR ficticomp dropper
 AR unisys dropper
 AR unisys sendem
 AR unisys shipper
 IA compucom dropper
 IA compucom sendem
 IA compucom shipper
 IA ficticomp dropper
 IL compucom shipper
 IL ficticomp dropper
 IL unisys dropper
 IL unisys sendem
 IL unisys shipper
 KY ficticomp dropper
 KY unisys dropper
 KY unisys sendem
 KY unisys shipper
 MO compucom dropper
 MO compucom sendem
 MO compucom shipper
 MO ficticomp dropper
 MO ficticomp sendem

It would be pretty easy to massage this data for use in an actual Cucumber test.

Finding Test Paths

When testing a complex system, it helps to have some kind of flow diagram. How does the data flow? What are the decision points? How many tests would be needed to get decent coverage?

Even when you have a diagram like that, it can be challenging to figure out what all the different paths are, so that tests can be generated.

An example flow diagram is here, which I’ll use for the code below. Couple things to note: My code doesn’t handle decision nodes, so I break those up into doesn’t handle circular paths well (or at all, actually), so I’ll have to be a little creative to get around those. And it also doesn’t handle decision nodes–so places where a node has a yes/no direction will get broken into two separate nodes. (#TODO)

The Feature file will specify what nodes connect to what other nodes using the “->” token, then the step that generates the different code paths will be given the node to start with.

generate.feature

Feature: Quote Generation

Scenario: Generate Paths
Given "Prepare Requisition" -> "Prepare Request for Quote"
And "Prepare Request for Quote" -> "Does Not Need Review"
And "Prepare Request for Quote" -> "Needs Review"
And "Does Not Need Review" -> "Review RFQ"
And "Needs Review" -> "Evaluate RFQ"
And "Evaluate RFQ" -> "Does Not Approve"
And "Evaluate RFQ" -> "Approves"
And "Approves" -> "Review RFQ"
And "Review RFQ" -> "Decides to Quote?"
And "Decides to Quote?" -> "Prepare Quote"
And "Prepare Quote" -> "Review Quote"
And "Review Quote" -> "Quote is Acceptable"
And "Review Quote" -> "Quote is Not Acceptable"
And "Quote is Acceptable" -> "Prepare Order"
And "Quote is Not Acceptable" -> "Review Quote Response"
And "Prepare Order" -> "Review Order"
And "Review Order" -> "Order Not Acceptable"
And "Review Order" -> "Order Acceptable"
And "Order Acceptable" -> "Fulfill Order"
And "Fulfill Order" -> "Receive Ordered Items"
And "Receive Ordered Items" -> "Prepare Invoice"
And "Receive Ordered Items" -> "Make Payment"
And "Prepare Invoice" -> "Make Payment"
And "Make Payment" -> "Receive Payment"
Then I generate the tests for all paths starting from "Prepare Requisition"

steps.rb

$nodes = {}
$all_paths = []

def paths(node,path='',&proc)
 if $nodes[node][:children].empty?
 $all_paths.push(path+node)
 else
 $nodes[node][:children].each{|c| paths(c,path+$nodes[node][:name]+"->",&proc)}
 end
end

Given(/^"([^"]*)" \-> "([^"]*)"$/) do |node, child|
 # set up a hash for this node if there's not one already
 if $nodes[node] == nil
 $nodes[node] = {}
 $nodes[node][:children] = []
 $nodes[node][:name] = node
 end

 # set up a hash for the child node if there's not one already
 if $nodes[child] == nil
 $nodes[child] = {}
 $nodes[child][:children] = []
 $nodes[child][:name] = child
 end

 # add this node to the hash, along with its children
 $nodes[node][:children].push(child).sort!.uniq!
end

Then(/^I generate the tests for all paths starting from "([^"]*)"$/) do |arg1|
 paths(arg1)
 $all_paths.each do |path|
 puts ""
 steps = path.split("->")
 puts steps.join("\n")
 end
end

Results

 Prepare Requisition
 Prepare Request for Quote
 Does Not Need Review
 Review RFQ
 Decides to Quote?
 Prepare Quote
 Review Quote
 Quote is Acceptable
 Prepare Order
 Review Order
 Order Acceptable
 Fulfill Order
 Receive Ordered Items
 Make Payment
 Receive Payment

 Prepare Requisition
 Prepare Request for Quote
 Does Not Need Review
 Review RFQ
 Decides to Quote?
 Prepare Quote
 Review Quote
 Quote is Acceptable
 Prepare Order
 Review Order
 Order Acceptable
 Fulfill Order
 Receive Ordered Items
 Prepare Invoice
 Make Payment
 Receive Payment

 Prepare Requisition
 Prepare Request for Quote
 Does Not Need Review
 Review RFQ
 Decides to Quote?
 Prepare Quote
 Review Quote
 Quote is Acceptable
 Prepare Order
 Review Order
 Order Not Acceptable

 Prepare Requisition
 Prepare Request for Quote
 Does Not Need Review
 Review RFQ
 Decides to Quote?
 Prepare Quote
 Review Quote
 Quote is Not Acceptable
 Review Quote Response

 Prepare Requisition
 Prepare Request for Quote
 Needs Review
 Evaluate RFQ
 Approves
 Review RFQ
 Decides to Quote?
 Prepare Quote
 Review Quote
 Quote is Acceptable
 Prepare Order
 Review Order
 Order Acceptable
 Fulfill Order
 Receive Ordered Items
 Make Payment
 Receive Payment

 Prepare Requisition
 Prepare Request for Quote
 Needs Review
 Evaluate RFQ
 Approves
 Review RFQ
 Decides to Quote?
 Prepare Quote
 Review Quote
 Quote is Acceptable
 Prepare Order
 Review Order
 Order Acceptable
 Fulfill Order
 Receive Ordered Items
 Prepare Invoice
 Make Payment
 Receive Payment

 Prepare Requisition
 Prepare Request for Quote
 Needs Review
 Evaluate RFQ
 Approves
 Review RFQ
 Decides to Quote?
 Prepare Quote
 Review Quote
 Quote is Acceptable
 Prepare Order
 Review Order
 Order Not Acceptable

 Prepare Requisition
 Prepare Request for Quote
 Needs Review
 Evaluate RFQ
 Approves
 Review RFQ
 Decides to Quote?
 Prepare Quote
 Review Quote
 Quote is Not Acceptable
 Review Quote Response

 Prepare Requisition
 Prepare Request for Quote
 Needs Review
 Evaluate RFQ
 Does Not Approve

Conclusion

There are some neat things you can do when tools start getting “misused”. Thinking outside the box helps you find hidden ways to get more mileage out of what’s sitting around us. What other ways can you come up with to use Cucumber beyond just BDD?

 

 

 

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s