iOS Testing Recipe: A bit of Cucumber and some Gherkin

10 March 2014

Written by

Russ Hill

We all know there’s a load of smartphone hardware out there and that testing our apps in the real world is therefore hard. But there are steps you can take that put you on the road to Continuous Integration that will make your life easier…

Test, Tester, Testing

As a mobile developer, you build apps based on other peoples’ designs all the time. They are their idea, but you’re the person responsible if the app has bugs - and quite right too! So what can you do to make things easier on yourself and better for the application? Well, you can test! You’ve used Continuous Integration (CI) in web development before, but never for mobile right? Maybe you’ve already setup tests in XCode that exercise your applications back-end services, but what about the UI interactions, how can we deal with that?

You can use the following crunchy ingredients mixed with your own testing recipes and sit back while your apps UI is tested, here’s how.

Toss the ingredients together carefully

You’ll need the following ingredients:

a touch of Ruby
a soupçon of Calabash-ios
some chopped Cucumber
some freshly grated Gherkin

A touch of Ruby

You’ll need Ruby 2.0+ preferably, but anything over 1.8.7 should be okay. If you use Mac OS X for development as we do, you may find the easiest way to install Ruby is via RVM (Ruby Version Manager):

\curl -sSL https://get.rvm.io | bash -s stable --ruby

Once installation completes, double check by executing:

ruby -v

You should see something like this:

ruby 2.1.1p76 (2014-02-24 revision 45161) [x86_64-darwin12.0]

If so, it’s time to move on to the rest of the ingredients.

A soupçon of Calabash-ios, some chopped Cucumber & some freshly grated Gherkin

Calabash provides the automated testing framework via which the tests you write will be executed. Cucumber is a well-known tool that executes plain-text functional descriptions as automated tests which are written in a language called Gherkin.

Full instructions for getting it installed can be found on the projects Github page. We used the “Fast track” option detailed there for getting Calabash, Cucumber and Gherkin installed and configured for a specific iOS project. Essentially it copies one of your build schemes and sets some additional options such as ensuring the Calabash framework is included.

It also creates a folder called features and some additional files and folders within, the most important being one called my_first.feature.

You should absolutely read the excellent documentation for Calabash which is all on their Github page, starting with (unsurprisingly) the Getting started guide

Write down the recipe

After you have things installed and working it’s time to write some tests. Now this is where it gets really cool: because the language used is so descriptive (and also non-technical) once you’ve added the subtle nuances you need for your specific testing process, it should in theory be possible for non-developers to write tests. This could be someone on the test team, or even the project manager or the product manager - people who know the product well and what it should do.

What you need to remember is that you are creating a feature which has a description of the feature being tested as well as one or more scenarios. These are specific steps that describe what can be done for that specific scenario.

Example .feature file:

Feature: Running a test
    As a Talis Network user
    I want to have access to my resources
    So I can take my content with me and consume it

Scenario: User login
    # start by logging the user in
    Given I am on the Login Screen
    Then I wait for table cell titled "Talis" to appear
    Then I touch table cell titled "Talis"  
    Then I touch button titled "Login"
    Then I wait for web element "Email" to appear
    Then I login
    Then I wait for navigation bar "Talis" to appear
    Then I wait for table cell titled "Emotions Video 3" to appear
    Then take picture

As you can see, you write your tests in the form of a story which works very well when you and your team work in an Agile way - Scrum or Extreme Programming - because the input to those processes are user stories!

Mind your language

There are a whole plethora of built in steps that you can include in your feature file _scenarios_, these are defined in the predefined steps documentation. They cover a variety of things, but it’s almost certain you will want to create your own to handle your specific needs.

This is where the my_first_steps.rb file comes in.

As an example, a screen in one of our mobile apps actually uses a web-based authentication method - in this case, Google - and in order to make sure that the web page had fully loaded, we created a specific step:

Then(/^I wait for web element "(.*?)" to appear$/) do |htmlField|
    wait_for_elements_exist(["webView css:'input##{htmlField}'"], :timeout => 5)
    sleep(STEP_PAUSE)
end

As with web-based testing - Selenium and so on - you want to try and avoid inserting wait methods with arbitrary values. These are so often foiled by slow network or other hardware that you can’t really rely on them. It’s far better to add a step that waits for a specific thing - whether that be a specific navigation bar title, or an element on the page. This way you’re far less likely to fall foul of false-negative testing failures where the problem was a fixed wait method was simply not long enough - resulting in the test failing.

Another reason to avoid specific wait methods is that taken cumulatively, they can really start to slow down your tests!

Note: one thing to note about the scenarios you create in your .feature file is that the iOS app is terminated between each of them. Take this example:

Scenario: User login
    # start by logging the user in
    Given I am on the Login Screen
    Then I wait for table cell titled "Talis" to appear
    Then I touch table cell titled "Talis"  
    Then I touch button titled "Login"
    Then I wait for web element "Email" to appear
    Then I login
    Then I wait for navigation bar "Talis" to appear
    Then I wait for table cell titled "Emotions Video 3" to appear
    Then take picture

Scenario: Download resource
    # we should be on the resources list screen
    Given I am on the Resources Screen
    Then I wait for navigation bar "Talis" to appear
    Then I wait for table cell titled "Emotions Video 3" to appear
    Then I touch table cell titled "Emotions Video 3"
    Then I wait for table cell titled "View resource" to appear

    # start resource download and allow it to complete
    Then I touch the offline switch
    Then I wait for download to complete
    Then I wait for 10 seconds
    Then take picture

Between the User login and Download resource scenarios the app will be stopped. In our case, this was not too much of an issue as we persist the user login, so when the app restarted in order to execute the Download resource scenario, the user was already logged in and the resources list screen is what is already being presented.

It’s definitely something you need to be aware of though.

Run the recipe

You can run your tests using the simulator or “real” devices, for now, we’re using the simulator. In order to execute the feature tests using the iOS simulator, go to the terminal, make sure you’re in your iOS project folder and type:

cucumber

By default Cucumber will look for a features folder and then within that, a file that has a .feature extension.

It will then load that file and begin executing the scenarios you have defined in it:

Successful step executions are shown in green, but should any of the steps fail, you’ll see them output in red:

It’s certainly fascinating to see your app being run and controlled by “ghostly” forces in front of your very eyes!

On a more serious note, this is of course an excellent way of performing regression testing as new features are added to the app during development. If you also operate a CI strategy for mobile development, the tests you’ve created can be executed as soon as your CI server has pulled the latest code and built it.

Washing the dishes (i.e. “the wrap up”)

We’ve covered a basic introduction to using Calabash-iOS here, so hopefully you’ll take the ideas and run with them - as we plan to do - and use them to evolve your CI strategy for mobile app development.

Have fun creating some great-tasting testing recipes!

Comments on HN