Test structure


By convention, tests are defined as individual files located in the tests subdirectory of the test repo and are expressed in YAML. YAML is a human-friendly data format which makes it very easy to create, read and modify the tests in any decent text editor and with minimal training. YAML is also source-control friendly, as opposed to XML, JSON, Excel or other formats. This is a vital requirement, especially with large teams. To learn more about the YAML syntax, read the YAML primer page.

A simple test example

The code snippet below is a Selenium test that performs three actions (test steps):

  1. Navigates to the GitHub homepage.

  2. Searches for the text "react" and presses ENTER.

  3. Validates that the React repo exists in the search results.

test-repo-path/tests/Go to React repo.yaml
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
description: Check the React repo on the GitHub website tags: [smoke-tests] actors: - actor: WEB segments: - segment: 1 actions: - description: Navigate to GitHub action: org.getopentest.selenium.NavigateTo args: url: https://github.com - description: Type "react" in the search box action: org.getopentest.selenium.SendKeys args: locator: { name: q } text: react sendEnter: true - description: Verify that the React repo is in the results action: org.getopentest.selenium.AssertElementVisible args: locator: { xpath: "//a[@href='/facebook/react']" }

The test actor section

Line 3 in the sample test (- actor: WEB) defines the start of a new test actor section and specifies the type of test actor that will execute the work defined in this section. It’s necessary to identify the actor type because complex projects might involve multiple types of test actors running simultaneously (e.g. a mobile actor, a web actor, an API actor, etc.). The test actor type is just a plain string whose value can be set to anything that is relevant for the application under test or the specific test automation project. Typically, it is set to an uppercase three-letter abbreviation like "WEB", "MOB" or "API". The only requirement is that you need to use the exact same string in the test file as was specified for the actorType parameter in the test actor configuration file.

The test segment section

Line 5 in the sample test (- segment: 1) defines the start of a new test segment. Test segments are executed sequentially, meaning that segment number 2 will only start executing after segment 1 has finished. This feature is only needed for distributed testing, to synchronize the work between multiple test actors running on different machines. However, typical test files only contain one single test segment, same as in our sample test.

The test actions section

The lines 7 through 22 from the sample test define the work that the test actor needs to do (the important stuff). This section consists of three blocks of text, each one of them representing one single test action that we want performed.

Let’s take a closer look at the first test action:

- description: Navigate to GitHub
  action: org.getopentest.selenium.NavigateTo
    url: https://github.com

The elements that make up this test action block are:

  • The - (dash) character. The dash is used to indicate that we are adding a new test action into the test flow. In the YAML format, the dash is used to add a new element to an array (the actions array, in our case).

  • The description property. This is an optional human-readable description of the test action that explains what the action is trying to accomplish. The action description is used in logs and test execution reports, so you should try to make it concise and meaningful.

  • The action property. The full name of the action (keyword) that we want to run. In this case, we are running the NavigateTo keyword from the org.getopentest.selenium package, which opens a browser window and navigates to the specified URL.

  • The args property. This is the list of arguments passed to the test action, whose value is a collection of name-value pairs. The NavigateTo action, in particular, takes one single argument named url, which points to the URL to navigate to.

A slightly more complex test

Here’s an example of a test that’s a little more involved, since it requires validating the response returned by an API call. Whenever you need to execute more advanced logic, like loops and some more complex verifications, you can always achieve this by embedding JavaScript code in the flow of the test:

description: Get repo branches from the GitHub API
tags: [smoke-tests]
  - actor: API
      - segment: 1
          - description: Perform the API call
            action: org.getopentest.actions.HttpRequest
              url: https://api.github.com/repos/facebook/react/branches
              verb: GET

          - description: Verify that the branches array exists and is not empty
            script: |
              var branches = $output.body;

              if (!branches || (branches.length == 0)) {
                  $fail("Response body invalid or contains no branch elements.");
              } else {
                    "The GitHub API returned {0} branches",

Final observations

  • The structure of the test definition file is controlled by indentation, as opposed to JSON, where the same goal is achieved using curly braces. If two properties start at the same column number, it means they belong to the same object. For example, it’s important that the description, action and args properties are perfectly aligned horizontally, otherwise you’ll end up with an invalid YAML file.

  • Indentation is always achieved using space characters and never TABs. With any decent text editor, the TAB key will automatically produce the right number of space characters, so this is not something that you have to keep track of yourself. We recommend setting the indent size at 2 spaces for the best compromise between readability and ease of use.