Visual GUI testing

The market demands for faster time-to-market delivery and higher quality software are becoming greater challenges for many software development companies. These companies try to mitigate these challenges through automated testing, i.e. unit testing. However, these tests are on a low level of system abstraction and an ongoing debate is if these techniques can work for higher level tests, i.e. System and Acceptance tests.

Our research aims to present, evaluate and verify the applicability of a "new" automated test technique that we call Visual GUI testing. Visual GUI testing is a tool supported technique that through scenario based scripts and image recognition allows for automation of manual test cases that were previously not possible to automate. Visual GUI testing interacts with the System Under Test (SUT) through its GUI bitmap components, i.e. images and button on the GUI. Hence, tests in this technique are conducted in the same way as a tester, or even user, would interact with the system. A typical test consists of providing the SUT with stimuli through GUI interaction followed by analysis of the GUI for some expected output. If the expected output is observed, through image recognition, the test would pass, otherwise it would fail.

The test technique views the SUT as a blackbox, i.e. does not require any deeper knowledge about the system to interact with it. Hence, the technique works for any type of Graphical User Interface (GUI) based system, including systems that are distributed over several computers, GUIs with custom GUI components and more!

script
Sikuli script example.

There are several tools for visual GUI testing but in our current research we are focusing on the open-source tool Sikuli. Our preliminary results of using this tool at Saab AB show that the technique has high applicability and potential within the industrial context. Through our research we have seen execution time improvements of more than 5 times compared to manual test case execution, possible quality gains through more frequent system test execution, applicability of the technique on almost all types of manual tests with reasonable script development costs.

Current research into this technique includes the development of a robustness framework for Visual GUI script development as well as evaluation of the maintenance costs of Visual GUI testing script suites that are currently unknown.

In summary we have found that Visual GUI testing is a more robust test technique than most other high-level test techniques with high cost and quality improvement potential compared to manual system testing. The technique has high perceived broad applicability and has been shown to work within the industrial context. Hence, Visual GUI testing is an important and valuable test technique for the future!