Doctoral Thesis: Visual GUI Testing: Automating High-Level Software Testing in Industrial Practice

This page presents an outline and (currently) the ongoing work of Emil Alégroth's Doctoral thesis. The thesis covers the subject of automated GUI based testing with the emerging technique Visual GUI Testing (VGT). For questions, please see contact information at the bottom of the page.

Defense: Completed on September 25, 2015
Location: House Jupiter at Campus Lindholmen in room Omega at 13.15.
Opponent:
Associate Professor Myra Cohen, University of Nebraska-Lincoln, USA.

Committee:
Professor Mika Mäntylä, University of Oulu, Finland.
Professor Ina Schieferdecker, Freie Universität Berlin, Germany.
Dr. Qing Xie, Accenture, USA.

Page last update: 2015-09-30

  • Thesis: Visual GUI Testing: Automating High-Level Software Testing in Industrial Practice
  • Last update: 2015-09-30, Final version.

    Abstract

    Software Engineering is at the verge of a new era where continuous releases are becoming more common than planned long-term projects. In this context, test automation will become essential on all levels of system abstraction to meet the market's demands of speedy delivery and high software quality. Hence, automated tests are required from low-level software components, tested with unit tests, up to the pictorial graphical user interface (GUI), tested with user emulated system and acceptance tests. Thus far, research has provided industry with a plethora of automation solutions for lower level testing but GUI level testing is still primarily a manual, and therefore costly and tedious, activity in practice.

    We have identified three generations of GUI-based automated testing. The first generation relied on recorded, exact, GUI coordinates but was soon discarded due to unfeasible maintenance costs caused by fragility to GUI change. Second generation tools, also referred to as component-, widget- or tag-based tools instead operate against some model of the GUI by hooking into the system's GUI architecture, libraries or application programming interfaces (APIs). Whilst tools of this approach are extensively used in practice, and are suitable to test system behavior, they are inflexible by being restricted to specific GUI technologies, programming languages and platforms.

    The third generation, referred to as Visual GUI Testing (VGT), is an emerging technique in industrial practice with properties that mitigate the challenges experienced with previous generations of GUI testing. VGT is defined as a tool driven automated test technique where image recognition is used to interact with, and assert, a system's behavior through its pictorial GUI as it is shown to the user in user-emulated system or acceptance tests. VGT thereby provides test results of quality on par with a human tester and is therefore an effective complement to reduce also the aforementioned challenges of manual testing in practice. However, despite these benefits, the technique is only sparsely used in industrial practice and the academic body of knowledge holds only limited empirical support for the technique's industrial viability.

    This thesis presents a holistic evaluation of VGT's capabilities in industrial practice, obtained through a series of case studies and experiments performed in academia and Swedish industry. The research follows an incremental methodology that began with academic experimentation with VGT tools, followed by sstudies in industrial practice that was concluded with a case study in a company that had used VGT for several years. Results of the research show that VGT is a viable technique for use in industrial practice with better defect finding ability than manual tests, ability to test any GUI based system, high learnability, feasible maintenance costs and both short and long-term company benefits. However, there are still challenges, problems and limitations that must be considered for the successful adoption, use and longevity of VGT in a company, the most crucial being that suitable development and maintenance activities are used. This research thereby concludes that software industry can use VGT but also provides a stepping stone for academia to explore new test solutions that build on image recognition technology.

    Thesis outline

    Figure 1 visualizes the outline of the thesis chapters and summarizes the content of each chapter. Figure 2, in turn, presents a roadmap overview of how the papers included in the thesis connect to each other.

    structure
    Figure 1: Thesis layout

    structure
    Figure 2: Thesis roadmap

    Title: Visual GUI Testing in Industrial Practice (TT)
    Overall Aim: Evaluate Visual GUI Testing from a holistic perspective to identify the technique's applicability and feasibility for automated software system and acceptance testing in industrial practice .
    Thesis Goals: Present support for, but also challenges, problems and limitations with, adopting, using and maintaining Visual GUI Testing for automated system and acceptance testing in industry from a holistic perspective.

    Tentative thesis structure and included papers:

    Papers not included in the thesis

    Time plan

    Below you will find a tentative time-plan up to the PhD thesis defense.

    • Thesis webpage and administrative documentation complete: 2014-12-05
    • Paper P4 submitted to Journal of Systems and Software (JSS): 2014-12-15
    • Paper P5 submitted to Information and Software Technology Journal (IST): 2015-02-15
    • Draft 1 of thesis done: 2015-05-03
    • Draft 2 of thesis done: 2015-05-10
    • Draft 3 of thesis done: 2015-05-17
    • Draft 4 of thesis done: 2015-05-25
    • Draft 5 of thesis done: 2015-05-31
    • Draft 6 of thesis done: 2015-06-15
    • Final Draft of thesis done: 2015-06-14
    • Thesis defense: 2015-09-25

    Contact

    Emil Alégroth
    E-mail: alegroth (at) chalmers (dot) se

    Robert Feldt (Main supervisor)
    E-mail: robert (dot) feldt (at) bth (dot) se

    Helena Holmström-Olsson (Second supervisor)
    E-mail: helena (dot) holmstrom (dot) olsson (at) mah (dot) se