This page presents an outline and (currently) the ongoing work of Emil Alégroth's Doctoral thesis. The thesis covers the subject of automated GUI based testing with the emerging technique Visual GUI Testing (VGT). For questions, please see contact information at the bottom of the page.
Defense: Completed on September 25, 2015
Location: House Jupiter at Campus Lindholmen in room Omega at 13.15.
Associate Professor Myra Cohen, University of Nebraska-Lincoln, USA.
Professor Mika Mäntylä, University of Oulu, Finland.
Professor Ina Schieferdecker, Freie Universität Berlin, Germany.
Dr. Qing Xie, Accenture, USA.
Page last update: 2015-09-30
Thesis: Visual GUI Testing: Automating High-Level Software Testing in Industrial Practice
Last update: 2015-09-30, Final version.
Software Engineering is at the verge of a new era where continuous releases are becoming more common than planned long-term projects.
In this context, test automation will become essential on all levels of system abstraction to meet the market's demands of speedy delivery and high software quality.
Hence, automated tests are required from low-level software components, tested with unit tests, up to the pictorial graphical user interface (GUI), tested with user emulated system and acceptance tests.
Thus far, research has provided industry with a plethora of automation solutions for lower level testing but GUI level testing is still primarily a manual, and therefore costly and tedious, activity in practice.
We have identified three generations of GUI-based automated testing.
The first generation relied on recorded, exact, GUI coordinates but was soon discarded due to unfeasible maintenance costs caused by fragility to GUI change.
Second generation tools, also referred to as component-, widget- or tag-based tools instead operate against some model of the GUI by hooking into the system's GUI architecture, libraries or application programming interfaces (APIs).
Whilst tools of this approach are extensively used in practice, and are suitable to test system behavior, they are inflexible by being restricted to specific GUI technologies, programming languages and platforms.
The third generation, referred to as Visual GUI Testing (VGT), is an emerging technique in industrial practice with properties that mitigate the challenges experienced with previous generations of GUI testing.
VGT is defined as a tool driven automated test technique where image recognition is used to interact with, and assert, a system's behavior through its pictorial GUI as it is shown to the user in user-emulated system or acceptance tests.
VGT thereby provides test results of quality on par with a human tester and is therefore an effective complement to reduce also the aforementioned challenges of manual testing in practice.
However, despite these benefits, the technique is only sparsely used in industrial practice and the academic body of knowledge holds only limited empirical support for the technique's industrial viability.
This thesis presents a holistic evaluation of VGT's capabilities in industrial practice, obtained through a series of case studies and experiments performed in academia and Swedish industry.
The research follows an incremental methodology that began with academic experimentation with VGT tools, followed by sstudies in industrial practice that was concluded with a case study in a company that had used VGT for several years.
Results of the research show that VGT is a viable technique for use in industrial practice with better defect finding ability than manual tests, ability to test any GUI based system, high learnability, feasible maintenance costs and both short and long-term company benefits.
However, there are still challenges, problems and limitations that must be considered for the successful adoption, use and longevity of VGT in a company, the most crucial being that suitable development and maintenance activities are used.
This research thereby concludes that software industry can use VGT but also provides a stepping stone for academia to explore new test solutions that build on image recognition technology.
Figure 1 visualizes the outline of the thesis chapters and summarizes the content of each chapter. Figure 2, in turn, presents a roadmap overview of how the papers included in the thesis connect to each other.
Figure 1: Thesis layout
Figure 2: Thesis roadmap
Visual GUI Testing in Industrial Practice (TT)
Evaluate Visual GUI Testing from a holistic perspective to identify the technique's applicability and feasibility for automated software system and acceptance testing in industrial practice .
Present support for, but also challenges, problems and limitations with, adopting, using and maintaining Visual GUI Testing for automated system and acceptance testing in industry from a holistic perspective.
Tentative thesis structure and included papers:
- C1. Introduction
- C2. Automated System Testing using Visual GUI Testing Tools: A Comparative Study in Industry,
International Conference on Software Testing (ICST), 2012,
- C3. Transitioning Manual System Test Suites to Automated Testing: An Industrial Case Study,
International Conference on Software Testing (ICST), 2013, PDF
- C4. Visual GUI Testing in Practice: Challenges, Problems and Limitations,
Empirical Software Engineering Journal (EMSE), 2014, PDF
- C5. Maintenance of Automated Testing in Industry: An Empirical study on Visual GUI Testing,
In submission, 2015, PDF (TBA)
- C6. On the Long-term Use of Visual GUI Testing in Industrial Practice: A Case Study (TT),
In submission, June 2015, PDF (TBA)
- C7. Conceptualization and Evaluation of Component-based Testing Unified with Visual GUI Testing: An Empirical Study,
International Conference on Software Testing (ICST), 2015, PDF (TBA)
- C8. Automated Defect Replication with Visual GUI Testing: An Success-story from Industry (TT),
Accepted for publication in the IEEE Software Journal, 2015, PDF (TBA)
- NIP1. Structuring Case Studies to Cover Multiple Perspectives,
Software Engineering and Knowledge Engineering (SEKE) conference, 2011.
- NIP2. JAutomate: A Tool for System- and Acceptance-test Automation,
International Conference on Software Testing (ICST), 2013.
- NIP3. Random Visual GUI Testing: Proof of Concept,
Software Engineering and Knowledge Engineering (SEKE) conference, 2013.
- NIP4. State-of-Practice of GUI-based Acceptance and System Testing - An Industrial Multiple-Case Study,
Software Engineering and Advanced Applications (SEAA), 2013.
- NIP5. On the Industrial Applicability of TextTest: An Empirical Case Study,
International Conference on Software Testing (ICST), 2015.
- NIP6. Definition and Evaluation of Mutation Operators for GUI-level Mutation Analysis,
International Conference on Software Testing (ICST), 2015.
Below you will find a tentative time-plan up to the PhD thesis defense.
- Thesis webpage and administrative documentation complete: 2014-12-05
- Paper P4 submitted to Journal of Systems and Software (JSS): 2014-12-15
- Paper P5 submitted to Information and Software Technology Journal (IST): 2015-02-15
Draft 1 of thesis done: 2015-05-03
Draft 2 of thesis done: 2015-05-10
Draft 3 of thesis done: 2015-05-17
Draft 4 of thesis done: 2015-05-25
Draft 5 of thesis done: 2015-05-31
Draft 6 of thesis done: 2015-06-15
Final Draft of thesis done: 2015-06-14
- Thesis defense: 2015-09-25
E-mail: alegroth (at) chalmers (dot) se
Robert Feldt (Main supervisor)
E-mail: robert (dot) feldt (at) bth (dot) se
Helena Holmström-Olsson (Second supervisor)
E-mail: helena (dot) holmstrom (dot) olsson (at) mah (dot) se