Project Assignment¶

Introduction¶

You’ll take the role of a Quality Assurance team for a software project. Your team will have to do the following:

• build/install the project
• do some exploratory testing
• write some test script
• model (part of) the software
• and finally, generate more tests from your model and write an adapter to execute those tests on the SUT

This mini-project is split into different assignments. You’ll have to submit the result of each assignment to the [fire] system. All submissions follow a common scheme. For each, you will have to submit the artifacts developed in the assignment (test cases, scripts, etc.) and additionally a report describing these artifacts. Check the assignment descriptions for details. In addition, one assignment require you to present your findings to the class. The presentation doesn’t have to be submitted.

The remainder of this page first lists important dates and deadlines regarding the assignments. This is followed by descriptions of each assignment in sections. You can access the sections via the menu to the left, as well.

Project Description¶

The project that we will be working with this year is IntelliJ-IDEA. It is an open-source integrated development environment (IDE) written in Java. Throughout the course this program will serve as our System Under Test (SUT). We will split up functionalities among the groups. So each group is going to test specific parts of the IDE’s source code and the respective functionality.

The source code can be found here.

The main source of documentation can be found here.

Important dates¶

• Deadline for Forming groups: Tuesday March 22
• Deadline for Assignment 1: Exploratory and Automated Testing: Friday April 15
• Deadline for Assignment 2: Modeling: Friday May 6
• Deadline for Assignment 3: Test generation and execution: Friday May 20
• Presentation Presentation: Final Presentation: Wednesday May 25
• Deadline for Final Report: Thursday May 26

Forming groups¶

First, you need to form groups of 3 students. If you cannot find other people to join, ask on the [forum]. If this doesn’t work either, contact the lab assistant. Groups cannot be more than 3 people and groups of less than 3 people will only be accepted under exceptional circumstances (contact the lab assistant if you think you qualify).

Distribution of the System Under Test (SUT)¶

On Wednesday March 23, we will provide each group with a specific SUT through the Fire system.

We will provide you with a detailed system description of the system when the course starts.

It is not decided yet, if the SUTs will be distributed by the course responsible or if a poll will be published in order for the student groups to choose.

Assignment 1: Exploratory and Automated Testing¶

Part 1:

Ensure that you are able to build the SUT from the source code and install it on your machines. Each member of your group has to get familiar with the project.

Part 2:

Start doing exploratory testing. That is

“a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

Test as many functions as you can and try to come up with difficult or unusual use cases that the software might not handle well. Read the documentation and check that the software behave as described.

Take some notes:

• was the SUT easy to install?
• is the documentation up to date?
• is the documentation complete?
• did you find some corner cases that aren’t properly handled?

Part 3:

Write 5 test cases as JUnit scripts against your SUT. Your test cases should test different functionalities of the SUT and be able to run without manual intervention. In particular, they should be able to decide automatically whether the test is a success or a failure.

You’ll need to find a way for your tests to control your SUT.

Part 4:

Inject 5 faults into the existing system. A simple example would be changing the instruction var = var + 1 into var = var - 1 or if(var > 5) into if(var < 5). Be creative!

The injected faults should make the previously created automated tests fail. Take notes on what you changed and how it influenced your tests.

Redistribution of the SUT¶

On Wednesday April 20, we will provide each group with a specific SUT.

We will provide you with a detailed system description of the system when the course starts.

Assignment 2: Modeling¶

In this assignment, you will create a model of your SUT that you will then use in the last assignment to generate tests. Your model should be an extended finite state machine that will allow you to generate interesting, arbitrary long tests sequences against your software under test. Your EFSM must have about 20 inputs, at least one internal variable and be able to generate infinite test sequences.

Your model doesn’t need to cover the whole SUT, in particular it has many functionalities, you can model only part of them. Start by modeling the most basic functionalities and add more details progressively until you reach the desired number of inputs.

Assignment 3: Test generation and execution¶

In this assignment, you will implement the model that you developed in Assignment 2: Modeling using ModelJUnit.

Start by implementing the model alone by implementing the interface FsmModel, then connect each action in your model to your SUT in a separate adapter class (See the ModelJUnit tutorial for an example).

ModelJUnit offers several testing strategies and coverage metrics. Read more about them in the ModelJUnit documentation (Testing strategies: AllRoundTester, GreedyTester, LookaheadTester, RandomTester ; Coverage metrics: ActionCoverage, StateCoverage, TransitionCoverage, TransitionPairCoverage).

Before running the tests generator, try to think about which strategy will work best? Which metric do you think is the most interesting to evaluate your model?

Generate tests using your model with the different testers available in ModelJUnit. Collect coverage metrics about the generated tests, report the collected value a table like the following.

Presentation: Final Presentation¶

Information regarding the final presentation will be added when the course starts.

The presentations are on Wednesday May 25. Attendance is mandatory. To pass the labs, you need to take part in the presentation session. If you are not able to come, you need to notify us at least 24 hours beforehand.

Each group must give an 8 minute presentation, reporting on your results from Assignment 2 and 3. After each presentation, there is some time for the audience to ask questions.

As a guideline, you can structure the presentationo as:

• 1 slide title-page, introductions
• 1-2 slide describing your part of the SUT
• 1-2 slide presenting what you tested (including your EFSM)
• 2 slides reflections and findings

The presentation session will start at 10:15, both slots before and after lunch are mandatory. After the presentation session is finished there are opportunities for further consultation regarding the final report.

Final Report¶

The final report is intended as a high-level document, summarizing your work in the assignments. In a normal testing process, a report is the usual outcome. Therefore, when writing the report, imagine that you are reporting to the manager of the fictional QA team that you are a part of.

Useful links¶