nz.ac.waikato.modeljunit
Class ModelTestCase

java.lang.Object
  nz.ac.waikato.modeljunit.ModelTestCase

Deprecated. Use one of the subclasses of Tester instead.

public class ModelTestCase
extends java.lang.Object

Test a system, based on a finite state machine (FSM) model of that system.

This class provides several methods that use model-based testing techniques to automatically generate test suites for a system under test (SUT) from an FSM model of that system. To use these methods, you write a special FSM class (see FsmModel) that models part of the behaviour of your SUT, then pass an instance of that class to one of the test generation methods (eg. randomWalk(int)). It will analyse the structure of your FSM model, then call various sequences of methods in your model to ensure that it is well tested. Each action method of your FSM can change the state of the FSM, and can also perform some tests on the SUT and change its state. So your FSM class is actually performing two roles: (1) Model: defining a simplified FSM view of the behaviour of your SUT; (2) Adaptor: mapping each transition of that FSM to a concrete test of your SUT.

TODO:

• DONE: record coverage statistics and make them accessible via an API.
• DONE: separate out the MBT traversal algorithms, the model manager and the coverage metrics into separate classes and make the methods non-static.
• DONE: build the graph *during* the random walk traversal.
• DONE: when a test fails, throw a specific exception containing lots of model information.
• DONE: make reset probability and buildGraph timeout user-settable.
• add more test generation algorithms, such as greedy random.
• add support for Actions with parameters.
• extend doRandomAction(_) to take into account a state-based probability for each Action. (i.e., Markov chains).

Acknowledgements: This model-based testing library uses the JDSL (Java Data Structure Library, see http://www.jdsl.org) graph libraries to store and traverse the graph of the FSM.

Field Summary

static double	DEFAULT_RESET_PROBABILITY Deprecated. During random walk (including buildGraph), this is the default probability of doing reset() rather than choosing a random transition.
static long	FIXEDSEED Deprecated.

Constructor Summary

ModelTestCase(FsmModel model)
Deprecated. Constructs an FSM model from the given FsmModel object.

Method Summary

static void	addCoverageMetric(CoverageMetric cover) Deprecated. Add a coverage listener.
void	allRoundTrips(int length) Deprecated. Same as allRoundTrips(length), but uses a fixed seed for the random number generator.
void	allRoundTrips(int length, java.util.Random rand) Deprecated. Uses a greedy random walk to try and test all loops in the model.
boolean	buildGraph() Deprecated. Equivalent to buildGraph(new Random(FIXEDSEED)).
boolean	buildGraph(java.util.Random rand) Deprecated. Equivalent to buildGraph(new Random(FIXEDSEED), 10000).
boolean	buildGraph(java.util.Random rand, int maxTransitions) Deprecated. Finish building the graph.
protected void	continueBuildGraph(Transition tr, int action) Deprecated. Records a transition in the graph, if it is not already there.
java.util.BitSet	currentEnabledActions() Deprecated. Return the bitset of all actions that are enabled in the current state.
protected boolean	doAction(int index) Deprecated. Try to take the given Action from the current state.
protected int	doGreedyRandomAction(java.util.Random rand) Deprecated.
int	doGreedyRandomActionOrReset(java.util.Random rand, boolean testing) Deprecated.
protected int	doRandomAction(java.util.Random rand) Deprecated. Take any randomly-chosen Action that is enabled.
int	doRandomActionOrReset(java.util.Random rand, boolean testing) Deprecated. Randomly take an enabled transition, or do a reset with a certain probability (see getResetProbability()).
void	doReset(boolean testing) Deprecated. Reset the FSM to its initial state.
protected void	doReset(java.lang.String reason, boolean testing) Deprecated. Reset the FSM to its initial state.
float	enabled(int index) Deprecated. Is Action number 'index' enabled? Returns 0.0 if Action number 'index' is disabled, or a positive number if it is enabled.
protected int	fsmFindAction(java.lang.String name) Deprecated. Looks up an Action by name and returns its number.
protected java.lang.Object	getActionName(int index) Deprecated. Returns the name of the given Action.
java.util.List<CoverageMetric>	getCoverageMetrics() Deprecated. A list of all the current coverage listeners
protected java.lang.Object	getCurrentState() Deprecated. Returns the current state of the implementation under test.
int	getFailedTests() Deprecated. Returns the number of test failures.
int	getFailureVerbosity() Deprecated. The amount of information printed when a test fails.
InspectableGraph	getGraph() Deprecated. Returns the graph of the FSM model.
protected java.lang.Object	getModel() Deprecated. Returns the model object that is begin tested.
protected java.lang.Class	getModelClass() Deprecated. Returns the FSM class that is the test model.
protected java.lang.String	getModelName() Deprecated. Returns the name of the FSM class that is the test model.
protected int	getNumActions() Deprecated. The total number of Actions.
double	getResetProbability() Deprecated. The probability of spontaneously doing a reset rather than a normal transition during random walks etc.
int	getVerbosity() Deprecated. Says how many progress messages will be printed.
Vertex	getVertex(java.lang.Object state) Deprecated. Maps a state to a vertex object of the FSM graph.
void	greedyRandomWalk(int length) Deprecated.
void	greedyRandomWalk(int length, java.util.Random rand) Deprecated.
protected void	loadModelClass(java.lang.Class fsm) Deprecated. Loads the given class and finds its @Action methods.
void	printFailure(int priority, java.lang.String msg) Deprecated. Print failure messages, when a test/tests fail.
void	printGraphDot(java.lang.String filename) Deprecated. Saves the FSM graph into the given file, in DOT format.
void	printProgress(int priority, java.lang.String msg) Deprecated. Print progress messages, during FSM-based testing.
void	printWarning(java.lang.String msg) Deprecated. Print a warning, during analysis of the FSM class.
void	randomWalk(int length) Deprecated. Calls fsmRandomWalk/3 with FIXEDSEED so that tests are repeatable.
void	randomWalk(int length, java.util.Random rand) Deprecated. Does a random walk through a finite state machine.
static void	removeAllCoverageMetrics() Deprecated. Remove all coverage listeners.
static boolean	removeCoverageMetric(CoverageMetric cover) Deprecated. Remove a coverage listener.
static void	resetCoverageMetrics() Deprecated. Reset all the coverage statistics.
void	setFailureVerbosity(int failureVerbosity) Deprecated. Sets the amount of information printed when tests fail.
void	setResetProbability(double prob) Deprecated. Set the probability of doing a reset during random walks.
void	setVerbosity(int verbosity) Deprecated. Sets the level of progress messages that will be printed as this class builds the FSM graph and generates tests.
protected void	startBuildGraph() Deprecated. Starts to build the FSM graph by exploring the fsm object.
static java.lang.String	stateName(java.lang.Object state) Deprecated. Converts a state into a name.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_RESET_PROBABILITY

public static final double DEFAULT_RESET_PROBABILITY

Deprecated.

During random walk (including buildGraph), this is the default probability of doing reset() rather than choosing a random transition.

See Also:

Constant Field Values

FIXEDSEED

public static final long FIXEDSEED

Deprecated.

See Also:

Constant Field Values

Constructor Detail

ModelTestCase

public ModelTestCase(FsmModel model)

Deprecated.

Constructs an FSM model from the given FsmModel object.

Parameters:

model -

Method Detail

fsmFindAction

protected int fsmFindAction(java.lang.String name)

Deprecated.

Looks up an Action by name and returns its number.

Parameters:

name - The name of an Action.

Returns:

The index within fsmActions/fsmGuards, else -1.

getFailedTests

public int getFailedTests()

Deprecated.

Returns the number of test failures.

getResetProbability

public double getResetProbability()

Deprecated.

The probability of spontaneously doing a reset rather than a normal transition during random walks etc.

Returns:

the reset probability

setResetProbability

public void setResetProbability(double prob)

Deprecated.

Set the probability of doing a reset during random walks. Note that the average length of each test sequence will be roughly proportional to the inverse of this probability.

If this is set to 0.0, then resets will only be done when we reach a dead-end state (no enabled actions). This means that if the FSM contains a loop that does not have a path back to the initial state, then the random walks may get caught in that loop forever. For this reason, a non-zero probability is recommended.

The default probability is DEFAULT_RESET_PROBABILITY.

Parameters:

prob - Must be at least 0.0 and less than 1.0.

getVerbosity

public int getVerbosity()

Deprecated.

Says how many progress messages will be printed.

setVerbosity

public void setVerbosity(int verbosity)

Deprecated.

Sets the level of progress messages that will be printed as this class builds the FSM graph and generates tests. 0 means no messages. 1 means statistical summaries only. 2 means also show each transition taken. 3 means also show progress messages as the FSM graph is built.

Parameters:

verbosity - 0..3

getFailureVerbosity

public int getFailureVerbosity()

Deprecated.

The amount of information printed when a test fails.

setFailureVerbosity

public void setFailureVerbosity(int failureVerbosity)

Deprecated.

Sets the amount of information printed when tests fail. 0 means none. 1 means print a statistical summary of the number of failed tests. 2 means print a one line summary of each failed test. 3 means print more information about each failed test.

Parameters:

failureVerbosity - 0..3

getModelClass

protected java.lang.Class getModelClass()

Deprecated.

Returns the FSM class that is the test model.

getModelName

protected java.lang.String getModelName()

Deprecated.

Returns the name of the FSM class that is the test model.

getGraph

public InspectableGraph getGraph()

Deprecated.

Returns the graph of the FSM model. Note that the graph may be incomplete (call buildGraph to explore the graph thoroughly).

getVertex

public Vertex getVertex(java.lang.Object state)

Deprecated.

Maps a state to a vertex object of the FSM graph. This may return null until after buildGraph has been called.

getModel

protected java.lang.Object getModel()

Deprecated.

Returns the model object that is begin tested.

getCurrentState

protected java.lang.Object getCurrentState()

Deprecated.

Returns the current state of the implementation under test.

getActionName

protected java.lang.Object getActionName(int index)

Deprecated.

Returns the name of the given Action.

getNumActions

protected int getNumActions()

Deprecated.

The total number of Actions.

printWarning

public void printWarning(java.lang.String msg)

Deprecated.

Print a warning, during analysis of the FSM class. By default, this prints warnings to System.out. Subclasses can override this if they to do something different.

printProgress

public void printProgress(int priority,
                          java.lang.String msg)

Deprecated.

Print progress messages, during FSM-based testing. This prints progress messages during the FSM analysis stages and during the actual testing. By default, this prints messages to System.out. Subclasses can override this if they want to do something different.

Parameters:

priority - 1..3

printFailure

public void printFailure(int priority,
                         java.lang.String msg)

Deprecated.

Print failure messages, when a test/tests fail. By default, this prints messages to System.out. Subclasses can override this if they to do something different.

Parameters:

priority - 1..3

resetCoverageMetrics

public static void resetCoverageMetrics()

Deprecated.

Reset all the coverage statistics. This can be called at any time, provided that an FSM model class has been loaded (that is, fsmGetClass() != null).

addCoverageMetric

public static void addCoverageMetric(CoverageMetric cover)

Deprecated.

Add a coverage listener.

removeCoverageMetric

public static boolean removeCoverageMetric(CoverageMetric cover)

Deprecated.

Remove a coverage listener.

removeAllCoverageMetrics

public static void removeAllCoverageMetrics()

Deprecated.

Remove all coverage listeners.

getCoverageMetrics

public java.util.List<CoverageMetric> getCoverageMetrics()

Deprecated.

A list of all the current coverage listeners

loadModelClass

protected void loadModelClass(java.lang.Class fsm)

Deprecated.

Loads the given class and finds its @Action methods. This method must be called before any fsm traversals are done.

startBuildGraph

protected void startBuildGraph()

Deprecated.

Starts to build the FSM graph by exploring the fsm object. This assumes that the current fsmState is the initial state. That is, a reset has just been done.

continueBuildGraph

protected void continueBuildGraph(Transition tr,
                                  int action)

Deprecated.

Records a transition in the graph, if it is not already there. This internal method should be called by doAction, so that it can analyze/record every transition. It is quite like a coverage listener, but I decided to make it more built-in because the graph is so fundamental to model-based testing.

Parameters:

tr - A possibly new transition (and state).

buildGraph

public boolean buildGraph(java.util.Random rand,
                          int maxTransitions)

Deprecated.

Finish building the graph. That is, keep exploring until all enabled actions in all reachable states that have explored. Of course, this may not have built ALL the graph if some guards are enabled sporadically in a given state, or if some transitions are non-deterministic.

Note that this method traverses the graph as it explores its shape, and this traversal is likely to increase any coverage statistics that are being recorded. For example, transition coverage will normally be 100% after this method returns. If you want to measure the coverage of a short traversal, you should call this method to explore the graph, then reset the coverage measures to zero before doing the traversal.

Parameters:

rand - A random generator to choose the exploration path.

Returns:

true if the graph seems to be completely built.

buildGraph

public boolean buildGraph(java.util.Random rand)

Deprecated.

Equivalent to buildGraph(new Random(FIXEDSEED), 10000).

buildGraph

public boolean buildGraph()

Deprecated.

Equivalent to buildGraph(new Random(FIXEDSEED)).

printGraphDot

public void printGraphDot(java.lang.String filename)
                   throws java.io.FileNotFoundException

Deprecated.

Saves the FSM graph into the given file, in DOT format. The DOT format can be converted into many other graphical formats, including xfig, postscript, jpeg etc. by using the 'dot' or 'neato' tools, which are freely available from http://www.graphviz.org. This method should only be called after buildGraph has built the graph.

Parameters:

filename - The filename should end with ".dot".

Throws:

java.io.FileNotFoundException

stateName

public static java.lang.String stateName(java.lang.Object state)

Deprecated.

Converts a state into a name. It calls toString on the state, and then adds quotes around the string if it is not a Java identifier.

Parameters:

state -

Returns:

A printable state name

doReset

public void doReset(boolean testing)

Deprecated.

Reset the FSM to its initial state. This is equivalent to doReset("User",testing), because it corresponds to a user-requested reset.

Parameters:

testing -

doReset

protected void doReset(java.lang.String reason,
                       boolean testing)

Deprecated.

Reset the FSM to its initial state. This does the fsmLoad of fsm.class if it has not already been done. It also calls the doneReset(testing) method of all the coverage listeners.

Parameters:

reason - Why the reset was performed (an adjective).

testing - False means we are just exploring the graph, so the fsm object could skip the actual tests if it wants.

enabled

public float enabled(int index)

Deprecated.

Is Action number 'index' enabled? Returns 0.0 if Action number 'index' is disabled, or a positive number if it is enabled. Missing guard methods return 1.0F, while boolean guard methods return 1.0F when true and 0.0F when false.

Parameters:

index - Index into the fsmActions array.

Returns:

The `enabledness' of this Action.

currentEnabledActions

public java.util.BitSet currentEnabledActions()

Deprecated.

Return the bitset of all actions that are enabled in the current state.

doAction

protected boolean doAction(int index)

Deprecated.

Try to take the given Action from the current state. Returns true if the Action was taken, false if it was disabled.

Parameters:

index - Index into the fsmTransitions array.

Returns:

True if taken, false if it is disabled.

doRandomAction

protected int doRandomAction(java.util.Random rand)

Deprecated.

Take any randomly-chosen Action that is enabled. Returns the number of the Action taken, -1 if all are disabled.

Parameters:

rand - The Random number generator that controls the choice.

Returns:

The Action taken, or -1 if none are enabled.

doRandomActionOrReset

public int doRandomActionOrReset(java.util.Random rand,
                                 boolean testing)

Deprecated.

Randomly take an enabled transition, or do a reset with a certain probability (see getResetProbability()).

Parameters:

rand - The Random number generator that controls the choice.

Returns:

The number of the transition taken, or -1 for a reset.

randomWalk

public void randomWalk(int length)

Deprecated.

Calls fsmRandomWalk/3 with FIXEDSEED so that tests are repeatable.

randomWalk

public void randomWalk(int length,
                       java.util.Random rand)

Deprecated.

Does a random walk through a finite state machine. It tests exactly 'length' transitions. If it has not finished testing, but gets into a state where there are no Actions enabled, then it will use the reset() method of the FSM to start from the initial state again.

If you want to test a different path each time this is called, then pass new Random() as the third parameter. If you want to test the same path each time (this makes the test results more predictable), then pass new Random(fixedSeed). (fsmRandomWalk(fsm,length) does this for you).

Parameters:

length - The number of transitions to test.

rand - A random number generator to control the traversal.

doGreedyRandomAction

protected int doGreedyRandomAction(java.util.Random rand)

Deprecated.

doGreedyRandomActionOrReset

public int doGreedyRandomActionOrReset(java.util.Random rand,
                                       boolean testing)

Deprecated.

greedyRandomWalk

public void greedyRandomWalk(int length)

Deprecated.

greedyRandomWalk

public void greedyRandomWalk(int length,
                             java.util.Random rand)

Deprecated.

allRoundTrips

public void allRoundTrips(int length)

Deprecated.

Same as allRoundTrips(length), but uses a fixed seed for the random number generator. This ensures repeatability. That is, the test results will be deterministic (if the SUT is).

Parameters:

length - The number of test steps to do.

allRoundTrips

public void allRoundTrips(int length,
                          java.util.Random rand)

Deprecated.

Uses a greedy random walk to try and test all loops in the model.

Parameters:

length - The number of test steps to do.

rand - The random number generator used to choose paths.