CITS5501 lab 3 (week 4) – ISP – solutions

Sample solutions:

This one test is not enough to be confident the method works. The method implementation ought to be extremely simple – just return age >= 18 – but if we want have high confidence that the method is implemented correctly, we should probably check, in addition to the “30” case:

2. The boundary at the minimum adult age (18)
3. One below that boundary (17)
4. Zero and/or or negative ages

Once those tests are in place, we still don’t know for sure that the method is correct in all cases – the only way of knowing that would be to test every single int value – but our confidence in it should be much higher. It’s possible that the implementer of the method has written it as

public static boolean isAdult(int age) {
  if (age == 999)
    return false;
  else
    return age >= 18;
}

But unless the implementer is trying to be deliberately unhelpful, that seems unlikely. More realistically, if there is a mistake, it would be

• A mis-typed comparator (e.g. <= or > instead of >=), or (less likely)
• An incorrect age (some number other than 18).

Given the sorts of mistake an implementer is likely to make, the four tests above (including the original test) should give us a high confidence the method works.

Beyond that, any additional tests reach a point of diminishing returns – they don’t increase our confidence much, and the extra tests constitute additional code that needs to be maintained. (You might like to think about: why don’t the additional tests increase our confidence? Then read the material in the textbook about Input Space Partitioning and see if that helps you formulate your answer.)

Sample solutions:

The line coverage will almost certainly not change – it will likely be exactly the same for both test suites.

As we’ve noted, the implementation is likely to be (assuming it is correctly written):

public static boolean isAdult(int age) {
  return age >= 18;
}

So there is only one line in the body – and all possible execution paths are short and go through the same single statement. Even with just the first test (age = 30), that line is executed. Adding more tests (e.g. boundary values) executes the same set of lines — so the line coverage number doesn’t increase. (It’s possible, though not likely, that a naive implementer might write the method using an if ... else statement – but even then, one “false” and one “true” test case suffice to exercise all the lines in it.)

What does this tell us about the limitations of line coverage?

Line coverage measures whether a line was executed at least once, not how thoroughly its behaviour was tested. In our example, we can have 100% line coverage while missing critical behaviours at boundaries (e.g., age == 18) or in invalid input cases. For a boolean expression (age >= 18) with only one variable in it, there will likely not be too many boundaries (just one in fact) – but hopefully you can see that more complex expressions would give rise to more complex boundaries. Good testing needs to explore different conditions and equivalence classes, not just “touch every line”. In the material from this week and the following weeks, we’ll look at ways of assessing coverage which go beyond simple metrics of “number of lines executed”. ISP, graph-based testing, logic-based testing, and syntax-based testing all do the following:

1. Suggest a model or possible way of viewing the subject under test – e.g. as a mathematical function (ISP), or as a set of logic expressions (logic-based testing).
2. Once we’ve modeled the subject under test in a particular way, they give us ways of assessing the thoroughness of a test suite.

Sample solutions:

a. Steps involved are:

1. Identify functions
2. Identify parameters
3. Model the input domain (partitioning it using characteristics)
4. Choose partitions, and values within them
5. Refine these into tests
6. Review

b. input domain

The input domain consists of:

• the set of all possible arrays of chars (for all practical purposes, we can consider this domain as being infinite in size)
• the set of all possible ints (2 parameters of this sort)
• the set of all possible chars (if we assume for simplicity that only ASCII characters are used,² there are 256 of them)

Technically, the input domain consists of a 4-tuple made up of (arrays-of-chars, ints, ints, chars). (We could, if needed, make the sets here a little more mathematically precise, but it’s not needed for our purposes.)

c. Characteristics and partitions:

Some characteristics we could use for our ints are:

1. Are they less than, greater than, or equal to zero?
2. Do one (or both) of them represent positions at the start of the array (i.e. are they zero)?
3. Do one (or both) of them represent positions at the end of the array (i.e. are they array.length - 1)?
4. Is the first parameter greater than, equal to, or less than the second?
   (this gives us 3 partitions)

Some comments on these:

Characteristic (i) is acceptable, but not a particularly useful characteristic in this case. Ask yourself: is it especially likely that the sorting routine would treat positive and negative values differently? If not, then what is the point of dividing an int up in this way? Recall that the purpose of partitioning is to divide a domain up into equivalence classes – values where any one value is likely to be as good as any other. For a sorting routine, that’s already true of an int – there is little value to be got from splitting it up further (though you might do so for completeness, once other, more useful characteristics have been applied).

In fact, this characteristic is something a machine might come up with (interface-based), as opposed to a person thinking about the intended behaviour of the method.

Characteristic (iii) is a characteristic of two parameters combined (which is fine).

For characteristic (iv) - all of these partitions are sensible, and are worth testing for. When the first parameter is greater than the second, then the method documentation says an IllegalArgumentException should be thrown – so we should test for that and make sure that it is.

(You might ask: why should we write a test just to make sure some exception is thrown? The answer is that this is part of the “contract” of the method, and is important. In practice, software developers do want to know what exceptions a method can throw, and do rely on methods throwing what they say they will.)

Some characteristics we could use for our array are:

• Is it of size zero, one, or is it larger than one?
  (Note that if it is of zero, our int parameters will necessarily be outside it, and exceptions will be thrown.)

Note that a characteristic like “is the array in ascending sorted order?” is not a useful characteristic here. The array must be in ascending sorted order; that’s a precondition of calling the method, and if it’s not true, the behaviour is undefined, so what “expected behaviour” could we possibly test for?

Some characteristics we could use for our char are:

• Is it equal to 0?
• Is it equal to 255?

(These are boundary values for the char type.)

Sample solutions:

a. We would model the push Java method as a mathematical function that maps from “the old state” and an integer, to a new state:

\[ push : ([\mathbb{Z}], \mathbb{Z}) \rightarrow [\mathbb{Z}] \]

(We use \([\mathbb{Z}]\) here to indicate an array of integers.)

If we wanted our mathematical notation to be a little more informative to a reader, we could say:

• Let \(StateOfStack = [\mathbb{Z}]\)
• Let push be a function with signature: \(push : (StateOfStack, \mathbb{Z}) \rightarrow StateOfStack\)

Basically, the function is treating the method as if it were something more like a static Java method with the signature static [int] push( int oldState[], int i), which takes in a state, and returns a new state.

For pop, we would model it as \(pop : [\mathbb{Z}] \rightarrow (\mathbb{Z}, [\mathbb{Z}])\) – a function that takes in the state of the stack, and returns a result and a new state. That is, it’s as if the method instead of having signature public int pop () had instead a signature something like static Pair<int, [int]> pop().

(See https://docs.oracle.com/javase/9/docs/api/javafx/util/Pair.html for the Pair type.)

b. Pop has effectively one parameter, the object state.

Some possible characteristics:

• Does the array have zero elements in it, or one, or more?
  (This partitions the domain into 3.)

Note that it doesn’t make sense to have “nullness” as a characteristic. If the array were null, the object would be in an invalid state, and what “expected behaviour” could we possibly test for?

a. Number of tests

The idea behind partitioning is that we’ve split the domain up into what are called equivalence classes, where any one value from each partition is “as good as any other” (so far as the behaviour of the method is concerned).

If that really is the case, then once we have three tests for myMethod, writing more tests adds nothing of value. In fact, excessive tests could make things worse: more tests means regression tests become slower to run, and we have more code to maintain. Every test you write should “carry its weight” – it should serve some useful purpose, and add more value than it costs to maintain.

So – if these really are equivalence classes, then writing more tests adds nothing of value. But in fact, we know that programmers tend to make mistakes around boundaries, so it’s not quite true that every value from each partition is “as good as” any other.

We could add in tests that use -1 and 1 as test values (or maybe even the maximum and minimum values of an int), and still have tests that “carry their weight”.

b. Costs of fixing defects

The main reason is that the component which contains the fault becomes a more and more integral part of the system, and affects more components – the fault has become “built in” to the documentation, larger components, other tests, etc.

This means that fixing the fault is (a) likely to take more effort, because we have to consider all the other components/documentation/tests that it affects, and (b) can have unexpected consequences – our components may interact in complex ways.