CITS5501 lab 5 (week 6) – version control, logic-based testing – solutions
The week 5 lectures include a workshop on using Git and GitHub for version control.
If this is the first time you have used Git and GitHub, you should work through the resources included there, including the Github Hello World Step by Step tutorial.
If you have used Git and GitHub before, you might like to think in more depth about how version control can contribute to (or pose problems for) software quality. The questions below ask you to reflect on this.
What is the basic purpose of version control? How does it differ from simply keeping copies of files with different names or dates?
How might version control make software development easier or harder? You may want to check online resources to help you answer these questions.
In group assessments from previous units, has your group adopted any conventions or best practices when using version control? If so, do you think they were useful – and in what ways? If not, did you encounter any problems where adopting a convention might have made things easier?
Think about the following:
Can you think of any indicators in a version control repository that might be a sign of good version control discipline (or conversely, or poor version control discipline)? If you were reviewing a peer’s repository, what would you look for as signs of good or bad version control discipline?
Version control tools can either support software quality or undermine it, depending on how they’re used. Good practices make repositories easier to work with, simplify testing, and reduce the risk of errors. But poor practices can lead to confusion, wasted time, and hidden bugs.
In a well-managed repository, the version control history should clearly show
Information about the first two items comes directly from the version control tool – all version control tools will show what content in what files was changed, and when a commit occurred. However, providing reasons why the change was made can only be done by the developer who committed it. In version control systems, commit messages are used to explain the reasons for a change. You can read more about what makes for good or bad commit messages in Simon Tatham’s guide to “Writing commit messages”.
When used properly, version control systems make it much easier to solve problems like
We won’t discuss version control best practices in exhaustive detail here, but will highlight a few important ones. For more detail, you can read Michael Ernst’s “Version control concepts and best practices”.
You might notice that some of these practices are in tension – if you commit early and commit often to stop work from being lost, how can you also ensure that you’ve provided good commit messages, or created commits at logical points?
The answer to this apparent paradox is that it’s possible to rewrite the local history of your repository, and in fact it’s a good idea to
Once you’ve pushed your commits to a remote repository, it’s generally a bad idea to alter them, since other team-mates’ work may be affected by any changes. But before you push your commits, it’s perfectly alright – in fact, recommended – to tidy up the history before pushing it. Read Carlos Schults’s post “Make Your Git History Look Beautiful Using Amend and Rebase” for more information on how to do this.
Pre-reading
The following exercises in this worksheet assume general familiarity with logic-based testing concepts, so if you haven’t already, it’s recommended you work through chapter 8 from the Ammann and Offutt textbook before attempting them.
The remaining sections of this lab focus on assessing whether a test suite thoroughly exercises (i.e., has good coverage of) the logic expressions contained in the component under test.
When writing logic expressions, we will often use mathematical notation for “and”, “or”, and “not”:
This notation is independent of any language; it could be turned into Java, or C, or Python – each of which uses different logical operators – depending on what language our system and our tests are implemented in.
If writing actual Java code, however, we use the normal Java logical operators:
&& – “and”|| – “or”! – “not”Other operators and languages
Java also has non–short-circuiting logic operators, |
and &.
We won’t be focusing on Python in this unit – but for reference, in Python, the logic operators are all spelled out as English words: “and”, “or” and “not”.
If you need to, review the lecture material and recommended readings that explain what predicates and clauses are.
What are the clauses in the predicates below?
\(((f \leqslant g) \wedge (x > 0)) \vee (M \wedge (e < d +c))\)
\(G \vee ((m > a) \vee (s \leqslant o + n)) \wedge U\)
Solutions
\(((f \leqslant g) \wedge (x > 0)) \vee (M \wedge (e < d +c))\)
There are 4 clauses:
\(G \vee ((m > a) \vee (s \leqslant o + n)) \wedge U\)
There are 4 clauses:
To make a particular clause \(c\) in some predicate active means to assign values to variables so that the truth-value of the whole predicate depends on \(c\).
When coming up with test values which make clauses active, the easiest way of showing your test values is in a table.
E.g. Suppose we have a predicate \(s \wedge (m \vee w)\), where
If asked to come up with test inputs which make each clause active in turn, and achieve Restricted Active Clause Coverage, we could show them like this:
| Test description | Inputs | Predicate value |
|---|---|---|
| Make s active, and s = true s = false |
s = true, m = true, w = false s = false, m = true, w = false |
true false |
| Make m active, and m = true m = false |
s = true, m = true, w = false s = true, m = false, w = false |
true false |
| Make w active, and w = true w = false |
s = true, m = false, w = true s = true, m = false, w = false |
true false |
(Here, we aren’t told what the expected outcome is if the predicate comes out true or false; if we were, we could add a column “Expected outcome” which listed this.)
If you aren’t told the exact types of variables or methods used in a predicate, that means you should be able to work them out from context. For example, for the predicate
\[ (x > 0) \vee (M \wedge (e < d +c)) \]
you can assume that \(M\) is a
boolean, and that \(x\), \(c\), \(d\)
and \(e\) are some numeric type (such
as int).
For each of the clauses in the predicates below, identify test inputs which will make the clause active (that is: state what values need to be assigned to the variables in the predicate), and vary that clause so it takes on both true and false values. (In other words: write test values that achieve Restricted Active Clause Coverage.) Explain your reasoning.
In the solutions, we explain our reasoning, and then give a table with test values. This is a good way to format your answers, if you’re asked to come up with test values.
Almost always, there are multiple possible solutions (because there are many possible choices for the test values), but we show only one.
For (a):
This one should be simple. Our predicate is \(A \vee (B \wedge \neg C)\), and so our clauses here are \(A\), \(B\) and \(C\).
Note that strictly speaking, we don’t need the parentheses in \(A \vee (B \wedge \neg C)\), since “\(\wedge\)” is considered to bind more
tightly than “\(\vee\)”, but we add
them for clarity. (This is reflected in most programming languages,
where “&&” has higher precedence than
“||”.)
Our table of test inputs would then look like this:
| Test description | Inputs | Predicate value |
|---|---|---|
| Make \(A\) active, and \(A\) = true \(A\) = false |
\(A\) = true, \(B\) = false, \(C\) = true \(A\) = false, \(B\) = false, \(C\) = true |
true false |
| Make \(B\) active, and \(B\) = true \(B\) = false |
\(A\) = false, \(B\) = true, \(C\) = false \(A\) = false, \(B\) = false, \(C\) = false |
true false |
| Make \(C\) active, and \(C\) = true \(C\) = false |
\(A\) = false, \(B\) = true, \(C\) = true \(A\) = false, \(B\) = true, \(C\) = false |
false true |
For (b):
Our predicate here is \(x > 0 \; \vee (M \wedge (e < d +c))\), so our clauses are \(x > 0\), \(M\), and \(e < d +c\).
If you look at the logical structure of this, it actually has a similar logical form as in (a): \(A \vee (B \wedge C)\). But here, some of the variables are of some integral type, so our test values will be any integers we choose that make the relevant clauses true or false.
Our table of test inputs would then look like this:
| Test description | Inputs | Predicate value |
|---|---|---|
| Make \(x > 0\) active, and \(x > 0\) = true \(x > 0\) = false |
\(x = 1\), \(M\) = false, \(e
= d = c = 0\) \(x = 0\), \(M\) = false, \(e = d = c = 0\) |
true false |
| Make \(M\) active, and \(M\) = true \(M\) = false |
\(M = \text{true}\), \(x = 0\), \(e = d
= c = 1\) \(M = \text{false}\), \(x = 0\), \(e = d = c = 0\) |
true false |
| Make \(e < d + c\) active,
and \(e < d + c\) = true \(e < d + c\) = false |
\(x = 0, M = \text{true}\), \(e = d = c = 1\) \(x = 0, M = \text{true}\), \(e = d = c = 0\) |
true false |
For (c):
Our predicate here is \(G \vee (m \geqslant a) \vee H \wedge U\), so our clauses are \(G\), \(m \geqslant a\), \(H\) and \(U\).
Note that because “\(\wedge\)” binds more tightly than “\(\vee\)”, this predicate is equivalent to
\[ G \vee (m \geqslant a) \vee (H \wedge U) \]
Our process for making each clause active is as follows:
Our table of test inputs would then look like this:
| Test description | Inputs | Predicate value |
|---|---|---|
| Make \(G\) active, and \(G\) = true \(G\) = false |
\(G = \text{true}\), \(m = a = 0\), \(H
= U = \text{false}\) \(G = \text{false}\), \(m = a = 0\), \(H = U = \text{false}\) |
true false |
| Make \(m \geqslant a\) active,
and \(m \geqslant a\) = true \(m \geqslant a\) = false |
\(m = a = 0\), \(G = H = U = \text{false}\) \(m = 0, a = 1\), \(G = H = U = \text{false}\) |
true false |
| Make \(H\) active, and \(H\) = true \(H\) = false |
\(H = \text{true}\), \(G = \text{false}\), \(m = 0, a = 1, U = \text{true}\) \(H = \text{false}\), \(G = \text{false}\), \(m = 0, a = 1, U = \text{true}\) |
false true |
| Make \(U\) active, and \(U\) = true \(U\) = false |
\(U = \text{true}\), \(G = \text{false}, m = 0, a = 1, H =
\text{true}\) \(U = \text{false}\), \(G = \text{false}, m = 0, a = 1, H = \text{true}\) |
false true |
Suppose a component under test has the following requirements:
If the lever is pulled and the chair is occupied, open the trap-door.
If the button is pressed, open the trap-door.
Represent the component as a set of logic expressions. You should explain what each variable in your expressions means. (For an example, look in section 2 at the way we gave definitions for the variables in the predicate \(s \wedge (m \vee w)\).)
(Hint: if you’re stuck, try writing out what the component does as
one or more “if” statements, in pseudocode. Then recall
that the set of all predicates in a system means the set of all logical
expressions found in things like “if” statements.)
Answer:
We have two predicates in our component: the first is “the lever is pulled and the chair is occupied”, and the second is “the button is pressed”.
We will define three variables to represent the clauses in these predicates:
Using these definitions, the set of logic expressions (i.e. predicates) in our component is therefore:
Incorrect answers
Note that it’s incorrect to try and represent “open the trap-door” as a clause.
This is because from the requirement we’re given, “open the trap-door” is clearly not a condition we have to detect, but rather an action our system must take (and something we can hopefully observe as part of seeing whether we get the expected outcome).
To see why this is so, and why we shouldn’t model “open the trap-door” as a clause, imagine writing pseudocode to represent what our component does. It might look something like this:
if lever pulled and chair occupied:
open trap-door
return
if button pressed:
open trap-door
returnThen recall that doing logic-based testing means to test the logical expressions in our system – the conditions following the “if” statements – and to ensure we’ve thoroughly tested the clauses that make them up.
Now imagine we are writing JUnit style tests to see if our component
(let’s call it TrapDoorController) behaves as expected.
We’ll assume we have mock objects called lever and
chair and button created in a
setUp method, and that our controller has a
control() method to which we pass the lever
and chair and button. For one of our tests, we
might have something like:
@Test
/** Test the case when lever pulled, chair not occupied,
* button not pressed */
public void testA() {
// prepare the test environment
lever.setPosition("pulled");
chair.setOccupancy("occupied");
button.setPosition("unpressed");
// invoke the method under test
TrapDoorController c = new TrapDoorController();
c.control(lever, chair, button);
// assert that the behaviour is as we expect
assertEquals( c.getStatus(), "closed",
"trapdoor should be closed");
}And we’d have a bunch of other tests to test other scenarios. So – what if “open the trap-door” were a clause, rather than an action to take? Then it would become part of the test environment. But if that were the case – what would be left to be the component under test? And what could we possibly assert in order to find out if the system behaved as expected or not? There are no sensible answers to these questions; hence it makes no sense to make “open the trap-door” a clause.
Note that for the exam and assignment, student answers that make this sort of mistake will generally be awarded very few (if any) marks – it will be taken to indicate a poor understanding of logic-based testing.
Suppose you are part of a team developing a website called “RateMyVeterinarian”, where people can log in and provide anonymous reviews of the veterinarian services they use.
Requirements for the site are currently being finalised, and one requirement is stated as follows:
When a user enters a user ID and password into the login page and hits the “log in” button, then if that user ID is listed in the “users” database, and the password matches against the password in the record for that user, and the user record does not state that the account has been disabled, a “Welcome” page should be displayed.
How easy to understand do you think this requirement is? If you think it could be made easier to understand, suggest how.
One of your colleagues suggests that because correctly authenticating users (and keeping their details secure) is an important feature, this requirement should be thoroughly tested – so you should design a test suite that meets RAC (Restricted Active Clause) levels of coverage. Do you agree? Why or why not?
a. Requirement clarity and readability
This requirement would probably be more readable if the various conditions were given as bullet points, rather than a run-on sentence (together with a little re-phrasing):
When a user enters a user ID and password into the login page and hits the “log in” button, then if:
a “Welcome” page should be displayed.
You may have other suggestions for how the requirement could be made clearer.
b. RACC coverage
There is no one correct answer to this question.
It’s certainly true that we should probably test the login feature thoroughly. However, once we have portions of the system implemented and at least a partial test suite in place, we may discover that we in fact already have achieved an RACC level of coverage.
So our colleague is not incorrect in saying that thorough testing is warranted, and they’re not wrong to suggest that RACC is a good level of coverage to aim for. However, if they are suggesting that the tests need to be designed right now, that doesn’t necessarily follow. The coverage our colleague wants may arise naturally out of applying techniques like Input Space Partitioning and graph-based testing.
For instance, suppose that when we come to implement the login feature, we end up with a method that looks something like this:
public void handleLogin(HttpRequest request) {
String userID = request.getParameter("userID");
String password = request.getParameter("password");
UserRecord userRecord = userDb.lookup(userID);
if (userRecord == null)
throw new NoSuchUserException("no user called " + userID);
if (! userRecord.getPassword().equals(password) )
throw new InvalidPassword();
if (userRecord.getDisabledStatus == DISABLED)
throw new UnauthorisedAccess("account is disabled");
// if still here, all is OK
renderWelcomePage();
}(Note that this is simplified from how a real login handler method would work – amongst other things, we shouldn’t be storing user passwords in a database in their raw, original format.)
We will probably write unit tests (using the ISP method) to make sure this method does the right thing. We might also check what level of graph coverage we have of the method, and perhaps write more unit tests and/or integration tests in response to that.
By the time we’ve done all that, it’s quite possible we will have achieved an RACC level of coverage anyway, and don’t have to write additional tests.
(If you’re familiar with what HTTP requests look like, you might like to consider how you’d write ISP-based tests for this method. You would probably decided to use partitioning characteristics like “HTTP request contains a valid userID” and “HTTP request contains a password matching the userID”, amongst other things. Once you apply something like Base Choice Coverage as criterion for your ISP tests, it’s highly likely RACC will be satisfied for the requirement.)
When solving problems that involve logic-based testing, make sure you understand the difference between expressions and statements in the programming language or languages that you’re using.
For instance, simple Java if statements are
of the form:
if ( Boolean expression
) { Statements
}
Statements are parts of the language that do things; we
usually execute them in order to achieve some side effect. Boolean
expressions evaluate to a Java boolean value, and we
typically intend that they should not have side effects – they
just evaluate to true or false.
When we do logic-based testing, we’re focusing on the collection of logic expressions of a program or system that control program flow, and working out how to write tests that thoroughly exercise the component parts (the clauses) of those expressions.
Confusing expressions with statements, or vice versa, is a common reason for students to do poorly in test or exam questions involving logic-based testing.