Execute tests

In the previous section we coded a manually created test. AutoTest will allow us to execute that test, or, in more practical terms, any set of tests that we select. But before we execute our test, let's take a look at what we will get out of such an execution.

About test results

It is important to understand that for AutoTest, test results are solely determined by whether an exception occurs during the execution of a test, and, in cases in which an exception does occur, what kind of exception it is. So, with AutoTest, it is not necessary for you to write anything special into a test that propagates the test's results.

When AutoTest executes a test, the result will be one of only three possibilities:

1. The test is successful
2. The test is failing
3. The test result is unresolved

These possibilities are defined as follows.

Definition -- Successful test: A test which has executed without causing and exception to occur.

Definition -- Failing test: A test which has caused an exception to occur during its execution, specifically during the execution of a target routine.

Definition -- Unresolved test result: A test which has caused an exception to occur during its execution, but exclusive of the execution of a target routine.

So, successful tests are easy enough to understand. The test executed with no exception.

Failing tests and unresolved test results both mean that an exception occurred during the execution of the test. The distinction is made based on the location of the feature that causes the exception.

When we execute our test {TEST_BANK_ACCOUNT}.test_deposit_01, we know that test_deposit_01 will make a call to {BANK_ACCOUNT}.deposit. If the exception occurs during the execution of a target routine (i.e., in {BANK_ACCOUNT}.deposit), then the test is considered failing. If the exception occurs anywhere else in the execution of {TEST_BANK_ACCOUNT}.test_deposit_01, then the test is considered to have an unresolved result.

Note: Be aware that some early versions of AutoTest reported some unresolved test results as failing tests.

This behavior can be helpful to us as testers. A failing test indicates that there is something amiss in the target routine. The routine has not completed in a state that satisfies its postcondition and class invariant, or is dealing with an unresolved exception from some routine that it has called. An unresolved test result indicates that something is amiss in our test. Something went wrong in the setup or cleanup of the test or perhaps the test called a target routine from a state that did not satisfy the target routine's precondition.

The AutoTest tool

In the last section, we created a manual test. The AutoTest tool shows us the new test in the Tests column. So, now the tool should look something like this:

Test execution

You see under "Tests" the project cluster accounts, the test cluster tests, the test class TEST_BANK_ACCOUNT, and the test test_deposit_01. You might have to expand some of the elements to see everything as shown above.

You see that the Status of test_deposit is "not tested", and that the Last executed date is empty.

To execute tests we use the "Run" button ( ) on the interface toolbar. By default, the Run button will run all tests matching the tags in the Filter box. However, there is a list of run options that you can access by clicking the black triangle just to the right of Run. You can choose to run all tests, only those with failing status, a filtered set of tests, or only those tests that you have selected in the tree below. We'll cover filtering a little later. For now, life is simple, we have only one test so just selecting Run all should execute it.

Examining test results

The test runs in background and the AutoTest interface now looks like this:

It's pretty clear that our test has failed. Its status is now marked with the Failing icon ( ) and in the box below the Execution tab we see that the status also includes a tag: balance_increased. More detail is provided in the Testing pane of the Outputs tool, as shown below.

We see that balance_increased is a postcondition tag on the target routine {BANK_ACCOUNT}.deposit. Upon examination of the code:

deposit (an_amount: INTEGER) -- Deposit `an_amount'. require amount_large_enough: an_amount > 0 do ensure balance_increased: balance > old balance deposited: balance = old balance + an_amount end

we realize that there is no implementation here. So we add the code to implement deposit:

... do balance := balance + an_amount ensure ...

After compiling, we can execute the test again. We could do this by selecting Run all as we did last time, or by selecting Run failing. Once the test executes we see now that it was successful:

This time we see that the test is successful, as indicated by the Success icon ( ) in the Status column.

The beginnings of a test suite

Of course we would not have had to use AutoTest to find that bug in {BANK_ACCOUNT}.deposit. We could have just written a simple class to exercise instances of BANK_ACCOUNT and truth would have come out.

The advantage of using AutoTest is that the test that we wrote to cover {BANK_ACCOUNT}.deposit can stay with us throughout the lifecycle of class BANK_ACCOUNT. We can expand the TEST_BANK_ACCOUNT with additional manual tests and run them after every development increment to ensure that all tests that were once successful are still successful.

Manual test summary

We have seen how to create and execute a manual test. You will find that manual tests form the backbone of your test suite. But there are two other types of tests available in AutoTest. Next let's take a look at these test types and in what ways they can be used.