Using extracted tests

About extracted tests

At any time that you are running a system in EiffelStudio debugger and your system is paused, you can ask AutoTest to extract a new test class and test from the current executable context. Most often you would use this capability in the case in which you experienced an unexpected failure or exception in one of your routines. It is possible, though, to extract at any point at which the system is paused.

The value of extracted tests is that they provide a kind of a snapshot in testing form that will reproduce the unexpected failure. An extracted test attempts to reproduce the context in which the offending routine executed. So, extracted tests supplement your manual tests. They serve to cover situations which you just may not have written manual tests to cover.

Extracted tests are intended to supplement the suite of manual tests that you have created to do the bulk of your testing. So, usually when you create an extracted test, it happens as a result of your being surprised. You will notice that each time you create an extracted test, you get a new test class, too. This is in contrast to manual tests, in which you might use the wizard to create a new test class and one new test to cover a particular target class and target routine. Then you might manually create, in that same test class, many additional tests covering the routine behavior of the same or other target routines in the same target class.

Creating an extracted test

Let's use the same test system we used for manual tests to demonstrate the creation of an extracted test. The example will be slightly contrived because it will find a problem that certainly we would already have discovered had we written a comprehensive set of manual tests against the BANK_ACCOUNT class. Still, the simplicity should help keep things clear.

If you remember, the root class for the example application was not very interesting, just a root procedure with a single instruction and a declaration my_account of type BANK_ACCOUNT:

make -- Run application. do create my_account end my_account: BANK_ACCOUNT

Now, let's add some code into the make procedure that will make use of my_account:

make -- Run application. do create my_account my_account.deposit (500) my_account.withdraw (100) end

If we run the application from EiffelStudio, we see that it stops when it incurs a postcondition violation in {BANK_ACCOUNT}.withdraw:

When we look at the feature pane, it's pretty easy to see where the problem is:

There is an error in the specification for withdraw. In the postcondition tagged withdrawn, the plus sign should have been a minus sign. Therefore, the assertion should read like this:

withdrawn: balance = old balance - an_amount

Certainly we will fix this, but AutoTest gives us the opportunity to extract a test based on this particular failure. So, let's do that.

So, we go to the AutoTest tool and click triangle next to Create new tests button and select the Extract tests from debugger from the drop-down menu. Because we are paused in the debugger, the drop-down menu appears with the Extract tests from debugger choice enabled this time:

When we select Extract tests from debugger, we are presented with the New Eiffel Test Wizard's Test Extraction pane. This wizard pane shows a depiction of the current call stack and asks us for which feature(s) on the stack we want to create the test:

The choice for withdraw is the selection we want. We can deselect the stack frame for make if it is pre-selected. If we click Next at this point we would be taken to the Tags pane, and from there to the General pane. But we really don't need to do this. AutoTest will sense that we are extracting a test for {BANK_ACCOUNT}.withdraw and tag the test properly. It will use the same test class name from the General pane, but add a numerical suffix. So, all we need to do now is to click Launch from the Text Extraction pane.

AutoTest creates the new test and returns us to the debugger, where our system is still on hold. We can stop execution and compile to include the new test.

Now we see the new test class and test in the AutoTest tool windows.

Run the tests, fix a problem, run the tests

We run our tests using Run all, and we see that the test on withdraw is still failing:

If we fix the error in the postcondition in withdraw, recompile, and then re-execute the test, we find that it is successful.

A closer look at an extracted test

Look at the code that was generated for the extracted test after the assertion violation occurred:

note description: "Regression tests reproducing application state of a previous execution." author: "Testing tool" class TEST_BANK_ACCOUNT_EXTRACTED_WITHDRAW_01 inherit EQA_EXTRACTED_TEST_SET feature -- Test routines test_withdraw note testing: "type/extracted" testing: "covers/{BANK_ACCOUNT}.withdraw" do run_extracted_test (agent {BANK_ACCOUNT}.withdraw, ["#1", {INTEGER_32} 100]) end feature {NONE} -- Access context: !ARRAY [!TUPLE [type: !TYPE [ANY]; attributes: !TUPLE; inv: BOOLEAN]] -- <Precursor> do Result := << [{BANK_ACCOUNT}, [ "balance", {INTEGER_32} 400 ], False] >> end end

You probably noticed immediately that it doesn't look much like the code that we wrote for our manual test in the previous section.

One reason for the difference is that the class does not inherit directly from EQA_TEST_SET as our manual test did. Instead, it inherits from EQA_EXTRACTED_TEST_SET which itself is a descendant of EQA_TEST_SET. EQA_EXTRACTED_TEST_SET provides additional functionality for extracted tests.

Notice that the call to the target routine {BANK_ACCOUNT}.withdraw is effected in the routine test_withdraw which passes an agent representing {BANK_ACCOUNT}.withdraw to the procedure run_extracted_test. The second argument to run_extracted_test is a TUPLE with the argument values which were used in the call to withdraw which caused the original assertion violation.

Another thing worth noting is the function context. This is how AutoTest recreates the state of the instance of BANK_ACCOUNT at the time of the assertion violation.

Caution: The extracted test recreates the state at the point at which execution has halted. So, in the case of a postcondition or invariant violation, the values of the attributes will reflect any changes that have been made during the execution of the routine. (In the example, the value of balance is set to 400, rather than 500 as it would have been when routine withdraw began execution.) This could make a difference in whether the test extracted after an exception is a valid recreation of the original failure. One way of dealing with this, at least in simple cases like this, is to change the test class code to reflect the proper value. A safer way would be rather than extracting the test after the exception, restart the system and stop execution as it enters the failing routine, then extract the test at that point.