ISEB:Fundamental Test Process

Testing: ISEB Foundation

Fundamental Test Process

The fundamental test process comprises planning, specifiation, execution, recording and checking for completion. In more detail:

Test Planning

The test plan should specify how the test strategy and project test plan apply to the software under test. This should include identification of all exceptions to the test strategy and of all software with which the software under test will interact during test execution, such as drivers and stubs.

Test Specification

Test cases should be designed using the test case design techniques selected in the test planning activity.

Test Execution

Each test case should be executed.

Test Recording

The test records for each test case should unambiguously record the identities and versions of the software under test and the test specification. The actual outcome should be recorded. It should be possible to establish that all of the specified testing activites have been carried out by reference to the test records.

The actual outcome should be compared against the expected outcome. Any discrepancy found should be logged and analysed in order to establish where its cause lies and the earliest test activity that should be repeated, e.g. in order to remove the fault in the test specification or to verify the removal of the fault in the software.

The test coverage levels achieved for those measures specified as test completion criteria should be recorded.

Checking for Test Completion

The test records should be checked against the previously specified test completion criteria. If these criteria are not met, the earliest test activity that must be repeated in order to meet the criteria should be identified and the test process should be restarted from that point.

It may be necessary to repeat the Test Specification activity to test further cases to meet a test coverage target.

As the object of a test should be to detect faults, a "successful" test is one that does detect a fault. This is counter-intuitive, because faults delay progress: a successful test is one that may cause delay. The successful test reveals a fault which, if found later, may be many more times more costly to correct so, in the long run, is a good thing.

Completion or exit criteria are used to determine when testing (at any test stage) is complete. These criteria may be defined in terms of cost, time, faults found or coverage criteria. Coverage criteria are defined in terms of items that are exercised by test suites, such as branches, user requirements, most frequently used transactions, etc.

The Psychology of Testing

Testing is performed with the primary intent of finding faults in the software, rather than of proving correctness. Testing can therefore be perceived as a destructive process. The mindset required to be a tester is different to that of a developer.

There are right and wrong ways of presenting faults to authors or management (give examples). It is important to communicate between developer and tester: e.g., changes to the application or menu structures that might affect the tests; or where the developer thinks the code might be buggy; or where there might be difficulty in reproducing reported bugs.

Generally it is believed that objective, independent testing is more effective. If author tests then assumptions made are carried into testing, people see what they want to see, there can be emotional attachment, and there may be a vested interest in not finding faults.

Levels of independence, such as:

a)	test cases are designed by the person(s) who writes the software under tests;
b)	test cases are designed by another person(s);
c)	test cases are designed by a person(s) from a different section
d)	test cases are designed by a person(s) from a different organisation;
e)	test cases are not chosen by a person.

Re-Testing and Regression Testing

Whenever a fault is detected and fixed then the software should be re-tested to ensure that the oriinal fault has been successfully removed. You should also consider testing for similar and related faults.

Tests should be repeatable, to allow re-testing/ regression testing.

Regression testing attempts to verify that modifications have not caused unintended adverse side effects in the unchanged software (regression faults) and that the modified system still meets its requirements. It is performed whenever the software, or its environment, is changed.

Regression test suites are run many times and generally evolve slowly, so regression testing is idea for automation. If automation is not possible or the regression test suite is very large then it may be necessary to prune the test suite. You may drop repetitive tests, reduce the number of tests on fixed faults, combine test cases, designate some tests for periodic testing, etc. A subset of the regression test suite may also be used to verify emergency fixes.

Expected Results

Expected results are synonymous with expected outcomes, but not the same as outputs. If expected results have not been defined then a plausible, but erroneous, result may be interpreted as the correct one. Expected results must therefore be defined prior to test execution.

The "oracle assumption" is that a tester can routinely identify the correct outcomes of a test. An oracle may be (e.g.) the existing system (for a benchmark), or a specification, or an individual's specialised knowledge, but not the code.

Prioritisation of Tests

There is never enough time to do all the testing you want, so you must prioritise.

Prioritise tests so that whenever you stop testing you have done the best testing in the time available.

Identify the ranking criteria used to prioritise, such as severity, probability, visibility of failure, the priorities of the requirements to be tested, what the customer wants, change proneness, error-proneness, business criticality, technical criticality and complexity.

This page is very heavily based on the ISEB Foundation Certificate Syllabus.
This is unavoidable, as to pass the exam, you have to use their terminology.