Test design

In software engineering, test design is the activity of deriving and specifying test cases from test conditions to test software.

Definition

A test condition is a statement about the test object. Test conditions can be stated for any part of a component or system that could be verified: functions, transactions, features, quality attributes or structural elements.

The fundamental challenge of test design is that there are infinitely many different tests that you could run, but there is not enough time to run them all. A subset of tests must be selected; small enough to run, but well-chosen enough that the tests find bug and expose other quality-related information.[1]

Test design is one of the most important prerequisites of software quality. Good test design supports:

  1. defining and improving quality related processes and procedures (quality assurance);
  2. evaluating the quality of the product with regards to customer expectations and needs (quality control);
  3. finding defects in the product (software testing).

The essential prerequisites of test design are:[2]

  1. Appropriate specification (test bases).
  2. Risk and complexity analysis.
  3. Historical data of your previous developments (if exists).

The test bases, such as requirements or user stories, determine what should be tested (test objects and test conditions). The test bases involves some test design techniques to be used or not to be used.

Risk analysis is inevitable to decide the thoroughness of testing. The more risk the usage of the function/object has, the more thorough the testing that is needed. The same can be said for complexity. Risk and complexity analysis determines the test design techniques to be applied for a given specification.

Historical data of your previous developments help setting the best set of test design techniques to reach a cost optimum and high quality together. In lack of historical data some assumptions can be made, which should be refined for subsequent projects.

Based on these prerequisites an optimal test design strategy can be implemented.

The result of the test design is a set of test cases based on the specification. These test cases can be designed prior to the implementation starts, and should be implementation-independent. Test first way of test design is very important as efficiently supports defect prevention. Based on the application and the present test coverage further test cases can be created (but it is not test design).

In practice, more test design techniques should be applied together for complex specifications.

Altogether, test design does not depend on the extraordinary (near magical) skill of the person creating the test but is based on well understood principles [3].

Automatic test design

Entire test suites or test cases exposing real bugs can be automatically generated by software using model checking or symbolic execution.[4] Model checking can ensure all the paths of a simple program are exercised, while symbolic execution can detect bugs and generate a test case that will expose the bug when the software is run using this test case.

However, as good as automatic test design can be, it is not appropriate for all circumstances. If the complexity becomes too high, then human test design must come into play as it is far more flexible and it can concentrate on generating higher level test suites.

gollark: Thanks!
gollark: I think this is technically possible to implement, so bee⁻¹ you.
gollark: This is underspecified because bee² you, yes.
gollark: All numbers are two's complement because bee you.
gollark: The rest of the instruction consists of variable-width (for fun) target specifiers. The first N target specifiers in an operation are used as destinations and the remaining ones as sources. N varies per opcode. They can be of the form `000DDD` (pop/push from/to stack index DDD), `001EEE` (peek stack index EEE if source, if destination then push onto EEE if it is empty), `010FFFFFFFF` (8-bit immediate value FFFFFFFF; writes are discarded), `011GGGGGGGGGGGGGGGG` (16-bit immediate value GGGGGGGGGGGGGGGG; writes are also discarded), `100[H 31 times]` (31-bit immediate because bee you), `101IIIIIIIIIIIIIIII` (16 bits of memory location relative to the base memory address register of the stack the operation is conditional on), `110JJJJJJJJJJJJJJJJ` (16 bit memory location relative to the top value on that stack instead), `1111LLLMMM` (memory address equal to base memory address of stack LLL plus top of stack MMM), or `1110NNN` (base memory address register of stack MMM).Opcodes (numbered from 0 in order): MOV (1 source, as many destinations as can be parsed validly; the value is copied to all of them), ADD (1 destination, multiple sources), JMP (1 source), NOT (same as MOV), WR (write to output port; multiple sources, first is port number), RE (read from input port; one source for port number, multiple destinations), SUB, AND, OR, XOR, SHR, SHL (bitwise operations), MUL, ROR, ROL, NOP, MUL2 (multiplication with two outputs).

References
  1. Test Design: A BBST Workbook, by Cem Caner and Rebecca L Fiedler, July 2016
  2. Practical Test Design: Selection of traditional and automated test design techniques, by István Forgács and Attila Kovács, August 2019
  3. A Practitioner's Guide to Software Test Design, by Lee Copeland, January 2004
  4. KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs, by Cristian Cadar, Daniel Dunbar, Dawson Engler of Stanford University, 2008
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.