An Extended Example: Program Testing

UMBC CMSC 211

An Extended Example: Program Testing

Edsger Dijkstra (the guy who said GOTOs where dangerous) has said that testing can only show the presence of bugs, not their absence. Yet most experts agree that testing is essential. Our author says that a program is not done until it is tested!

Even proving a simple programs correct is an incrediably complex undertaking. Still, we can do things that will do a better job of testing if plan the test data well. That will result in an increase confidence in the software, even it we can not guarentee 100% correctness. Basically, we must do everything possible to ensure that our programs are correct.

Our testing should at least satisfy the following items:

Every instruction in the program should be executed in at least one test.
Test boundary values -- largest and smallest values on either side of a place where the treatment of the data changes. These values are places where 'off by one' errors commonly occur.
Test some 'random' values between the boundaries. Often a programmer's preconceived notions of how a program is suppose to work or how it is 'normally' used causes important test conditions to be omitted.
Use one (better to use more) real-world set of data for which you know the results to test your software with. Then when you get a difference(and you probably will), determine where the difference represents a software error or times the person doing this manually makes mistakes.) a manual error. (Don't assume which one is the 'correct' one, because it could be either one.)

Trying these values does not mean you will never have any bugs in your code, but it will increase confidence in the program. Such tests are described by William Jones as "stupid not to", since if the program eventually fails in such simple ways, the programmer looks like a real idiot.

By running it against real data for which you know the results helps to prevent "tunnel vision",revent "tunnel vision", where you overlook possibilities that were not obvious in the artificial test data. Also, it is typical that the users do not know (or remember) what data they are working with, and then the first time you try to run the program for real, it fails. It fails in front of witnesses, and you will feel very embarassed!

A good test plan is to keep all the data being used in files so that the testing can be automated and when the program is modified somehow, you can repeat all the tests plus any new ones needed to show the modifications are working. This retesting from the beginning is called regression testing. It is very common for one modification to cause other parts of the program to fail. This will normally show up in regression testing. The new tests should then be added to the testing suite, so they can also be re-used in the future.

Taking automation a step further, it is helpful to have a way to automate comparing the results of the test to the correct answers. This prevents the humans from missing errors in the test results. It also make regression testing easier!

In the last chapter, we looked at a procedure PUTDEC. To make sure that it is running correctly, we must test it according to the rules.

To exercise every instruction, we need to use the values -32768, another negative value and a positive value.
To check the boundaries, we To check the boundaries, we need to check -32767, -1 for the negative values, 0 and 32767 for the positive values. Since zero might be a special cases, we should also use 1.
For random values the author arbitarily picked -12345, -864, -9, 10, 1057.

If we put the numbers into sequential order, we have:

This will be the results of our program if it is working correctly.

To set a value, we need a sequence of:

	mov	ax, aNumber
	call	PutDec
	_PutCh	13, 10

Since we have 11 numbers in our test day, we could copy and paste for a total of 33 lines or we could have a macro to reduce the amount of work we have to do:

;; TESTPROG.ASM -- Example 8.2-2, a test suite for the PutDec procedure
;;
;;  Program text from "Assembly Language for the IBM PC Family" by
;;   William B. Jones, (c) Copyright 1992, 1997 Scott/Jones Inc.
;;
INCLUDE	PCMAC.INC

DispNum	MACRO	aNumber
	mov	ax, aNumber
	call	PutDec
	_PutCh	13, 10
	ENDM

	.MODEL	SMALL
	.STACK	100h
	.DATA
	.CODE
	EXTRN	PutDec: NEAR
TestProg	PROC
	mov	ax, @data
	mov	ds, ax
	DispNum -32768
	DispNum -32767
	DispNum -12345
	DispNum -864
	DispNum -9
	DispNum -1
	DispNum 0
	DispNum 1
	DispNum 10
	DispNum 1057
	DispNum 32767
	_Exit	0
TestProg	ENDP
	END	TestProg

Improved version

It would help if the test program could tell us what the results were supposed to look like in the form of: supposed-to-print = actually printed We can modify our test program to do that:

;; TESTPRG2.ASM -- Example 8.2-2, a test suite for the PutDec procedure
;;
;;  Program text from "Assembly Language for the IBM PC Family" by
;;   William B. Jones, (c) Copyright 1997 Scott/Jones Inc.
;;
INCLUDE	PCMAC.INC

DispNum	MACRO	aNumber	;; second version
	local	NumMsg
	.DATA ; Switch temporarily back to the .DATA section
NumMsg	DB	'&aNumber& = $'
	.CODE ; Switch back to .CODE section
	_PutStr	NumMsg
	mov	ax, &aNumber&
	call	PutDec
	_PutCh	13, 10
	ENDM

	.MODEL	SMALL
	.STACK	100h
	.DATA
	.CODE
	EXTRN	PutDec: NEAR
TestProg	PROC
	mov	ax, @data
	mov	ds, ax
	DispNum -32768
	DispNum -32767
	DispNum -12345
	DispNum -864
	DispNum -9
	DispNum -1
	DispNum 0
	DispNum 1
	DispNum 10
	DispNum 11057
	DispNum 32767
	_Exit	0
TestProg	ENDP
	END	TestProg

UMBC | CSEE