A Simple Paper Tool
By Matt Pierce
Introduction
Pilots use it before every flight to improve safety. Building
Inspectors make extensive use of it to aid in the discovery of construction defects. Banks
use it to screen loan applicants. Couples can use it to plan weddings. And…testers
can use it to find bugs. What is it? It is a simple tool from the ‘paper era.’
It is a checklist.
Simplicity & Versatility
Checklists are a simple tool. A checklist’s function is immediately
obvious. Learning how to use a checklist is easy. Learning where to apply a checklist in
new ways is also easy because of the tool’s simplicity. A checklist document is
readily available to others (no special app required for viewing, etc.).
As the opening paragraph indicated, simple tools like a checklist can be used in a wide variety of applications. Of all the tools in a tester’s toolbox, checklists are among the most versatile. Checklists can be used throughout the life cycle of a project: from conception through design, development, stabilization, and even into the maintenance phase. Checklists can be used to aid in the discovery of virtually every class of bug: ranging from requirement bugs to data-related bugs to installation bugs to boundary condition bugs.
Like a knife to the survivalist, a simple tool can be used for an astonishing array of tasks.
Any standard $15 hammer can be used to pound a nail, knock a dent out of the side of a car, or remove a nail. A tool’s simplicity makes it easy to couple with other tools. The hammer can be coupled with a chisel to rough out woodwork, or coupled with a punch to snap rings into a sheet of canvas. Now, take this simple tool and apply a little technology to it…the result is a $50 electric nail gun whose function is limited to embedding nails. Electric nail guns obviously have many advantages over the $15 hammer and yet every carpenter at the job-site still has a hammer in his tool-belt. The electronic nail gun did not replace the hammer, but rather supplemented it within the narrowly defined scope of embedding nails. Checklists are to the hammer what many of our current testing utilities are to the electric nail gun. The checklist is a versatile horizontal tool providing a framework into which vertical tools such as test cases, or test matrices can plug-in.
Checklists are a simple yet powerful tool.
Knowledge Capture
How many of you have read testing books or articles containing dozens of
great ideas or methods? You think to yourself, "I want to adopt those methods!"
However, at the next post-mortem you reflect upon your testing errors and omissions. It is
then that you realize, "If only I had followed through and carried out that neat idea
from Alan Cooper’s book, then we would have caught that big UI bug…" Then,
you ask yourself, "Where did I go wrong? How did I omit creating a test case based
upon that nifty concept from his book? I was going to do it, but then got busy and
forgot."
How many of us have made testing blunders (typically omissions) due to inexperience? We tell ourselves that we learned our lesson and will never repeat that testing error again. Then, two or three project cycles later, while sitting at a post-mortem, we realize that we did indeed repeat the testing error. "How could I do that again?" we ask ourselves.
Good intentions are not sufficient to ensure self-improvement. It is virtually impossible to try and commit hundreds of new testing methods or test cases to memory. It is far easier to merely insert new or modify existing line items within a testing checklist. Let the checklist perform its job as your ‘surrogate’ memory. Let the checklist remind you of those nifty new test checks you read about. Let the checklist remind you of the typical test failures to look for.
Every bug that you encounter should have an associated test check built into the checklist. Make it a priority to mentally check all bugs against the checklist just as you would use the checklist to check for bugs in the app. This will lead to a much faster evolution of the Test Checklist.
Checklists help guard against omission by serving as a master list against which you can check your coverage.
Change Maintenance
A pilot’s checklist is static because his/her plane is
built once and remains structurally intact for years. His/her airplane is not dramatically
rebuilt from month to month or year to year as is often the case with software. The nature
of software is change. Thus, a testing checklist is dynamic and will change to reflect
changes in the software (O.S., business rules, etc.). Yes, this extra effort of constant
maintenance can be a viewed as a negative. However, I prefer to think of the change
maintenance as a net positive in that the testing checklist is a living, constantly
evolving document. As you learn an application’s business rules and new feature set,
or learn new testing techniques, or discover new classifications of bugs, take the
opportunity to build that knowledge into your checklist(s). With that little effort, you
have captured information and placed it into a concise medium of communication. Note that
this concise captured knowledge can be an excellent teaching tool. Take note that only
about half of the testing checks in a checklist are high maintenance—the other half
are generic test checks that change little milestone to milestone, project to project, or
year to year. It is the application specific test checks that can be high maintenance.
During the stress of an approaching milestone or release, checklist consistency can be an enormous comfort. During the monotonous times when there doesn’t appear to be much to test, checklists can be an eye-opener that gets you up and running quick.
A checklist is a living document that must be maintained to achieve its maximum potential.
Test Plans / Test Specs
I am going to tread on ‘holy ground’ here by
comparing and contrasting the use of test plans / test specifications to the use of test
checklists. Test plans and test specifications are used to provide structure for test
cases, organize the testing approach, reduce omissions, etc. By borrowing upon the
strengths of test plans and test specs, hybrid test checklists can be built to perform the
same function, but more efficiently. How? Well, I have attached a simplified Generic Test
Checklist (GTC) below as an example of how a hybrid test checklist might look.
How many of you have at one time or another exerted substantial effort in writing a test plan and accompanying set of test specs, only to watch the docs sit up on a shelf, out of site, collecting dust for the remainder of the project? How many of us really sit down and read a training manual page by page when we want to learn a new programming language? How many of us read the instructions when building a model, or setting up a new computer at home? We naturally avoid this extra excise effort where possible.
There is a significant excise effort penalty associated with producing a test plan / test specification document set. These large bulky documents contain a lot of information, all displayed in full sentences. Checklists also display a lot of information, but in compact sentence fragments broken down in such a manner that the structure itself conveys information (hierarchy of sections and subsections provide structure and checkboxes indicate status).
When properly done, I would guess that 70-80% of the verbiage in a checklist would be read through in detail whereas roughly 40-70% of the verbiage in a test plan or test specification will be read through in detail. That means that 20-30% of a checklist and 30-60% of a test plan / test spec will not be used. That unused 30-60% is substantial wasted effort—if it’s not used, why write it in the first place? (These estimates are highly speculative and most likely vary significantly from group to group and company to company.)
More importantly, a test checklist is designed as a tool to be repeatedly re-read build after build, or milestone after milestone (build verification checklists, etc.) whereas a test plan / test spec. is read through one time and rarely referenced thereafter. A test plan / test spec. is a tool limited to organizing the test approach. A hybrid test checklist is a multi-purpose tool used not only to organize the test approach, but also to actively find bugs.
Test checklists *can* be used as an efficient alternative for test plans and test specifications for appropriate projects (limited time, limited scope, limited resources, etc).
Specific Applications
Checklists can be used in many different ways to assist
testers. Checklists can be used to provide a structured breakdown for Build Verification
Testing, Bug Triages, and rapid testing of Maintenance Builds or Patches. When pressed for
time, checklists provide excellent structure for adhoc testing. Checklists can be used to
test the Requirements List, or the Design Documents. Checklists can be used as a
horizontal tool to check coverage of a test plan, or test specification (any omissions).
Checklists can be used as a vertical tool to check depth in test cases (were all
boundaries checked, etc.).
Checklists have many potential applications in software testing. Because it is a simple tool, checklists are malleable and can be easily shaped to fill many roles.
Sample Checklist Attachment
Table of Contents
I have attached a sample Generic Test Checklist (GTC) below. The
GTC layout is as follows:
Unique vs. Generic Component Testing
The two most important sections in the GTC are the "Unique
Component Testing" and "Generic Component Testing" sections. There is a
significant distinction between these two categories of test checks as discussed below.
Both sections contain many test checks. A test check is just a preliminary test case in
the form of a brief question (sentence fragment).
Unique Component Testing (UCT) is focussed on new or changed features, new or changed business rules, and user scenarios. These application specific issues need to be identified and broken down into separate line items in the UCT section. You can easily come back later and build distinct test cases that map directly into individual UCT line items. Because this verbiage is unique (as opposed to generic boilerplate), you will put a lot of time into developing this area of the checklist. Having substantial detail and a logical Work Breakdown Structure (WBS) in this section will go a long way towards easing the construction of test cases later.
Use discretion in breaking down these unique line items. For example: do not list out 45 business rules for each Primary Key / Foreign Key relationship in the app. Rather, use just one line item check to remind yourself to generate 45 business rule queries in a single Business Rule Enforcement script. (A Business Rule Enforcement script is a fancy phrase for a large set of test cases in the form of SQL queries combined in a SQL script. The script is used to locate and report data-related anomalies).
Generic Component Testing (GCT) is focussed on generic, yet repeatable test line items. These generic line items can give you immediate payback. Just copy the pre-built line items from the "Generic Component Testing" section of the GTC into your own test spec. or checklist and you have just saved yourself hours of work. Furthermore, there is no reason why these generic line items can not be centralized (on \\toolsvr, etc.) so that you not only have your ideas and your test group’s ideas built into the generic components, but also ideas from hundreds of other testers.
Title Page
The Title Page uses Bookmarks to define the app, version,
tester, company, and group. When changing the bookmark values, be careful because deleting
all of the underlined characters (characters in bookmark) will remove the bookmark. The
easiest way I have found to change the bookmark values is to highlight all the inner
characters (leaving first and last character), press delete, enter the new text, then
delete the leftover first and last characters. (Yuck! There must be a better way that I am
overlooking.)
Other Sections
The various tables on the General Application Info. section
(pg 2) were lifted from several Test Plans and Test Specifications that I have
encountered.
The Checklist Instructions section contains notes about items such as why the .rtf file format was chosen (WinWord 6/7/8 compatible), how to copy bullet formatting from line to line, useful test websites URL’s, etc.
The Preparation section is very important. Here is where you make a first pass at determining front end and backend tools to use. Here is where you also list risks and contingencies.
There are other sections including: Integration Testing, Unit Testing, Design Testing, etc. However, these sections are not very developed so I won’t discuss them here.
There are many sections in the attached sample GTC; but the most important concept depicted here is the distinction between Generic vs. Unique Component Testing, and the breakdown of Generic Components.
Future Improvements There are many areas of improvement that can still be incorporated into the test
checklist concept.
The simple GTC attachment below could be dramatically improved.
Below is a list of potential improvements:
Conclusion
I hope that this article has been of some value to you. A
simple checklist is more than just a passive backup for your memory. A checklist can be a
powerful tool when actively used to provide structure for your test cases, or used to
capture knowledge accumulated from your experience, your reading, etc.
If nothing else, I hope you take away the following concept: that a simple paper tool, such as a checklist, can be very powerful, and still has its place in a tester’s toolbox.