Testing Tips and Tricks of the Trade
A ‘top ten’ list of State Change Tests by Matt Pierce, Volt
As testers, we have all had our share of nasty bugs whose detection is difficult. The bug might be intermittent, so we spend hours attempting to discern why the failure is inconsistent over time. The bug might work on all test machines except one, so we spend hours attempting to discern why the failure is inconsistent across machines. Worst of all, the bug might have occurred only once, so we spend hours attempting to determine the exact repro path that broke the app. In the worst cases, we do not find an answer, and are faced with the decision of what to do with the bug.
Do we just report the bug’s symptom(s) and classify it as non-reproducible, "confident" that we tried everything possible to isolate a consistent repro path? Or is there additional testing that we can do to further identify the bug so that it will actually get fixed?
I have encountered these difficult bugs several times in my career. Occasionally, I felt compelled to take the easy way out by writing-off a bug as non-reproducible. I rationalized my actions: "Well, I’ve spent three days trying to repro this bug and have to move on," or "Well, it only fails on that weird machine in the lab but runs everywhere else". Ironically, this negative mentality prevented me from performing my job (finding bugs and ensuring they get fixed). I should have been thinking positively: "I know the app works here but not there; thus I just need to determine the differences between machines, or differences over time and therein will I find the cause of the bug." If I were thinking positively, I would have realized that I was just a couple of State Snapshots and a File Comparison away from detecting additional symptoms, or even the causal agent.
What is a State Snapshot and how can it assist you in isolating additional symptoms, or even identifying causal agents? A State Snapshot is a picture of the state of a computer (or network) at a specific point in time in a specific condition. A State Snapshot is captured and then placed into a file. The use of files enables easy capture and comparison of multiple State Snapshots so that the differences are readily apparent. Causal agents and additional symptoms will virtually always exist somewhere in the snapshot delta. Thus, by filtering out the similarities, you are reducing the effort it takes to identify additional symptoms, or even causal agent(s).
The State Snapshot Process consists of four general steps:
File Comparison: Finally, review the state differences with a file comparison utility such as ExamDiff.exe (
Below, I have listed 10 uses and methods for the State Snapshot Process:
RegEdit.exe has a menu item "Export Registry File" under the "Registry" menu permitting you to take a snapshot of the entire registry. You can use this feature to compare registry snapshot files before and after install, before install vs. after uninstall, between two machines where one fails and the other passes, etc. Shotgun.exe and similar tools help to automate registry state tracking.
You can shell out and use the DOS DIR command to take snapshots of the hard-drive file names, file sizes, and file date/time stamps in multiple states. These State Snapshots can then be compared to determine which files changed. To make a snapshot, shell out to DOS and run DIR <path> /s > <target file>. For example: DIR C:\ /s C:\TEMP\PASSED.TXT
If yes, then a schema comparison will assist you in quickly identifying what component changed or is missing. Go into SQL Server Enterprise Manager and select a Server and Database. Then select the menu item "Object", and the sub-item "Generate SQL Scripts…". Next, check "All Objects", select each of the checkboxes in the "Scripting Options" frame (ignore the Security frame if you want), and be sure the "File Options" is set to "Single File". Next, click the "Script" button. Finally, enter a target filename when the "Save As" dialog pops up and press "Save". You now have a large text file containing all generated scripts for all tables, stored procedures, triggers, views, defaults, rules, etc. Take a similar snapshot of the same database in another state (on another server, before or after upgrade patch is run, etc.) and compare the two schemas.
Save the results of a SQL query execution window into a text file for multiple states. Then, compare the State Snapshot files to look for data mismatches, record counts variances, etc. For example, you could take multiple state snapshots of a query that lists the record counts for all tables. Then compare the files to quickly isolate which tables had data added or removed. You could also do a simple select to dump the content of target table to a file, then compare the contents over time, or across systems.
Compatibility issues are great for file comparison utilities. To test printed output: setup a new printer that outputs to a file (not a COM nor LPT port). Use "Start" button > "Settings" > "Printers" > "Add Printer", "My Computer", "File", … If you can, setup a text type printer; but if you must (due to nature of app) go ahead and setup binary output via HP or other printer. Next, run both the old and new versions of the test app. Print out identical reports, one from each system. Perform a file comparison on the two report files to locate any differences. The same snapshot comparison between old and new can be applied to tables modified by a test case in SQL Server, to data files created by the app, etc.
WinDiff.exe gives you a slick way of comparing all files in a directory against what should be a mirror image in your release drop point.
If yes, then the failing machine is probably just missing a DLL, or other support file. Grab a utility like WinNT’s Wps.exe, or MSOffice’s SysInfo to export a list of all loaded .DLL’s to snapshot files for comparison (sort by running column in SysInfo).
When testing a new build, it is often a good idea to know where changes occurred and focus testing accordingly for that build-test cycle. If your group is using VSS, then you can quickly sort all source files by date, and then do a "Show History" and "Diff" the new build against previous builds. If you are not using VSS, then you can use WinDiff (or other util) to locate areas of source code change.
For those of us occasionally testing legacy apps, this could occur. To use the file comparison utility, just rename your .INI files to represent the different states (i.e.: Failed.ini, Passed.ini, etc.) and then compare them.
Well, the Win95 / WinNT4.0 "Find File or Folder" dialog can be of great assistance. Unlike the state change tracking methods listed above, this one does not require a file comparison utility. Simply right click the "Start" button and click "Find…" Select the "Look In" folder to C:\, and the File Spec to "*.*". Now for the most important part: select the "Date Modified" tab and set the date option to "During the Previous 1 days" Click the "OK" button, and presto, you now have a list of all the files touched by today’s activities.
In conclusion, don’t give up on those seemingly intermittent, non-reproducible bugs, or otherwise bizarre bugs. By using the State Snapshot process, you might be able to isolate addition symptoms that give you a consistent repro path, or triggers identification in the developer’s mind. On bugs that are external to the code, you might even be able to identify the subtle causal agent behind the bug (e.g.: wrong version of a DLL, or ODBC drivers missing, or registry setting is corrupt, etc.).