Digital Test Manager
Digital Test Manager (DTM) is a layered product distributed by VMS Software, part of DECset, that can be used to run, review, and store software regression tests and test results. DTM organizes and automates the software regression testing process.
DTM supports three testing environments:
- Noninteractive (command procedure)
- Interactive terminal (recorded input)
- Interactive DECwindows
The three basic components of DTM are:
- A DTM library - a data structure that DTM uses to store the information it needs to manage a test system in an OpenVMS directory.
- A DTM test - a collection of fields that contains the information that VSI Digital Test Manager needs to run a particular test. A test description requires a template file and can have other optionally specified test-related entities. A template file is an OpenVMS command procedure that executes a noninteractive test, or a session file containing a recorded interactive terminal or DECwindows session. Chapter 3 describes tests and test descriptions in more detail. A benchmark file contains the expected output for the test's execution.
- A collection - a set of tests selected for execution. You can execute a test only in the context of a collection. You can select tests for inclusion in a collection by test name or groups. Chapter 4 describes collections in detail.
Before any DTM tests can be created, a DTM library needs to be created first. To do that, you create an empty directory and then run the CREATE LIBRARY command on this directory.
DTM tests are characterized by test descriptions. A DTM test description consists of the following elements:
- Test name: identifies the test description.
- Test prologue: identifies a DCL command file that runs immediately before the template file. You use a prologue file to set up any special environment that the test requires. Output from a prologue file does not appear in the test results.
- Test epilogue: identifies a DCL command file that runs immediately after the template file. You use an epilogue file to clean up operations, or to apply user-created filters to the result file. Unlike the prologue file, the epilogue file can directly alter the test results.
- Template: identifies a DCL command file for an on interactive test, or the session file for an interactive terminal or DECwindows test. This field defaults to test-name.SESSION for an interactive terminal or DECwindows test and test-name.COM for noninteractive tests.
- Benchmark: identifies a file that contains the expected test output. It is the standard against which VSI Digital Test Manager compares the results of a test run. This field defaults to test-name.BMK.
- Variables: identifies the variables and associated values used with the template, prologue, or epilogue files for this test.
- Groups: identifies the groups to which the test description belongs.
- Test type: identifies the test as either an interactive terminal, DECwindows, or noninteractive test.
- Command: Identifies a DCL command to be spawned when a DECwindows test is recorded or executed. This command can be used to invoke applications for inclusion in the test.
- Comparison type: identifies the comparison type: screen, record, or character.
- Filters: identifies one or more filters to remove run-timedependent information from the result file.
- Remark: identifies a comment that you add to the history.
A collection is a snapshot of specified test descriptions and the VSI Digital Test Manager library as they exist at the time you create the collection. You must organize tests into collections before you can execute them to produce result files for comparison.
When you execute a collection, VSI Digital Test Manager sets up the test environment and executes all the tests in the collection (this can be done in batch mode). Each test in a collection generates a separate result file. The result file contains the output generated by the template file. The result file is used for comparison against a test's benchmark file.
When you execute a collection, VSI Digital Test Manager compares the test results with the benchmark file for each test that has been run. If the comparison is unsuccessful (differences are detected), VSI Digital Test Manager creates a difference file. If the comparison is successful (no differences are detected), the result file is deleted and no difference file is created.
You can then review the differences using the Review subsystem, which gives you access to the results obtained by executing the collection, as well as other information about the collection. It also enables you to invoke the Performance and Coverage Analyzer (PCA) to gather performance and coverage data for the test. Section 5.2 describes the Review subsystem. Using VSI DECset for OpenVMS Systems describes using VSI Digital Test Manager with PCA. You must execute and compare a collection before it can be reviewed.
In a DECwindows environment, you can review test results by direct manipulation in the collection view. However, if you need to update benchmark files, you must pull down the Testing menu, choose the Review menu item, and choose the Open menu item to lock the collection so only you can perform an update operation.
