Calculator Testing
Server Calculator Testing is a process that simulates the running of a calculation through the server environment. This is a useful tool that allows you to verify new calculation code by executing a large number of calculations. As part of the calculation run, the process will compare the XML output and the audit report that it produces against previously run output to check for discrepancies.
To utilize the server calculation testing process you will need a plan Repository File, input XML file(s), a text file, and have set [SERVER] HIRTDIR, and if appropriate, [SERVER] RegPath and [SERVER] SummPymtMnthly, in the PROADMIN.INI file. The plan repository is the file that contains the plan rules for your benchmark cases. The input XML files are configured the same way as input XML files that would initiate a calculation through the server engine. The text file indicates the location and name of the plan repository, the location of the XML input file(s), the name(s) of the XML file(s) that will be run and the location that the output results are written to.
The text file, containing the information regarding the locations of the repository and XML files, has the following layout:
repfile=directory_name\plan repository name or rf=directory_name\plan repository name
This is a required line that controls which plan repository to use and where it is located.
diffile=directory_name\diffs file name or df=directory_name\\diffs file name
This is an optional line that controls where the audit report differences are written (if you are using the audit report and differences are found). By default, the audit report differences are written to separate files named "input filename"_DIFFS.txt, but by using diffile or df, all the audit report differences are written to a single file. (If the file already exists, differences are appended to the existing file; you are responsible for deleting these files.) You can specify multiple differences files in the script file. They take affect as they are sequential encountered within the script, but you can't specify the diffile or df within a "in=" section. You can turn off the difference file setting by setting diffile or df to an empty or blank string (e.g., df=), and then you can turn it back on by setting it to a new or existing file name.
pathin=directory_name or pi=directory_name
This is a required line that controls which directory the input XML documents are located.
pathout=directory_name or po=directory_name
This is an optional line that controls the directory where the audit file and the xml documents need to be written to. If this line is omitted, the output is written to the pathin= directory by default.
in=name1.xml
in=name2.xml
...
These are the names of the xml files that will be used to execute each calculation. Each contains the participant data, assumptions, decrement date(s), commencement date(s) and other information for selecting and running a system plan in the repository.
INALL
Instead of listing all the names of the xml files in the pathin directory (using in=), you can use the INALL option. Now, every file in the pathin directory with an xml file extension will be used.
The pathin and pathout statements are at the beginning of the list of input XML files and can be repeated to change the location of these files. This gives you the flexibility to run calculations from different directories as part of one calculation process.
Once you have created the text file, you are ready to begin. To start the process, click on Server Maintenance | Calculator Testing under the File menu to bring up the Server Calculator Testing dialog box:
Test Script File: presents a box where you can type in the location and name of the text file, or click on the Browse… button to select the file. To facilitate modifications to the script file, there is also an Edit Script button that opens the script in your text file editor to allow you to edit it and save any changes. The Browse begins in same folder as Script File checkbox, if checked, may facilitate your search for a new script file.
The Recent Test Script Files: box lists the Client name, Last Run date and Script File name of all of the recently executed script files. To re-run one, double click on it's line and it will be placed in the Test Script File box.
Output: is a multi-choice field for selecting the type of output you need: XML & Audit Report, XML Only, or Audit Report Only. XML indicates that an output XML file will be provided for every input XML. Audit Report indicates that individual calculation results will be provided, in summary form, as a text file for every calculation that completes.
Stop after ____ errors (leave blank for no limit) allows you to set a limit to the number of calculations that abort before the server calculation testing halts.
The Minimize false differences comparing to prior versions (or initialize option to do so) checkbox allows you to choose between two (2) different methods of comparing old and new results:
If the box is unchecked, ProAdmin compares the XML output and audit reports created in the first (or baseline) run to the XML output and audit reports generated in the second run. The reports generated in the second run are only saved to file (as _out2.xml, _audit2.txt and diffs.txt) when differences are found between the first and second runs. The _diffs.txt file contains just the differences between the _audit.txt and _audit2.txt files.
This methods work great unless your baseline was run in an older version of ProAdmin. In this situation, many of the differences are immaterial (e.g., labels changed, new columns added, additional information is now output, etc.). This can make it hard to find material differences (e.g., a benefit or payment form amount changed).
When the Minimize false differences comparing to prior versions (or initialize option to do so) checkbox is checked, ProAdmin stores all the results used to create the baseline (first run) XML output and audit reports in a file that ends with the phrase "_AUDITRES.DEFLATE" (one file for each person). The XML output and audit reports are still created in the baseline run, but they are not used by ProAdmin -- they are there to allow manual comparisons. During the second run (when the _AUDITRES.DEFLATE files exist), the first run's XML output and audit reports are re-created on-the-fly from the results stored in the _AUDITRES.DEFLATE file and compared to the XML output and the audit reports normally generated during the second run. This comparison of "new format to new format" reveals only material changes in results. Again, the _out2.xml, _audit2.txt, and _diffs.txt are created when differences (now just material differences) are found between the first and second runs.
This new methods works great when your baseline was run in an older version of ProAdmin. Using this methodology, the first run's XML output and the audit reports are re-generated from the previous results stored in the _AUDITRES.DEFLATE file using the current version of ProAdmin, so now the first run's reports will have the new labels, new columns, new additional information, etc.), minimizing the immaterial differences. However, this method uses more storage space (because of the _AUDITRES.DEFLATE file), and both the first and second calculations may take longer to process (depending on network traffic and speed).
NOTE: A run is a second run only if the _audit.txt, _out.xml, or _AUDITRES.DEFLATE files already exist.
After you have entered the information about the input file, click the Run button. This begins the execution of the calculation(s). At the completion of the each calculation, the process will check (depending on the Minimize false differences comparing to prior versions (or initialize option to do so) checkbox) to see if an _AUDITRES.DEFLATE, output XML file and/or audit text file exists. If the process finds a match, it will compare the new files against the existing files to check for any discrepancies. If the process cannot find a match or if there are discrepancies, it will create new file(s) and move on to the next calculation. At the completion of the final calculation listed in the test script file, the process will display a report summarizing the results of the run.
The testing result format is as follows when _AUDITRES.DEFLATE files are not used:
ProAdmin Testing Results: Dec 4, 2003 8:51 PM
Software timestamp: Nov 25, 2003 1:11:33.850 PM
User: Mary Lincoln
Number of calculations: 4
Average time per calculation: 0.03
Number of application aborts: 1
Number of user defined aborts: 0
Number of XML file differences: 1
Number of Audit Report differences: 1
Number of repository loads: 1
Average time per load: 0.61
Total time: 0:01
Repository: c:\ProAdmin_Files\Testing\Repositories\complete_file_070102.sf
Calc: 10_49_37_055_AM.xml completed.
XML output matched
Audit Report output matched
Calc: 11_46_37_055_AM.xml aborted, check XML output for message..
XML output differed, new output written to:
c:\ProAdmin_Files\Testing\Output\11_46_37_055_AM_out2.xml
Calc: 14_57_37_055_AM.xml completed.
XML output matched
Audit Report output matched
Calc: 15_59_31_034_AM.xml completed.
XML output differed, new output written to:
c:\ProAdmin_Files\Testing\Output\15_59_31_034_AM_out2.xml
Audit Report output differed, new output written to:
c:\ProAdmin_Files\Testing\Output\15_59_31_034_AM_audit2.xml
If you are using the _AUDITRES.DEFLATE file method, your comparisons will indicate that the deflate file was used, for example:
Calc: 10_49_37_055_AM.xml completed.
XML output matched (_AUDITRES.DEFLATE)
Audit Report output matched (_AUDITRES.DEFLATE)
Calc: 11_46_37_055_AM.xml aborted, check XML output for message..
XML output differed (_AUDITRES.DEFLATE), new output written to:
c:\ProAdmin_Files\Testing\Output\11_46_37_055_AM_out2.xml
And during the baseline run:
Calc: 10_49_37_055_AM.xml completed.