INTRODUCTION 1 2 The DICOM Validation Tool (DVT) test framework is a flexible architecture and uses service–object pair (SOP) class definition files, making it adaptable as the DICOM standard evolves. It includes a graphical user interface (GUI) and a command line interface, DICOM media validation, service class user (SCU) and service class provider (SCP) emulators, and a rich scripting language. For more advanced testing, VBScript (Microsoft Visual Basic Scripting edition; Microsoft; Redmond, WA) or JScript (Microsoft) can be executed by DVT, and a set of .NET assemblies is available for developing stand-alone test tools with languages such as Visual Basic.NET (Microsoft) and C# (Microsoft). If a validation tool is to be taken seriously, it must be vendor neutral. With the release of DVT 2.1 in 2005, DVTk now exists as an open-source community project. Not only does this encourage more vendors to contribute (because of the incentive of reduced development and integration costs), but the adoption and proliferation of standards make it easier for individual developers to contribute to open-source projects such as DVTk. Independent software developers are sometimes discouraged from proceeding because of the chance that their work will provide a “one-off” solution only and not be widely used. DVTk, with the backing of the mature DICOM standard, is attracting talented developers with a desire to create something useful and lasting. This success assures that time devoted to learning this toolkit will not be wasted. This article focuses primarily on the DVT main application. We provide background on the toolkit and specific examples about the two intended roles of DVT: service and development. Current efforts within the DVTk project and future directions are also highlighted. DVTk HISTORY AND ARCHITECTURE In 2000, Agfa HealthCare (Mortsel, Belgium) and Philips Medical Systems (Eindhoven, The Netherlands) decided to coordinate activities around DICOM validation testing by bringing together efforts already started by both companies under a joint DVT project. The intention was to produce a DVT that could not only be used internally by both companies to test their own products but also made freely available to other original equipment manufacturers (OEMs) as a means of testing their products to the same level of detail. The ultimate aim was to reduce the time spent integrating proprietary systems by first exposing these systems’ equipment to tests run using DVT. DVT project has a steering committee with responsibility for guiding legal, technical, and commercial aspects. The steering committee meets every 6 months to discuss past progress, current issues, and future requirements. A project manager was elected by the steering committee to manage the DVT project on a daily basis and report back to the committee. Development tasks were divided up based on the available skills of developers who report to the project manager. In its first years, Agfa and Philips provided the personnel to staff the DVT project. http://www.dvtk.org http://sourceforge.net/projects/dvt, SOURCEFORGE.NET The dvtk.org Web site is now the location for the latest downloads, defect tracking details, and forums on using DVTk. Interested individuals and companies may join the project through the Web site. A weekly project telephone conference coordinates the activities of the development team. 1 Fig 1. DICOM Validation Framework.  For more advanced testing scenarios, DVTk includes the Script Support library and the High-Level Interface (HLI) library. The HLI library is a newer abstraction built on top of the core that makes it easy to write multithreaded tests. The application program interface (API) exposed by this library encapsulates many of the low-level core API classes and methods that make writing VBScripts similar to writing scripts in DVTk’s native DICOMScript language. VBScripts can be executed entirely within the DVT GUI application or command-line executable or debugged in Visual Studio .NET. Other DVTk-based applications built on the core and currently available on dvtk.org include: DICOM Network Analyzer, a network sniffer and DICOM protocol analyzer; DICOM Editor, for displaying and editing DICOM files; DICOM Compare Tool, for comparing the attributes and values of two DICOM files; DICOM Attribute Validator, for validating DICOM files, including Structured Report objects, against definition files; DICOM File Stripper, for removing all but mandatory attributes from a DICOM file; and DICOM File Anonymizer, for removing patient and physician information from a DICOM file. Medical Communications (UK) provided the underlying Transmission Control Protocol/Internet Protocol (TCP/IP) Capture File to the DICOM protocol data unit (PDU) conversion utilities used by the DICOM Network Analyzer application. 3 GETTING STARTED WITH DVTk OUT OF THE BOX http://www.dvtk.org Starting DVT presents the user with an empty workspace. DVT is designed to work on a single project at a time, although multiple views of the project may be opened from which multiple tests may be simultaneously executed. A DVT project is a container for one or more test sessions. A session is a container for the configuration of one or more tests to be performed against a system under test (SUT). Project and session configuration properties are stored in flat files. The DVT GUI exposes most of the configuration properties, although some of the more advanced settings, such as STRICT-VALIDATION, are available only by directly editing the session file. Some settings can also be modified in script files. For example, the STRICT-VALIDATION script command overrides the value of the STRICT-VALIDATION session property. Some settings, such as CALLED_AE_TITLE, also have built-in default values that are assumed if the property is not defined in a script or session file. Session files come in three types: emulator, script, and media. Emulator sessions are used when DVT should act as an SCU or SCP emulator to test the DICOM transfer syntax. The emulators have support for Verification SCU/SCP, Storage SCU/SCP, and Print SCP. Script sessions are used when DVT is used to execute a DICOMScript, DICOMSuperScript, or VBScript. A DICOMSuperScript is simply a script that calls one or more DICOMScripts. Script sessions have support for network SCP/SCU message exchange and DICOM media file creation. Media sessions are used to validate the DICOM file format as a DICOMDIR and/or DCM media files. When validating a DICOMDIR file, any referenced files are also validated. An easy way to gain familiarity with DVT is go through the examples in the subfolder using DVT as both the test tool and the SUT. Opening up the example project displays a tree view of all test sessions defined in the project and provides details on the session currently selected in the tree. Clicking on the Window menu and selecting the New Project View and Tile item displays two views of the project. 2 Fig 2. DVT-Worklist Management SCP and SCU test sessions.  At the top of the Session Information tab are general settings, including the session type, session ID (used in uniquely naming results files), and location of files used/created by the test session. The DVT Roles Settings section defines the Application Entity (AE) title and some connection settings that DVT assumes during the test session. The SUT Settings section defines the AE title and some connection settings, including the TCP/IP address, of the system the session is testing. Because this example uses DVT as both the testing system and the SUT, the DVT role settings in the top view are synchronized to the SUT settings in the bottom view, and the SUT settings in the top view are synchronized to the DVT role settings in the bottom view. Locations of the SOP class definition files required for the test session are selected under the Specify SOP Classes tab. The selected definition files are loaded into memory when the test session starts, and DVT uses these to validate the DICOM messages and objects it sends and receives. A definition file describes a single DICOM SOP class in terms of the combination of DICOM Message Service Element (DIMSE) commands and Information Object Definitions (IODs) that make up the SOP class. In this case, both test sessions specify the Modality Worklist Information Model–FIND SOP Class item. The standard definition files that come with DVT in the definitions subfolder are taken directly from the DICOM standard parts 3 and 4. Private definition files can be made that extend the validation capabilities of DVT by copying one of these standard files and modifying it with private user IDs (UIDs), modules, and attributes. A typical customization might specify that when a particular SUT is known to always send a value for the Patient Name (0010,0010) attribute, the corresponding definition file can be modified to define the Patient Name attribute as a type 1 (mandatory, nonzero-length) attribute. The received object is first validated against any loaded definition files. This step ensures that the correct attributes are present in the object and that they are encoded correctly. This step is optional and applied only if a reference object is present in the script. Checks made here are that the expected number of attribute values has been received and that each attribute value matches the corresponding reference object attribute value. http://www.dvtk.org Following a successful association validation and negotiation, the SCU script SENDs a C-FIND-RQ message of type Modality Worklist–FIND. The SCP script RECEIVEs the C-FIND-RQ, validates it, and proceeds to SEND three hard-coded C-FIND-RSP messages. The SCU script was written to mirror the SCP script, so the three reference C-FIND-RSP messages it defines match exactly with those sent by the SCP. These scripts also demonstrate two DVT scripting features: Value Mapping and Value Representation (VR) Keywords. Value mapping allows the user to substitute a label for a value defined in a script, generated by DVT, or received from the SUT, and then refer to that value with the label further down in the script. The LABEL: keyword is used to map an attribute value defined in the script or received from the SUT. The NEW: keyword tells DVT to generate a new value and assign it to the given label. An example of this is in the SCP script, where the Study Instance UID attribute (0020,000D) is assigned a new value generated by DVT and named with the label StudyInstanceUid1. The generated UID value can be referred to subsequently in the script as StudyInstanceUid1 (although this was not done in this example). VR keywords, such as the AUTOSET keyword used in both the SCU and SCP scripts, tell DVT to generate a value and assign it to the corresponding attribute. In the example case, AUTOSET is used in the Scheduled Procedure Step Start Date attribute to ensure that values match between the two scripts. Both Value Mapping and VR Keywords are sensitive to the type of attribute to which they are applied. This test is begun by right clicking on the SCP script and selecting execute. When DVT is executing a test session, all tabs except the Activity Logging tab are hidden, the session tree in the corresponding view is disabled, and the test stop button in the toolbar is enabled. The SCP’s Activity Log will indicate that it is waiting for a connection on port 104. Executing the SCU script in the lower window will result in a number of messages in the Activity Logs, followed by the termination of the test sessions. Comparing and correlating the SCP and SCU activity log entries provides a good visual reference for the DICOM message flow between two AEs. By default, the AUTO-TYPE-2-ATTRIBUTES session property is set to true in the session file, which means that DVT will automatically add any zero-length type 2 (type 2 attributes must be included; however, they may be encoded with a zero-length value or no value) attributes from the definition file to the dataset before sending to the SUT. This behavior is recorded in the SCU’s activity column by the “Automatic Type 2 Attributes population...” statement. This feature means that only type 1 (required, nonzero-length) attributes must be explicitly stated in the scripts. After completion of the scripts, each view displays the Validation Results tab, where the results of the test sessions are displayed. DVT stores the results of each test in a _res.xml file in the test session’s configured results directory. The file name takes the form <Detail∣Summary>_nnn_<scriptName>_res.xml, where nnn is the session ID from the session properties. Varying the session ID from one test execution to the next allows storage of multiple sets of results in the results directory. Two session Boolean properties define whether summary or detailed results are generated: SUMMARY-VALIDATION-RESULTS and DETAILED-VALIDATION-RESULTS. If the test session’s STORAGE_MODE property is set to as-media or as-dataset, any media files received by DVT during the test are also stored in the results directory, again using the session ID to help uniquely name the files. The generated results files are listed in the session tree under the script name. Selecting one of the results files displays it in the Validation Results tab. The Validation Results tab is a Hypertext Markup Language viewer allowing navigation between the summary results, detailed results, and any generated media files using hypertext links. DVT does not include a DICOM file viewer. To automatically view a generated .dcm file by clicking on a link in the Validation Results tab, a DICOM image viewer must be installed and associated with the .dcm file type. The SCP’s Validation Results tab will show a test result of PASSED, and a .dcm media file is created that contains the Modality Worklist–FIND dataset received in the C-FIND-RQ message sent from the SCU. The SCU’s Validation Results tab also shows a result of FAILED, with nine errors reported. The Summary Results File lists the errors; in this case, three type 1 attributes are missing from each C-FIND-RSP message sent from the SCP. Clicking on a Link to Detailed Result link displays the errors in the context of the complete C-FIND-RSP message. The detailed results file contains the complete contents of each message sent and received in chronological order. Any comment lines in the DICOMScript that begin with ## are copied directly to the detailed results file. This is a good way to insert additional test information directly into the results. For additional troubleshooting output in the Activity Logging tab and detailed results file, the LOG-RELATION, LOG-DEBUG, LOG-DULP-STATE, and PDU-DUMP test session properties can be enabled. In the example described here, it is left as an exercise for the user to add the missing type 1 attributes to the SCP script so that the SCU test session result becomes PASSED. DICOMSuperScripts (script files with a .dss extension) enable the reuse of DICOMScripts in various test scenarios. The storage example script sessions included with DVT demonstrate the benefits of DICOMSuperScripts. DVTk FOR SERVICE TROUBLESHOOTING As demonstrated in the previous section, DVTk can be a useful tool when troubleshooting modality worklist problems. Another common use case would occur when adding new modalities to a PACS network. It can be frustrating to ensure that AE titles, ports, and IP addresses match between the modality and PACS configuration. This section will illustrate ways in which DVT can be used to troubleshoot a PACS-modality interface problem and describe another, more service-orientated tool available from DVTk.org. http://www.dcm4che.org, dcm4chee-2.x If the verification test succeeds, then one can assume the patency of DICOM network connectivity between the modality and PACS. The next test is to attempt image storage to the PACS. This test requires a set of DICOM test images, preferably from the modality that is experiencing the problem. The images should contain no names, IDs, or UIDs that will conflict with real-world data. It is important, however, that the data used to perform the test matches the real-world data in structure as closely as possible, including any private data elements the modality creates. A copy of the modality’s DICOM conformance statement comes in handy here. This will help simulate the behavior of the modality as closely as possible. Under the Specify Transfer Syntax (TS) button on the Session Information tab, all transfer syntaxes supported by the modality as a storage SCU can be selected. For example, to test a computed tomography (CT) modality, one would select the CT Image Storage SOP Class (1.2.840.10008.5.1.4.1.1.2). For this test, the storage SOP class and transfer syntax that match the DICOM test images that will be stored to the PACS should be selected. Note that this step is not actually necessary; when emulating a storage SCU, DVT will automatically add to the association negotiation the transfer syntaxes and SOP Classes from the DICOM media files being stored. Selecting the correct transfer syntaxes and SOP Classes is necessary when emulating a storage SCP, however. Executing the Storage SCU Emulator opens the Storage SCU Emulator dialog box. In the dialog box, the Add button is used to add the test images to the list that will be sent to the SCP. Selecting the Validate before export option tells DVT to validate the media files against the corresponding definition file prior to sending them. The number of associations the modality uses when sending multiple images is also specified here (most modalities would send multiple images on a single association). Finally, clicking the Send button will tell DVT to execute the storage test and close the dialog. DVT’s detailed results will show the verification of each media file followed by each storage transaction. The Storage SCU emulator does not automatically do storage commitment as the SCP emulator does. To simulate the modality performing a storage commitment request, one could create a DICOMScript session similar to the Commit_SCU example session, modifying the SOP instance UIDs to match the images stored. This test should be performed immediately after the storage test. To demonstrate some of the more advanced DICOMScript features and the Data Warehouse in DVT, one can imagine a scenario where a CT modality has recently undergone a software upgrade. Since the upgrade, technicians have been reporting that although they can successfully postprocess images on the modality before sending them to the PACS, processing of images retrieved from the PACS back to the modality fails. No errors have been reported by the PACS or modality on storage or retrieval. One possible reason is that the PACS is doing something to the stored images that is preventing the modality from processing them. To evaluate, one can perform what is called a PACS transparency test. The idea is to make what the PACS is doing (if anything) to the images transparent by comparing the before and after images. This test will require a DICOM CT image file from the modality on the DVT workstation and a DVT project with two sessions. A script session playing the role of SCU is used to put the image in the DVT Data Warehouse, store it to the PACS, and initiate a C-MOVE from the PACS back to DVT. An Emulator SCP session is used to receive the image moved back from the PACS and for performing the validation of the received image against the original image in the Data Warehouse. 3 4 5 Fig 3. PACS transparency test using DICOMScript-reading image into the Data Warehouse.  Fig 4. PACS transparency test using DICOMScript-exporting image from the Data Warehouse.  Fig 5. PACS transparency test using DICOMScript-moving the image back to DVT.  This test requires that the PatientRootQueryRetrieve-MOVE.def definition file is loaded in the script session and the CTImageStorage.def definition file is loaded in the Emulator SCP session. If, as suspected, the PACS is modifying the CT Image object before returning it to the modality, the Emulator SCP results data will identify where these modifications occurred. In this case, the hypothetical tester notices that DVT is indicating that a set of private attributes in the Data Warehouse copy of the image are missing from the image returned from the PACS. Upon further investigation, it turns out that the software upgrade performed on the modality included the addition of a set of new private attributes required by the modality to perform new postprocessing algorithms on the images. The PACS does not support these new private attributes for some reason and is stripping them from the image objects. Instead of using a combination of a DICOMScript test session and Emulator test session, this test could have been implemented with a single VBScript session using the HLI library. In this case, separate threads would be created for the SCU and the SCP parts. This approach is left up to the user as an exercise. 6 Fig 6. DICOM Sniffer & Analyzer tool in capturing mode.  7 Fig 7. DICOM Sniffer & Analyzer tool in analyzing mode.  Two more tools built on the DVTk framework that should be in the imaging professional’s toolbox in working with DICOM on a regular basis are DICOM Compare and DICOM Editor. DICOM Compare can compare the attributes and values of two DICOM files. It can also filter out of the compare process any attributes and sequences to make it easier to identify offending elements. A known “good” DICOM object can be compared to one causing a problem that was captured from the network with DICOM Sniffer. With the DICOM Editor, one can add/delete/modify any attribute or sequence and sequence item and save the modified DICOM file to any location. This tool goes hand-in-hand with the DVT, DICOM Sniffer, and DICOM Compare tools. After those tools have identified the DICOM elements that may be causing a DICOM object storage problem, the editor can be used to manually fix the DICOM object and then resend it to the system where the failure is occurring. If the store succeeds, chances are the source of the problem has been identified. DVT AS A DEVELOPMENT TEST TOOL Another use of DVT is automated unit or system testing in a software development environment. Most modern source code management systems include automated build-and-test subsystems. Scripts can be created to test the DICOM interfaces of a vendor’s product, and the command line version of DVT can then be launched to execute those scripts against the product as part of the normal automated build-and-test processes. The DVT results and output files can then be saved along with the rest of the build/test artifacts. The command line version of DVT can be called on a DICOMScript as follows: dvtcmd Modality_System_Test.ses Modality_System_Test.ds. The summary and detailed results Extensible Markup Language files output by DVT make it easy for an automated test system to determine the results of the test, generate a report, and take any other appropriate action, such as e-mailing the development team. A tool as programmable and rich in output as DVT is valuable to the entire iterative development process. Software validation teams can link test session files, scripts, and result files to software defect issues, so that a developer can then use them to reproduce the defect. A validator can subsequently use those same scripts to test the fixed software. Maintaining a repository of DVT scripts is also an excellent way to run regression tests. 4 DVTk makes it quick and easy to perform repeatable, detailed testing of DICOM interfaces. In the following example, we posit a stress test to run against a storage SCP service. Because this is a stress test, the test application must hit the SCP simultaneously from multiple SCUs, so that multiple threads will be needed. Writing multithreaded Visual Basic (VB) test applications is a breeze with the new HLI library. In fact, it looks quite similar to a DICOMScript, with high-level abstractions for creating and releasing associations and for sending and receiving messages. This example using the HLI requires DVTk alpha version 2.1.007.006 or greater , which can be downloaded from dvtk.org after registering as a Plus User. Plus Users have access to early releases and draft documents, such as the draft version of the HLI API help file. 8 Fig 8. Multithreaded stress test using VB.NET-main subroutine and imports.  9 Fig 9. Multithreaded stress test using VB.NET-MainDicomThread class.  10 Fig 10. Multithreaded stress test using VB.NET-CtStorageScu class.  11 Fig 11. Multithreaded stress test-activity logging.  RECENT EXTENSIONS AND FUTURE WORK 5 In the future, the aim is to enhance the capabilities of the DVTk IHE Actors framework by supporting additional functionality in the existing actors and supporting other actors needed in the various IHE Integration Profiles. Support for other protocols is also envisioned. The DVTk development team is also involved in the new IHE Connectathon Toolkit (code-named “Gazelle”), which is being developed as a successor to the MESA Toolset. It is hoped that the DVTk DICOM validation engine can be wrapped as a Web service for use in Gazelle. Various new tools will be developed using the DVTk frameworks. A GUI for the DVT-IHE framework; various new emulators, such as a radiology information system emulator and modality emulator; and stand-alone validation applications are likely candidates. The current number of development resources limits what can be done—anyone wishing to contribute can do so via the Web site. The aim for future IHE extensions is to integrate any other validation services into the DVTk framework as Web services. This may include Cross-Enterprise Document Sharing validation, for example. It is hoped that the Gazelle cooperation will result in web services that can be reused for this purpose. CONCLUSION Originally developed by Agfa and Philips to test their products, DVTk has grown into a professionally managed, open-source, vendor neutral, DICOM Validation Framework. Flexible for the changing standard, programmable, and extensible, DVTk is a powerful tool for anyone working with the DICOM standard. The toolkit includes GUI and command line versions of the main validation application, DVT, and a collection of .NET libraries for creating new validation and test tools. The large collection of example validation sessions that come with the toolkit are a great place to start for understanding how the DICOM standard works in practice. Hospital IT staff can use DVTk for simple tasks, such as pinging the network for the existence of modality AE titles, or for more detailed troubleshooting, such as checking for the presence of specific DICOM attributes and values in messages and media files. Other DVTk-based tools are available for tasks such as editing or comparing DICOM files or for capturing and analyzing messages on a live DICOM stream. The XML-structured test results and media files created by the validation framework provide great evidence for IT staff when discussing a problem with vendors. Developers and testers can create scripts for testing their products DICOM conformance, or build on the framework by creating new standalone test tools. NUnit integration and the command line version of DVT make it possible to incorporate DVTk and the generated test results into automated build-and-test systems. Current and future efforts include support for IHE actor validation, including HL7 validation, and new device emulators, such as a Radiology Information System (RIS) emulator. As the digital hospital enterprise continues to grow, DVTk is well positioned to take on these new validation roles. We encourage the reader to continue working with DVTk as a means to master the changing DICOM environment. With the move to an open-source community approach, DVTk is well on its way to becoming the independent gold standard for DICOM interface testing.