Track 4. Implementation & Testing
Health data standards must be implemented into software systems and workflows to provide a new or enhanced functionality. Assessing the feasibility and value for these enhancements requires piloting software in increasingly realistic settings.
Both implementation and testing are necessary for each use case, standard, and developed software. Furthermore, it is an opportunity to encourage community members to become actively involved in project efforts and attract new organizations by through demonstrating value in increasingly realistic settings. For work to progress effectively, activities and necessary resources must be included in the use case plans, and prepared for as the initiative moves forward.
The rest of this section covers experience gained in (a) Implementation, (b) Testing, and (c) Feedback on the previously discussed Use Case & Planning and Standards Development tracks.
Implement FHIR IG(s) into Systems and Workflows
Every use case requires implementing at least part of a FHIR IG into software to support the workflow to be piloted. It is highly recommended to engage market-leading software vendors in the specialty space as these vendors typically have prior experience testing, a customer base that may be interested in new approaches, and credibility that will attract others. Given their important role and valuable experience, vendors should be engaged as early as possible in the community's use cases.
If in the early phases engaging leading vendors is not possible, implementation into prototypes and/or testing software may be a satisfactory first step to allow useful assessments.
Questions that should be addressed during Implementation include:
- Is the IG understandable based on technical specifications and documentation?
- Is the IG implementable?
- Is the system within which the IG is implemented consistent and interoperable with other systems with which data exchange would be needed?
- Is the workflow represented within the use case and IG realistic and consistent for both developers and end users?
In all cases, it is critical to receive comments and recommendations for change that can be taken back to the FHIR IG and use case teams so that adjustments can be considered.
Test and Pilot
Testing and pilots within use cases tends to start simply and become increasingly realistic, via multiple phases. Testing of the implemented IG should start as early as possible, even if on a subset of the future functionality.
The following table is a simplified set of testing axes that aim for a higher degree of realistic challenges from the top of the table to the bottom. A particular test or pilot might combine components from different rows, and not just across a single row.
| Difficulty | Use Case Workflow | Software | Data | Participants | Environment |
|---|---|---|---|---|---|
| *Synthea is an open-source, synthetic patient generator. With no cost and no restrictions, Synthea has empowered multiple CodeX pilots as well as testing and research around the world. | |||||
| Lower | 1 data element via 1 exchange in the workflow | Simple test scripts | Synthetic data* | Developers | Developer's computers |
| Higher | More elements and exchanges | Prototypes of needed components of future software systems | De-identified data | Some developers and intended users | Connectathons with multiple organizations |
| Highest and Most Realistic | All required data elements via all envisioned exchanges in the workflow | Software systems intended for deployment and adoption | Real patient data | Envisioned end users/actors from multiple entities of each type of organization required by the workflow | Full-scale real-world pilots within the envisioned organizations (e.g., health system, lab ...). Likely includes some testing sandboxes in those organizations |
Note that the use of real patient data (bottom row) involves patient and institutional consent, appropriate privacy and security protections, and should be planned for many months before testing aims to start.
Early testing provides early feedback and engages key community members in activities that bring the use case concept and IG closer to life.
CodeX use cases have found it useful to start with relatively simple tests that can be run 6-12 months after the start of the project, including components similar to the first row in the table above. Over time, CodeX use cases, for example the Radiation Therapy Treatment Data Case Study, typically ramp up through increasingly ambitious tests to real-world pilots. These CodeX use cases include components similar to the bottom row: e.g., demonstrating the full use case workflow, using software systems intended for sale or otherwise deployed, using real patient data, driven by the intended users, and within many organizations across which the future use case is planned to operate.
Obviously, these real-world pilots require substantial planning, commitment, preparation, and legal/regulatory steps, well in advance of pilot execution. This advanced planning is particularly important for testing within real-world healthcare settings and when working with real patient data.
For each of the tests and pilots have considerations that should be finalized when initial planning is complete, especially the commitment of the necessary resources, including:
- Objectives
- What improvements in health care and research will the use case provide? (e.g., What will be made better, faster, less expensive, more accessible?)
- Test Configuration
- What parts of the use case workflow are being tested, and where are they being tested?
- Resources
- Skill sets (e.g., project management, clinical, technical, logistical, legal, operational, and other skills).
- Software (e.g., EHRs, specialty systems, payer systems and others).
- Facilities (e.g., health system, labs, homes, registries, and others).
- Funding (e.g., for facilities and expertise)
- Success
- What metrics are available to assess current capabilities?
- What level of improvement will be measured during the test?
- How will success be measured?
Feedback to Update Previous Work, As Needed
Even early phase pilots can provide valuable feedback on how implementable the IG is in software, the usability of the updated systems, and the costs, human burden, and potential benefits of the new workflow. Further, pilots can identify the need for project update and necessitate further testing on the revised software, hence early testing is critical to quickly developing valuable software for the community.