Peer [HL1] review of clinical information models: a Web 2.0 crowdsourced approach
Heather Lesliea,b, Silje Ljosland Bakkea,c
a Co-lead, Clinical Modelling Program, openEHR Foundation,
b b CMIO, Ocean Health Systems, Australia
c Information Architect, Nasjonal IKT HF, Norway Norway
Abstract
Over the past 8 years the openEHR Clinical Model program has been developing a Web 2.0 approach and tooling to support the development, review and governance of atomic clincial information models, known as archetypes. This paper describes the background and review process, and provides a practical example where cross standards organisation collaboration resulted in jointly agreed clinical content which was subsequently represented in different implementation formalisms that were effectively semantically aligned. The discussion and conclusions highlight some of the socio-technical benefits and challenges facing organisations who seek to govern atomic clinical information models in a global and collaborative online community.
Keywords:
Informatics, crowdsourcing, common data elements
Introduction
Historically, most collaboration in the health technology domain has been through formal balloting of message or document specification standards within standards development organisations (SDOs) such as ISO TC215 or HL7 International. This approval process has had some significant success over many years in supporting interoperability of health data, however this approach is not transparent, responsive or agile enough for development, maintenance and governance of larger numbers of more atomic clinical information models.
...
This openEHR methodology is now in use by a number of national and jurisdictional eHealth programs around the world who are using archetypes published in this manner to underpin their local health IT infostructure.
Background
...
The Clinical Models Program at the openEHR Foundation is responsible for management of a set of clinical information models, known as archetypes, on behalf of the international openEHR community. The scope of responsibility includes:
...
There are other communities using separate CKM instances in Norway, Australia, United Kingdom, Slovenia, Canada and Brazil. These groups actively share archetypes and collaborate together to minimise ‘reinventing the wheel’ and [BSL3]
openEHR peer-review process
A small number of Clinical Knowledge Administrators are appointed to take responsibility for the operations of CKM as a a whole. They appoint Editors who are charged with the responsibility to develop and enhance the clinical content of each archetype from its initial draft through to a published life cycle state. The CKM tool support this iteration by enabling the Editors to run a series of review rounds to gather and collate reviewer feedback and manage the associated version and audit controls.
...
All review comments plus the responses of the Editors to each reviewer comment is captured and made available to any other registered use. In this way, the process is transparent and the Editors can be held accountable to the community.
Approach
The following narrative outlines the openEHR approach by describing a complex collaboration between the openEHR and HL7 communities, using the recent experience of cross SDO review of clinical content for representation of Adverse Reaction Risk, also referred to as Allergy/Intolerance within the HL7 community. The narrative has been pieced together retrospectively from audit train and review date recorded in the respective CKM tools.
...
An independent FHIR resource evolved throughout the CKM review process, adopting the changes agreed through the review process. This was the artefact that was reviewed by the FHIR community. At the time of archetype publication, the content of the archetypes and the FHIR resource were aligned.
Results
...
It has been possible to collate review related data from each of the three CKM instances that have been used as part of the evolution of the Adverse Reaction Risk archetype through to publication.
...
Table 1– Contributors statistics for each archetype
Archetype | Number | Number of reviewers | Number of reviews |
openEHR – initial | 2 | 19 | 26 |
openEHR – openEHR/FHIR | 4 | 38 | 69 |
NEHTA | 5 | 37 | 66 |
Norway | 2 | 32 | 42 |
Total | 13 | 126 | 203 |
Unfortunately the parallel processes in the HL7 ballot processes are not available to include in this analysis. So the actual numbers of contributors will be much larger than indicated by these numbers, but we have no indication of the size of this contribution.
Discussion
...
This Adverse Reaction Risk archetype started its’ journey as the brainchild of a single clinical informatician. It was published with the collective input of over 126 contributors, each contributing according to their professional background and expertise.
There has been no further formal joint collaboration between the openEHR and FHIR communities since the November 2015 publication. At that point in time, the great majority of content in both the openEHR archetype and the FHIR resource were aligned as a consequence of the joint review process.
Figure 1 – Cross SDO Adverse Reaction Risk archetype collaboration process
Subsequently it appears that the FHIR resource has continued to evolve in isolation, effectively diverging from the jointly agreed artefact, due to further requirements being identified in HL7 implementations [23]. This is unfortunate and not the desired outcome that was hoped for by the openEHR Clinical Program Leads, however it does highlight that for successful and ongoing coordination and collaboration on standardisation and alignment of clinical information models, all participants need to keep this as a priority.
...
- Participation is open and free – participation is open to anyone who is willing to participate to the extent of their ability. It is not limited to individuals or organisations who have current paid memberships, who are nominated as ‘experts’, or who have been designated as ‘credentialled’ experts. This may be challenging to many but it supports input from the broadest professions, health domain expertise and geographical sources.
- Everyone can participate according to their expertise. The user interface and review processes in the CKM tool has been developed specifically to ensure that non-technical experts, such as grassroots clinicians, can participate equally alongside the technology savvy. It removes the need for clinicians to acquire additional technical skills in order to participate. All feedback is encouraged, from the smallest grammatical correction through to those who complex informatics or implementation solutions.[BSL6]
- 3. Transparency. All of the activities and decision-making withing CKM is transparent to registered users, including but not limited to:
- Archetype reviews, especially:
- acknowledgement of all participants and their roles;
- clear association between reviewer comments and resulting Editorial decisions;
- number of contributions; number of review rounds; and
- professional and health domain expertise background of all reviewers to ensure that an appropriate reviewer community has been involved.
- Threaded, unmoderated discussion threads;
- Change requests by registered users and Editorial responses; and
- Archetype audit trail.
If a registered user is not happy with decisions there are a number of ways of raising this with Editors or via public discussion boards.
4. Rapid and agile archetype publication. In the work we have done to date, the typical archetype review process involves 4 review rounds to achieve broad agreement on the structure and data points. Sometimes further review rounds are required, usually focussed on refinement of archetype descriptions and metadata. With an average review round duration of two weeks, this means that an archetype requiring six review rounds could potentially be published in twelve weeks. Archetypes based on established and agreed clinical content such as evidence-based
...
scales and scores can often be published in one or two review rounds – corresponding to between two and four weeks.
Assuming modest Editorial resources are available, when multiple archetypes are being reviewed simultaneously it is possible to publish archetypes in an efficent and effective timeframes.
By contrast, the traditional SDO ballot process would not be sustainable in the openEHR environment where the intent is to develop, review and publish all clinical archetypes required for all clinical data recording. There is a practical need for archetype review rounds to be:
- Managed as a sequence of short, frequent review rounds that result in progressively refined iterations of the archetype;
- Initiated independently of other archetypes and for a variety of reasons, including initial publication, management of change requests and maintenance processes; and
- Run when required - sometimes in parallel with other archetype reviews and at other times on an ad hoc basis to resolve a specific issue.
5. Shared archetypes amongst communities. There are now a number of groups using the CKM tool as the basis of national or jurisdictional standardisation of data sets.
The traditional SDO process does not usually reveal the primary authors or contributors to their published standards, although they will possibly be known to SDO members. However the openEHR approach places enormous weight on transparency at every level of governance and for Editors to be accountable to the CKM community:
- Free and open membership;
- Detailed audit trails to ensure accurate provenance and recording of Editorial changes;
- Visibility of reviewer contributions and Editorial responses
- Statistics about the review process, including:
- acknowledgement of all participants;
- number of contributions;
- number of review rounds; and
- background of all reviewers to ensure an appropriate reviewer community.
Conclusion
Clinical information modelling governance has been a new and largely untested challenge until recently – most of our experience in governance of health data standards has been at the complete message or document data set level. The Clinical Knowledge Manager tool was developed directly in response to identification of the need for efficient and responsive iterative refinement of the archetypes in response to identified requirements, especially during implementations – finding the sweet spot in the tensions between governance and evolution to ensure that the information models were safe and fit for use.
Clinical knowledge governance is a complex, evolving and poorly understood domain. Technical governance is one critical aspect but the main challenges were actually socio-technical, related to clinciain engagement and participation, models for collaboration and governance of a set of artefacts that needed to dynamically evolve with a multitude of dependencies governance during implementations.
Disclosure
The Clinical Knowledge Manager tool was developed by Dr Heather Leslie and Ocean Health Systems.
References
[1] Clinical Models Program [Internet]. London: openEHR Foundation; [cited: 2016-12-23]. Available from: http://www.openehr.org/programs/clinicalmodels/
...