openEHR Conformance

Status

Development.

All community input welcome.

Overview

Conformance testing is used as the basis of product certification, and has the following goals:

  • Tendering: enable tendering authorities to state formal criteria for compliance in tenders
  • Protection for solution developers: enables bona fide vendors and other developers to prove the quality of their solutions compared to other offerings claiming conformance
  • Protection for procurement: provides a way of ensuring that purchased solutions can be contractually guaranteed to perform in certain ways

Conformance is tested in two categories:

  • Functional: test correctness of implementation with respect to specifications, represented by specific test sets

  • Performance: test capacity of system to process data and transactions

All conformance specifications, test sets and related materials are part of the Conformance Specification component.

References

Conformance Certificate

An openEHR Conformance Certificate consists of a set of ratings in a number of dimensions and detailed report. The ratings are as follows, derived from the Conformance Schedules below:

  • Functional: 1 | 2 | 3 + O
  • Enterprise: D | M | X
  • Performance & volumetrics: POC | S | L | R
  • Security & Privacy

Evaluation

Conformance testing is intended to be performed by the interaction of a system under test with a well defined openEHR web service that implements the various tests and captures the results.

Conformance schedule - Functional

This category includes tests to determine conformance to base specifications, i.e. things like the Reference Model, querying and so on.

Conformance
level
Test CategoryDescriptionInputsOutputs & Conformance PointsTest setsCapabilities exercised
1Template injection test

End-to-end template => data test.

  • Inject a new template
  • Instantiate from screen form or TDD
  • Export canonical XML

Composition Template, includes:

  • ALL RM types, including all DV_* types, including all reasonable generic derivations, e.g. DV_INTERVAL<DV_QUANTITY> etc.
  • ALL typical compositional hierarchy structures, e.g.
    • Composition → Entry[*]
    • Composition → Section → Section → Entry
    • etc
Regression test exported openEHR canonical XML v test reference XMLTBD: Ian's monster template
  • Archetypes
  • Templates
  • RM
  • Canonical XML
1EHR API lifecycle test

EHR create, commit, read via EHR API.

  • create new EHR
  • Contribution 1: admin Composition (admission)
  • Contribution 2: persistent Composition (meds)
  • Contribution 3: event Composition (vital signs)
  • Contribution 4: update meds + event Composition (vital signs)
  • display & output all Contributions & meta-data as JSON
  • display & output latest version of Compositions in canonical XML
  • display & output all versions of Compositions in canonical XML

Composition Templates

?Contribution JSON v regression JSON

Regression test exported openEHR canonical XML v test reference XML

Template set:

  • admission
  • medications
  • vital signs
  • EHR API
  • Contributions and versioning
2Import - TDD

Data commit using TDD based on provided TDS

  • instantiate TDD from TDS by any method
  • commit TDD
  • export as canonical XML
TDSRegression test exported openEHR canonical XML v test reference XMLVital signs template
2Basic AQL

Test AQL query against single data set (template).

  • Commit vital signs OBS x 50 samples
  • Execute query and return result set XML

Template

Test query

Regression test Result set XML v reference XML

Vital signs OBS template:

  • BP
  • heart rate
  • body temp
  • SaO2

VS Test query

  • Basic AQL
  • AQL result set
3REST API




3Advanced AQLExercise AQL queries with WHERE clause across multiple data sets (templates)

Templates

Test query

Regression test Result set XML v reference XML
  • AQL WHERE processing
OPTIONALDemographics API lifecycle testExercise openEHR demographic content create and retrieve



OPTIONALExport - EHR Extract

Export selected content in EHR Extract form.

TBD





Conformance test design

Conformance testing on a product for a given test category is is performed by:

  • select a specific test set for that category
  • setting up initial conditions, e.g. inject a template
  • executing specific transactions on a given API or other interface (e.g. query)
  • retrieving result
  • performing regression comparison of actual v reference

Conformance schedule - Enterprise

This category contains tests for conformance to functional requirements typically regarded as relevant in real deployment contexts.


Conformance
level
Test CategoryDescriptionInputsOutputs & Conformance PointsTest setsCapabilities exercised
DOpen access - data

Dump / Load cycle using canonical form data.

  • perform full EHR dump in canonical form
  • import into second empty instance
  • regression test using random query set

TBD: security / privacy aspects:

  • encryption?
  • id pseudonymisation?
  • EHR Index service (EHR id / Subject id Xref table) done separately

Selection criteria:

  • date range
  • versions
  • etc

Regression test two instances for:

  • identical content
  • correct EHR / subject id correlations
  • appropriate / matching data volumes, etc.

Various synthesised EHR test set:

  • demo: 1,000 EHRs
  • POC: 10,000 EHRs
  • clinic: 100,000 EHRs
  • enterprise: 1,000,000 EHRs
  • region: 10,000,000 EHRs
1 EHR = ave 100 Composition versions.

Ability of system to openEHR implementation to losslessly export all operational data and meta-data into a fully defined external format.

MEHR managementTest merge of EHRs where discovered to be same subjects





Test split of EHR where content shown to be two different subjects





Test move of EHR from openEHR instance to second instance



XX-enterprise sharingTest synchronisation merging of EHRs in two different openEHR instances where asynchronous updates made in each.


Test ability of openEHR EHR content to be synchronised across different enterprise instances. E.g. medication list and care plan in GP and in hospital, with different transactions in each.

Conformance schedule - Performance & Volumetrics

NB: There may be better schemes for formalising performance levels - please comment.

Conformance
level
Test CategoryDescriptionInputsOutputs & Conformance PointsTest setsCapabilities exercised
POCPOC

Minimal level of:

  • 5 Concurrent users
  • Commit transaction rate



System suitable for demonstration of all functional characteristics.
S

Small Enterprise

Characteristics:

  • 100 Concurrent users
  • 2 sec screen latency
  • XX Commit transaction rate
  • XX Query transaction rate
  • 100,000 EHRs
  • 20 concurrent encounters / inpatients



Usability for smaller health care facilities, e.g. clinic, GP surgery, small hospital.
LLarge Enterprise

Characteristics:

  • 1000 Concurrent users
  • 1.5 sec screen latency
  • XX Commit transaction rate
  • XX Query transaction rate
  • 1,000,000 EHRs



Usability in large hospital / hospital + community health area (other clinics, labs etc).
REGRegion

Characteristics:

  • 10,000 Concurrent users
  • 1.5 sec screen latency
  • XX Commit transaction rate
  • XX Query transaction rate
  • 10,000,000 EHRs



Usability for large-scale health data infrastructure, e.g. whole country, state etc.

Conformance schedule - Security and Privacy

This category tests conformance to security and privacy requirements. These are not primarily based on the openEHR architecture, but the method of testing will be specific to openEHR.

Conformance
level
Test CategoryDescriptionInputsOutputs & Conformance PointsTest setsCapabilities exercised
1BASIC

TBD




EHR / demographic information separation.