openEHR Conformance

Status

Development.

All community input welcome.

Overview

Conformance testing is used as the basis of product certification, and has the following goals:

Tendering: enable tendering authorities to state formal criteria for compliance in tenders
Protection for solution developers: enables bona fide vendors and other developers to prove the quality of their solutions compared to other offerings claiming conformance
Protection for procurement: provides a way of ensuring that purchased solutions can be contractually guaranteed to perform in certain ways

Conformance is tested in two categories:

Functional: test correctness of implementation with respect to specifications, represented by specific test sets
Performance: test capacity of system to process data and transactions

All conformance specifications, test sets and related materials are part of the Conformance Specification component.

References

Conformance Certificate

An openEHR Conformance Certificate consists of a set of ratings in a number of dimensions and detailed report. The ratings are as follows, derived from the Conformance Schedules below:

Functional: 1 | 2 | 3 + O
Enterprise: D | M | X
Performance & volumetrics: POC | S | L | R
Security & Privacy:

Evaluation

Conformance testing is intended to be performed by the interaction of a system under test with a well defined openEHR web service that implements the various tests and captures the results.

Conformance schedule - Functional

This category includes tests to determine conformance to base specifications, i.e. things like the Reference Model, querying and so on.

Conformance level	Test Category	Description	Inputs	Outputs & Conformance Points	Test sets	Capabilities exercised

Conformance level	Test Category	Description	Inputs	Outputs & Conformance Points	Test sets	Capabilities exercised
1	Template injection test	End-to-end template => data test. Inject a new template Instantiate from screen form or TDD Export canonical XML	Composition Template, includes: ALL RM types, including all DV_* types, including all reasonable generic derivations, e.g. DV_INTERVAL<DV_QUANTITY> etc. ALL typical compositional hierarchy structures, e.g. Composition → Entry[*] Composition → Section → Section → Entry etc	Regression test exported openEHR canonical XML v test reference XML	TBD: Ian's monster template	Archetypes Templates RM Canonical XML
1	EHR API lifecycle test	EHR create, commit, read via EHR API. create new EHR Contribution 1: admin Composition (admission) Contribution 2: persistent Composition (meds) Contribution 3: event Composition (vital signs) Contribution 4: update meds + event Composition (vital signs) display & output all Contributions & meta-data as JSON display & output latest version of Compositions in canonical XML display & output all versions of Compositions in canonical XML	Composition Templates	?Contribution JSON v regression JSON Regression test exported openEHR canonical XML v test reference XML	Template set: admission medications vital signs	EHR API Contributions and versioning
2	Import - TDD	Data commit using TDD based on provided TDS instantiate TDD from TDS by any method commit TDD export as canonical XML	TDS	Regression test exported openEHR canonical XML v test reference XML	Vital signs template
2	Basic AQL	Test AQL query against single data set (template). Commit vital signs OBS x 50 samples Execute query and return result set XML	Template Test query	Regression test Result set XML v reference XML	Vital signs OBS template: BP heart rate body temp SaO2 VS Test query	Basic AQL AQL result set
3	REST API
3	Advanced AQL	Exercise AQL queries with WHERE clause across multiple data sets (templates)	Templates Test query	Regression test Result set XML v reference XML		AQL WHERE processing
OPTIONAL	Demographics API lifecycle test	Exercise openEHR demographic content create and retrieve
OPTIONAL	Export - EHR Extract	Export selected content in EHR Extract form. TBD

Conformance test design

Conformance testing on a product for a given test category is is performed by:

select a specific test set for that category
setting up initial conditions, e.g. inject a template
executing specific transactions on a given API or other interface (e.g. query)
retrieving result
performing regression comparison of actual v reference

Conformance schedule - Enterprise

This category contains tests for conformance to functional requirements typically regarded as relevant in real deployment contexts.

Conformance level	Test Category	Description	Inputs	Outputs & Conformance Points	Test sets	Capabilities exercised

Conformance level	Test Category	Description	Inputs	Outputs & Conformance Points	Test sets	Capabilities exercised
D	Open access - data	Dump / Load cycle using canonical form data. perform full EHR dump in canonical form import into second empty instance regression test using random query set TBD: security / privacy aspects: encryption? id pseudonymisation? EHR Index service (EHR id / Subject id Xref table) done separately	Selection criteria: date range versions etc	Regression test two instances for: identical content correct EHR / subject id correlations appropriate / matching data volumes, etc.	Various synthesised EHR test set: demo: 1,000 EHRs POC: 10,000 EHRs clinic: 100,000 EHRs enterprise: 1,000,000 EHRs region: 10,000,000 EHRs 1 EHR = ave 100 Composition versions.	Ability of system to openEHR implementation to losslessly export all operational data and meta-data into a fully defined external format.
M	EHR management	Test merge of EHRs where discovered to be same subjects
		Test split of EHR where content shown to be two different subjects
		Test move of EHR from openEHR instance to second instance
X	X-enterprise sharing	Test synchronisation merging of EHRs in two different openEHR instances where asynchronous updates made in each.				Test ability of openEHR EHR content to be synchronised across different enterprise instances. E.g. medication list and care plan in GP and in hospital, with different transactions in each.

Conformance schedule - Performance & Volumetrics

NB: There may be better schemes for formalising performance levels - please comment.

Conformance level	Test Category	Description	Inputs	Outputs & Conformance Points	Test sets	Capabilities exercised

Conformance level	Test Category	Description	Capabilities exercised
POC	POC	Minimal level of: 5 Concurrent users Commit transaction rate	System suitable for demonstration of all functional characteristics.
S	Small Enterprise	Characteristics: 100 Concurrent users 2 sec screen latency XX Commit transaction rate XX Query transaction rate 100,000 EHRs 20 concurrent encounters / inpatients	Usability for smaller health care facilities, e.g. clinic, GP surgery, small hospital.
L	Large Enterprise	Characteristics: 1000 Concurrent users 1.5 sec screen latency XX Commit transaction rate XX Query transaction rate 1,000,000 EHRs	Usability in large hospital / hospital + community health area (other clinics, labs etc).
REG	Region	Characteristics: 10,000 Concurrent users 1.5 sec screen latency XX Commit transaction rate XX Query transaction rate 10,000,000 EHRs	Usability for large-scale health data infrastructure, e.g. whole country, state etc.

Conformance schedule - Security and Privacy

This category tests conformance to security and privacy requirements. These are not primarily based on the openEHR architecture, but the method of testing will be specific to openEHR.

Conformance level	Test Category	Description	Inputs	Outputs & Conformance Points	Test sets	Capabilities exercised

Conformance level	Test Category	Description	Inputs	Outputs & Conformance Points	Test sets	Capabilities exercised
1	BASIC	TBD				EHR / demographic information separation.