openEHR Conformance
- Thomas Beale
Status
Development.
All community input welcome.
Overview
Conformance testing is used as the basis of product certification, and has the following goals:
- Tendering: enable tendering authorities to state formal criteria for compliance in tenders
- Protection for solution developers: enables bona fide vendors and other developers to prove the quality of their solutions compared to other offerings claiming conformance
- Protection for procurement: provides a way of ensuring that purchased solutions can be contractually guaranteed to perform in certain ways
Conformance is tested in two categories:
Functional: test correctness of implementation with respect to specifications, represented by specific test sets
- Performance: test capacity of system to process data and transactions
All conformance specifications, test sets and related materials are part of the Conformance Specification component.
References
Conformance Certificate
An openEHR Conformance Certificate consists of a set of ratings in a number of dimensions and detailed report. The ratings are as follows, derived from the Conformance Schedules below:
- Functional: 1 | 2 | 3 + O
- Enterprise: D | M | X
- Performance & volumetrics: POC | S | L | R
- Security & Privacy:
Evaluation
Conformance testing is intended to be performed by the interaction of a system under test with a well defined openEHR web service that implements the various tests and captures the results.
Conformance schedule - Functional
This category includes tests to determine conformance to base specifications, i.e. things like the Reference Model, querying and so on.
| Conformance level | Test Category | Description | Inputs | Outputs & Conformance Points | Test sets | Capabilities exercised |
|---|---|---|---|---|---|---|
| 1 | Template injection test | End-to-end template => data test.
| Composition Template, includes:
| Regression test exported openEHR canonical XML v test reference XML | TBD: Ian's monster template |
|
| 1 | EHR API lifecycle test | EHR create, commit, read via EHR API.
| Composition Templates | ?Contribution JSON v regression JSON Regression test exported openEHR canonical XML v test reference XML | Template set:
|
|
| 2 | Import - TDD | Data commit using TDD based on provided TDS
| TDS | Regression test exported openEHR canonical XML v test reference XML | Vital signs template | |
| 2 | Basic AQL | Test AQL query against single data set (template).
| Template Test query | Regression test Result set XML v reference XML | Vital signs OBS template:
VS Test query |
|
| 3 | REST API | |||||
| 3 | Advanced AQL | Exercise AQL queries with WHERE clause across multiple data sets (templates) | Templates Test query | Regression test Result set XML v reference XML |
| |
| OPTIONAL | Demographics API lifecycle test | Exercise openEHR demographic content create and retrieve | ||||
| OPTIONAL | Export - EHR Extract | Export selected content in EHR Extract form. TBD |
Conformance test design
Conformance testing on a product for a given test category is is performed by:
- select a specific test set for that category
- setting up initial conditions, e.g. inject a template
- executing specific transactions on a given API or other interface (e.g. query)
- retrieving result
- performing regression comparison of actual v reference
Conformance schedule - Enterprise
This category contains tests for conformance to functional requirements typically regarded as relevant in real deployment contexts.
| Conformance level | Test Category | Description | Inputs | Outputs & Conformance Points | Test sets | Capabilities exercised |
|---|---|---|---|---|---|---|
| D | Open access - data | Dump / Load cycle using canonical form data.
TBD: security / privacy aspects:
| Selection criteria:
| Regression test two instances for:
| Various synthesised EHR test set:
1 EHR = ave 100 Composition versions. | Ability of system to openEHR implementation to losslessly export all operational data and meta-data into a fully defined external format. |
| M | EHR management | Test merge of EHRs where discovered to be same subjects | ||||
| Test split of EHR where content shown to be two different subjects | ||||||
| Test move of EHR from openEHR instance to second instance | ||||||
| X | X-enterprise sharing | Test synchronisation merging of EHRs in two different openEHR instances where asynchronous updates made in each. | Test ability of openEHR EHR content to be synchronised across different enterprise instances. E.g. medication list and care plan in GP and in hospital, with different transactions in each. |
Conformance schedule - Performance & Volumetrics
NB: There may be better schemes for formalising performance levels - please comment.
| Conformance level | Test Category | Description | Inputs | Outputs & Conformance Points | Test sets | Capabilities exercised |
|---|---|---|---|---|---|---|
| POC | POC | Minimal level of:
| System suitable for demonstration of all functional characteristics. | |||
| S | Small Enterprise | Characteristics:
| Usability for smaller health care facilities, e.g. clinic, GP surgery, small hospital. | |||
| L | Large Enterprise | Characteristics:
| Usability in large hospital / hospital + community health area (other clinics, labs etc). | |||
| REG | Region | Characteristics:
| Usability for large-scale health data infrastructure, e.g. whole country, state etc. |
Conformance schedule - Security and Privacy
This category tests conformance to security and privacy requirements. These are not primarily based on the openEHR architecture, but the method of testing will be specific to openEHR.
| Conformance level | Test Category | Description | Inputs | Outputs & Conformance Points | Test sets | Capabilities exercised |
|---|---|---|---|---|---|---|
| 1 | BASIC | TBD | EHR / demographic information separation. |