Conformance Test Options

Owned by Thomas Beale
Last updated: 11-Jun-2018 by Seref Arikan
6 min read

Background

See the part of the spec describing the System Under Test (SUT). This shows (in a simplifed way) the test environment. The general picture of a component being tested is as follows:

Here, the capability actually being tested (particularly truth conditions, e.g. pre- and post-conditions) is marked by the red arrow, e.g. a API call logically defined as create_ehr(). These abstract definitions are formalised in the Abstract Platform spec, and are documented using UML and the usual class definition approach.


The actual thing being tested by any concrete test approach is of course the interface exposed for a concrete protocol - what the blue arrow points to in the first diagram. So for REST, we have a spec describing that, with definitions like the following.

So far so good. Now, in the conformance spec, we have a schedule table, with a row for each testable service call that looks like the following:


In this, in the second last column, is a logical test script, of which some are written, in Java / Java-like pseudo-code. E.g.

Then, in the last column, we have links to scripts for some tool (currently thought to be postman) for writing test scripts for the REST API. More columns would be added for other protocols.

Now, what we need to achieve is:

  • a way of writing all those logical test scripts
  • a way of writing the concrete test scripts for multiple protocols (currently we'll just do today's flavour of REST), so they are executable
  • a way of executing the concrete test scripts for a given protocol

Cucumber Approach

How Does it Work

If we were to approach this job with Cucumber and associated tools, as used by Seref Arikan for various projects in the UK (Ripple and related) for some time it might look like the following. We'll use the example of the abstract call I_EHR_COMPOSITION.update_composition():

In the REST spec it is:

In the Cucumber approach, the first thing we write is a Feature definition. This can be done at different levels of granularity. Here's the basic idea:

The text at the top after the Feature line should correspond to the text in column 1 of the rows containing the tests we want to write, in the table above, i..e it should say something like 'provide access to Composition CRUD operations' or similar. 

Then we have a Scenario for each test case, i.e. each row such as the highlighted one (`update_composition()`) becomes a scenario. The line of text below the keyword Scenario: corresponds to the test in col 3 of the table, for that row. This is natural language only.

Now, we want to write the abstract script in some way. This is done with using some keywords, and structured text statements whose phrases (strings of words) that are regex-matched to generated Java code that performs the test. The keywords include Given, When, Then, and logical operators ( And, Or etc).

So in the above, the first Given is followed by the text 'There is an EHR with a subject' (a set up condition). The tooling generates this code for that:

We can see the @Given annotation following by a regex match pattern. Code fragments like this are generated for each phrase that corresponds to some sort of CRUD or other operation. The central statements of course have to be written by hand. (i.e where the 'throw new ...' bit is).

Now, often we want some set up to have been done before one or more tests are run, so this can be done by more code fragments connected to the Background keyword, as follows:

There are pieces of code connected to the Given and And phrases under the Background keyword, that create and store a prescription Composition that is described by prescription.opt. So a more complete version of this test would look like:

Some real test code (i.e. with the relevant code added manually) for the first Given statement under Background looks like this:

So here we have the static method given() imported from Response, on which a builder pattern piece of code hangs. The final parts of the code would need further work to implement exactly the test case, but it's not hard to see how to do it.

For a different protocol, the Background code could be the same, if a REST interface can always be assumed for test purposes, with just the code fragments for the Scenario parts of the test text being different.

Running The Tests

Tests written in the above way can all be run as a full suite in a Continuous Build (aka Devops) environment as is common these days, and it will generate a report in HTML, PDF etc, of exactly the kind described in the certificate section of the Conformance test spec.

SA: I'm pretty sure the tests can also run outside of CB setting and without maven. Furthermore, it may be possible to create standalone native executables for linux and windows platforms using the the recently released Graal VM

Test Data

One of the key issues in the above is having not just test scripts, but test data, generally described by templates and underlying archetypes. To make conformance testing really solid, we need to create numerous (e.g. 10) variant sets that are not necessarily all released to vendors, or else test templates  generate on-the-fly by the Conformance test environment.

SA: Yes, this is absolutely necessary. I'm repeatedly coming across this requirement during testing various stacks.

How Would we use it?

If we were to adopt this approach, we would probably replace the test Schedule tables with something generated from the Cucumber test environment, or else we retain it as a table, but fill it with links to the test environment. As described above, the logical test script is now replaced with the English language (but computable) Scenario script as shown above, and the concrete test code by numerous fragments of Java code whose boilerplate is generated by the tooling.

The obvious advantage is that all of the above is immediately executable, for whatever tests have been written, and can be set up by any vendor or other developer to continuously test their product and report on it. 

SA: For an example of how automatically executed (continuous build) test results can be presented, please see this page and click on cucumber test reports link

The availability of all the tooling means that openEHR Foundation can perform official Conformance testing itself, since all of the tools are widely available, the same environment can be set up by any product developer to do their own testing.

SA: At some point we could allow vendors/implementers to simply register their base URLs in a foundation provided test service and run tests. This would save any implementers from even having to run the test code. They'd simply get reports.

How much work?

There are around 75 rows, i.e. Scenarios in the current conformance spec. If we assume that each one takes on average 3 hours to develop to completion (if developed together as a block of work), plus another say 40h to set up environment, tools etc, we can roughly estimate 265 h, i.e. 33 days, or roughly 7 weeks.

There will certainly be an ongoing work effort to add new tests, correct existing tests and so on, but if the incremental work per scenario can be kept to say 4 hours on average, this appears to be sustainable.