...
One of the key issues in the above is having not just test scripts, but test data, generally described by templates and underlying archetypes. To make conformance testing really solid, we need to create numerous (e.g. 10) variant sets that are not necessarily all released to vendors, or else test templates generate on-the-fly by the Conformance test environment.
SA: Yes, this is absolutely necessary. I'm repeatedly coming across this requirement during testing various stacks.
How Would we use it?
If we were to adopt this approach, we would probably replace the test Schedule tables with something generated from the Cucumber test environment, or else we retain it as a table, but fill it with links to the test environment. As described above, the logical test script is now replaced with the English language (but computable) Scenario script as shown above, and the concrete test code by numerous fragments of Java code whose boilerplate is generated by the tooling.
The obvious advantage is that all of the above is immediately executable, for whatever tests have been written, and can be set up by any vendor or other developer to continuously test their product and report on it.
SA: For an example of how automatically executed (continuous build) test results can be presented, please see this page and click on cucumber test reports link
The availability of all the tooling means that openEHR Foundation can perform official Conformance testing itself, since all of the tools are widely available, the same environment can be set up by any product developer to do their own testing.
SA: At some point we could allow vendors/implementers to simply register their base URLs in a foundation provided test service and run tests. This would save any implementers from even having to run the test code. They'd simply get reports.
How much work?
There are around 75 rows, i.e. Scenarios in the current conformance spec. If we assume that each one takes on average 3 hours to develop to completion (if developed together as a block of work), plus another say 40h to set up environment, tools etc, we can roughly estimate 265 h, i.e. 33 days, or roughly 7 weeks.
...