NOTE: The galeon mailing list is no longer active. The list archives are made available for historical reasons.
Hi,
Below is the note from Martin on the issues related to conformance tests
for the CF-netCDF encoding specs.
-- Ben
---------- Forwarded message ----------
From: Martin Desruisseaux <martin.desruisseaux@xxxxxxxxxxx>
Date: Fri, Jan 18, 2013 at 3:54 AM
Subject: [CF-NetCDF-1.0.swg] On executable tests for OGC standards
To: "GeoAPI 3.0 SWG" <geoapi-3.0.swg@xxxxxxxxxxxxxxxxxxxxxxxx>,
cite.sc@xxxxxxxxxxxxxxxxxxxxxxxx, CF-NetCDF-1.0.SWG@xxxxxxxxxxxxxxxxxxxxxxxx
Hello all
This email is sent to the GeoAPI, CITE and NetCDF lists because I think it
is a transversal topic. The purpose is only to share some though about
executable test suites for some specific kind of implementations - this is
not a request for change in the CITE tests, since they already cover well
the most common kind of OGC standards.
At the OGC meeting in Redlands we had some talks about executable tests for
products implementing OGC standards, like what CITE can already do for WMS
and other Web Services. The NetCDF group has expressed the wish for testing
NetCDF encoders. The CITE group mentioned the value of developing test
suites together with specifications, since it sometime help to spot issues.
I can confirm this later argument: on the 5 changes requests submitted at
the GeoAPI meeting this week, 3 of them address problems identified during
the development of the GeoAPI conformance test suites.
There is currently (to my knowledge) 2 series of tests at OGC: The CITE
tests are by far the most well known. But GeoAPI also provides tests in its
geoapi-conformance module. Currently, both test suites are living
independently. About six months ago, we started to talk about putting them
closer, but this didn't happened yet because we are testing things that are
different enough for proposing different solutions (at least for now).
Kind of tests: Validators versus Test Cases
In GeoAPI, we distinguish two kind of tests:
- Validators (taken in a broader sense than XML validators)
- Test cases (i.e. execution of a piece of the software to test)
CITE also provides validators: we can pass a file (currently XML, but could
also be NetCDF) to a CITE command-line tools, and the file is validated (in
the broad sense): the validator checks for well-formed document, data
consistency and other OGC or standard-specific constraints. It is not
necessary for the product to implement a standard set of interfaces in
order to execute this kind of tests; we just need to pass a file or a URL.
For this reason, validators are not the subject of this email.
The subject of this email is the other kind of tests: invoking an operation
with parameters, and verifying the output. The verification step may imply
the above-cited validation, but this is not the subject of this email.
Here, I'm interested only in the process of invoking some kind of method
with parameters for executing an operation, then receiving a reply. For
example the operation may be "*encode a NetCDF file*", the parameters may
be the data, and the result is the NetCDF file. This could be done by a WPS.
Kind of interfaces: Web Services versus Programming Language Interfaces
The above-cited test cases are possible only if there is some system
interfaces between the test cases and the product to be tested. If there is
many different products to test, then the system interfaces need to be
standard. The system interfaces can be the conventions on the URL encoding
of the request, together with the XML schema of the reply - this is what
WMS, WCS, etc. do. The system interfaces can also be defined in REST, or be
CORBA interfaces, or Java interfaces, or "D-Bus" messages on Linux/GNOME,
etc. They key point that I want to stress out is that it is absolutely
impossible to run this kind of tests on arbitrary implementations without
some kind of standard interfaces, no matter if the interfaces are defined
using Web technologies, programming languages or POSIX daemons.
In my understanding (please correct me if I'm wrong), CITE currently
performs every test cases (excluding validators) through web-related
technologies: the product to be tested must run as a server; CITE sends
requests to that server through HTTP, gets the reply again through HTTP
(could be FTP or SSH, the idea is similar) and verifies that reply. TestNG
classes are provided, but they are wrappers allowing to integrate automatic
execution of CITE tests in build environments like Maven, Ant, NetBeans or
Eclipse. TestNG allows that integration because they define an Application
Programming Interface (API) for the *test suite* - this is not an
Application Programming Interface for the *product* to be tested. The
ultimate interface between the tests and the products still the OGC
standards built on Web technologies, i.e. every CITE tests will end up in a
WMS, WCS or other W* request to a HTTP server, no matter if the CITE test
has been executed through TestNG or not.
In my understanding, this implies that for allowing CITE to test NetCDF
encoders (i.e. testing how the encoder reacts to various parameter values,
not validating a static file), someone need to setup a WPS server (may be a
temporary local server) providing an "encode" operation which use the
NetCDF encoder to be tested.
As an alternative, the interfaces between the products and the tests can
also be defined as Java interfaces. In this approach, there is no more HTTP
server - the NetCDF encoder (for example) is tested directly, at least for
Java implementations (or through JNI for C/C++ implementations). It may not
be worth to define Java interfaces only for the purpose of testing. But if
those Java interfaces are defined anyway in an inter-operability effort,
then a test suite can leverage them. This is what GeoAPI does.
System tests versus Unit tests
Each of the above-cited approaches have advantage and inconvenient: Java
interfaces are more efficient (in term of performance and memory usage)
while HTTP is more platform-independent. However they do not work at the
same granularity level. CITE tests are system tests: they test the system
as a whole. In the NetCDF case, CITE would test the whole system composed
of the NetCDF encoder together with the WPS server. On the other hand, unit
tests work on very small components: they could test some specific parts of
the NetCDF encoding process isolated from all other operations. Both kind
of tests have their use.
The geoapi-conformance module tests libraries, because GeoAPI defines the
interfaces between libraries and applications using them. On the other hand
CITE (in my understanding) tests servers and clients (ignoring CITE
validators), because the OGC Web Services specifications define the
interfaces between those two kind of entities. The fact that the
quasi-totality of OGC non-abstract specifications are web-related may give
the impression that Web Services are the only system interfaces, but this
doesn't have to be the case. Indeed, GeoAPI may be the only exception in
OGC at this time.
I think that a NetCDF encoder/decoder is more like a library than a client
or a server. For this reason, GeoAPI may be an appropriate place for
testing such encoders/decoders. In its current state, GeoAPI defines
interfaces for ISO 19115 (metadata) and ISO 19111 (referencing) abstract
specifications, but does not yet define interfaces for ISO 19123 (coverage)
or data input/output. Consequently we are able to test NetCDF metadata
(including CRS definitions) but not yet the NetCDF values. Indeed, a very
simple test for such NetCDF metadata already exists [2]. For now we
workaround the lack of data input/output interfaces by using the UCAR
NetCDF library API.
More information on the binding between GeoAPI and the UCAR NetCDF library
can be found at [1]. An other worthy class to note is [3].
I would like to stress out that this is not an attempt to deviate the test
efforts from CITE. My intend is to point out that different kind of tests
(unit tests, integration tests, system tests) sometime imply different
tools. The CITE TestNG tests are very convenient for automatic testing, but
they are wrappers around the CITE System tests rather than Unit tests on
their own.
My vision is that CITE and geoapi-conformance could share some code,
especially around file validators, while staying different on the Test Case
aspects. I don't know if there is any interest for exploring this path. But
if so, maybe I could try to clarify those aspects at CITE or NetCDF meeting
(if any) in Frascati if considered appropriate, or by email if there is
questions.
Regards,
Martin Desruisseaux
[1] http://www.geoapi.org/geoapi-netcdf/index.html
[2]
http://www.geoapi.org/geoapi-netcdf/testapidocs/org/opengis/wrapper/netcdf/NetcdfMetadataTest.html
[3]
http://www.geoapi.org/geoapi-netcdf/apidocs/org/opengis/wrapper/netcdf/NetcdfTransformFactory.html
_______________________________________________
CF-NetCDF-1.0.swg mailing list
CF-NetCDF-1.0.swg@xxxxxxxxxxxxxxxxxxxxxxxx
https://lists.opengeospatial.org/mailman/listinfo/cf-netcdf-1.0.swg
--
Ben
galeon archives: