Re: [netcdfgroup] NetCDF external links

We have a number of operational products based on fixed lat/lon grids that we 
disseminate in near-real time.

The ability to be able to send the lat/lon grid once and link to it as a 
coordinate variable from within the file would be very useful as it would save 
considerably on bandwidth costs while still keeping the products user-friendly.

So this would be a welcome development for our purposes.

Tim

_________________________________________________________________________________________
Dr. Tim Patterson
Instrument Data Simulation Expert
Product Engineering/Test Data Coordination
MTG Programme
GEO  Division

EUMETSAT
Eumetsat-Allee 1
64295 Darmstadt
Germany

Tel: +49 6151 807 487
Fax: +49 6151 807 7
E-mail: timothy.patterson@xxxxxxxxxxxx
Web: www.eumetsat.int






-----Original Message-----
From: netcdfgroup-bounces@xxxxxxxxxxxxxxxx 
[mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Eugen Betke
Sent: Wednesday, July 13, 2016 1:09 PM
To: netcdfgroup@xxxxxxxxxxxxxxxx
Subject: [netcdfgroup] NetCDF external links

Dear NetCDF-Group,

we have been working on NetCDF external link functionality. This allows NetCDF 
applications to create dimension variables which values are stored in an 
external file. Therefore, it uses the HDF5 virtual dataset
(VDS) functionality. This is useful for, e.g., climate applications that rely 
on a variable per file and timestep configuration. The idea is to store the 
grid in a separate file and link our data to this grid. We already have our 
first working version. You find the patch and the examples on our page:

http://wr.informatik.uni-hamburg.de/research/projects/bullio/netcdf_external_links/start

Under the hood it uses HDF5 virtual datasets. VDS has the advantage of being 
compatible to the functions that are supported by oridinary datasets. 
Therefore, files containing VDS should be supported by the most software.

There is a minor issue related to HDF5, the call H5F_try_close function fails, 
when ncdump trys to read data from an external dimension. So far we found a 
workaround, but we will fix this issue.

It would be great if external link functionality could be supported by netCDF 
at some timepoint. We would like to improve our patch and for that reason we 
need your feedback. If you have some idea to the issue above, we would be 
grateful for each hint.

Regards,
Eugen

_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are recorded in 
the Unidata inquiry tracking system and made publicly available through the 
web.  Users who post to any of the lists we maintain are reminded to remove any 
personal information that they do not want to be made public.


netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/ 

Any email message from EUMETSAT is sent in good faith but shall neither be 
binding nor construed as constituting a commitment by EUMETSAT, except where 
provided for in a written agreement or contract or if explicitly stated in the 
email. Please note that any views or opinions presented in this email are 
solely those of the sender and do not necessarily represent those of EUMETSAT. 
This message and any attachments are intended for the sole use of the 
addressee(s) and may contain confidential and privileged information. Any 
unauthorised use, disclosure, dissemination or distribution (in whole or in 
part) of its contents is not permitted. If you received this message in error, 
please notify the sender and delete it from your system.