Hi there
We've noticed a large difference in single variable read times between
netCDF-4/HDF5 reads made with the netCDF-4 API (slower) and reads made with
the HDF5 API (faster). Single variable reads are a common access pattern in
fusion research, where 1000s of files are often scanned to analyse
experimental data for a particular variable.
This came to light when benchmarking reads of a netCDF-4/HDF5 file with 3000
variables, where a single variable read (open, read, close) took 11ms using
the HDF5 API and 1300ms using the netCDF-4 API. In contrast, multiple
variable reads of the open file with the netCDF-4 API took 0.7ms each.
Evidently the netCDF-4 API builds an internal data structure upfront to
assist possible later access, while the HDF5 API postpones that until access
is actually needed.
We are considering using the HDF5 API for fast single variable access. Is
there any other way to avoid the netCDF-4 file open overhead?
Regards
John
--
John Storrs, Experiments Dept e-mail: john.storrs@xxxxxxxxxxxx
Building D3, UKAEA Fusion tel: 01235 466338
Culham Science Centre fax: 01235 466379
Abingdon, Oxfordshire OX14 3DB http://www.fusion.org.uk
This message has been scanned for viruses by BlackSpider MailControl -
www.blackspider.com
