Hi John,
> We've noticed a large difference in single variable read times between
> netCDF-4/HDF5 reads made with the netCDF-4 API (slower) and reads made with
> the HDF5 API (faster). Single variable reads are a common access pattern in
> fusion research, where 1000s of files are often scanned to analyse
> experimental data for a particular variable.
>
> This came to light when benchmarking reads of a netCDF-4/HDF5 file with 3000
> variables, where a single variable read (open, read, close) took 11ms using
> the HDF5 API and 1300ms using the netCDF-4 API. In contrast, multiple
> variable reads of the open file with the netCDF-4 API took 0.7ms each.
> Evidently the netCDF-4 API builds an internal data structure upfront to
> assist possible later access, while the HDF5 API postpones that until access
> is actually needed.
>
> We are considering using the HDF5 API for fast single variable access. Is
> there any other way to avoid the netCDF-4 file open overhead?
Can you make a sample file available? We might be able to speed up
opens or provide a flag for fast opens that specifies not reading all
the metadata initially.
Thanks!
--Russ
_____________________________________________________________________
Russ Rew UCAR Unidata Program
russ@xxxxxxxxxxxxxxxx http://www.unidata.ucar.edu
