Hi Heiko: I think these calls to nc_sync have been there a long time.
I don't recall the original reason for them. Before netcdf version
4.1 was nc_sync just a no-op? If this is the case, then maybe we should
put in an AUTOSYNC option with the default = 0 (do not sync).
If the netcdf group has ideas about the utility of nc_sync before netcdf
version 4.1, then perhaps we should add the AUTOSYNC option with default =
1 (do sync).
Another alternative would be to remove all calls to nc_sync and then make
available and advertise a sync method in PDL::NetCDF.
NetCDF group: Was nc_sync useful before netcdf version 4.1?
Thanks!
--Doug
dhunt@xxxxxxxx
Software Engineer
UCAR - COSMIC, Tel. (303) 497-2611
On Tue, 5 Mar 2013, Heiko Klein wrote:
Hi Doug,
we just upgraded to Ubuntu Precise (12.04) and have for the first time a
netcdf version >= 4.1. With this upgrade PDL::NetCDF became awfully slow when
writing data, in particularly when writing small amounts of data.
Reason for that is that netcdf now calls 'fsync' when nc_sync is called.
Syncing the complete filesystem is very costy and I don't really understand
why the netcdf-folks did that by default (it might make sense in some HPC
filesystems - and fsync is not available from FORTRAN). It can be disabled in
build-time, but who really does that - most people just don't use nc_sync, in
particular since 'close' does this automatically.
But PDL::NetCDF calls nc_sync automatically after each put*. I would like to
just remove the nc_sync calls from PDL::NetCDF, and let users call them
manually if they really need syncronisation. If you oppose to that, I would
like to put a flag to new: (AUTOSYNC => 0|1) (with default to 1). What do you
think?
Best regards,
Heiko
--
Dr. Heiko Klein Tel. + 47 22 96 32 58
Development Section / IT Department Fax. + 47 22 69 63 55
Norwegian Meteorological Institute http://www.met.no
P.O. Box 43 Blindern 0313 Oslo NORWAY
