Hi Leon,
> I've got a -- possibly ill posed -- question about the NF90_SYNC command:
> does calling it append new data to the already existing file (in the same
> way that a few more 'write(1,*)'s would) and modify a few existing things,
> or does it rewrite the file from scratch? Does it do any/much reading in
> the process?
It calls the C nc_sync() function, which just flushes memory buffers to
disk (for a netCDF file open for writing) and updates the record count,
if new records have been added. For a reader, it just re-reads the
record count and makes sure the next read comes from disk, rather than
memory buffers, in case the file has been updated since the last read.
As described in the nc_sync() documentation:
https://www.unidata.ucar.edu/software/netcdf/docs/netcdf-c/nc_005fsync.html
readers have to be careful if a writer changes the netCDF schema.
Readers that have the file open must be informed by some mechanism
external to the netCDF library to call nc_sync() before reading any more
data from the file, so they can flush their buffers and reread the
schema.
> I'm using a netCDF-4/HDF5 file with compression.
>
> I'm using HTCondor <http://research.cs.wisc.edu/htcondor/>'s standard
> universe, which effectively captures all file input/output and redirects it
> over a network. Although my code only produces ~100MB of new data per day,
> I'm told that the resulting network usage is dangerously high. The program
> calls NF90_SYNC every 5 minutes, so I'm wondering if that could be
> the culprit. (i.e. if it's doing a lot of reading/writing in the process.)
Maybe the program is assuming simultaneous writing and reading and
always calls nc_sync() after each write and before each read, just to
be safe. This eliminates the performance benefits of buffering and
caching data, and would usually cause some re-reading by readers.
Large chunk sizes might mean a lot of extra I/O, as well as extra CPU
for uncompressing the same data chunks repeatedly. You might see if
lowering your chunk size significantly improves network usage ...
> (Also, I just want to confirm that the NF90_SHARE flag doesn't work for
> netCDF-4/HDF5 files. The NF90_SYNC
> page<http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-f90/NF90_005fSYN
> C.html>
> recommends using
> the flag, but doesn't mention netCDF-4/HDF5file incompatibility; mentioning
> the incompatibility is saved for the NF90_CREATE
> page<http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-f90/NF90_005fCRE
> ATE.html>
> .)
Yes, as far as I can determine, the NF90_SHARE flag for Fortran 90, the
NF_SHARE flag for Fortran 77, and the NC_SHARE flag for C are all
ignored for netCDF-4/HDF5 files, so there's not an efficient alternative
to using nc_sync() calls for a writer and multiple readers, unless you
can use parallel I/O
Not mentioning this in the NF90_SYNC(), NF_SYNC(), and nc_sync()
documentation is an oversight that we'll fix. Thanks for pointing out
those documentation bugs!
--Russ
> Thanks.
> -Leon
>
> --f46d0438914b0a7a0704d62ae7bc
> Content-Type: text/html; charset=UTF-8
> Content-Transfer-Encoding: quoted-printable
>
> I've got a -- possibly ill posed -- question about the=C2=A0NF90_SYNC c=
> ommand: does calling it append new data to the already=C2=A0existing=C2=A0f=
> ile (in the same way that a few more 'write(1,*)'s would) and modif=
> y a few existing things, or does it rewrite the file from scratch? Does it =
> do any/much reading in the process?<br>
> <br><div>I'm using a netCDF-4/HDF5 file with compression.</div><div><br=
> ></div>I'm using <a href=3D"http://research.cs.wisc.edu/htcondor/";>HTCo=
> ndor</a>'s standard universe, which effectively captures all file input=
> /output and redirects it over a network. Although my code only produces ~10=
> 0MB of new data per day, I'm told that the resulting network usage is d=
> angerously high. The program calls=C2=A0NF90_SYNC every 5 minutes, so I'=
> ;m wondering if that could be the=C2=A0culprit. (i.e. if it's doing a l=
> ot of reading/writing in the process.)<br>
> <div><br></div><div>(Also, I just want to confirm that the NF90_SHARE flag =
> doesn't work for netCDF-4/HDF5 files. The <a href=3D"http://www.unidata=
> .ucar.edu/software/netcdf/docs/netcdf-f90/NF90_005fSYNC.html">NF90_SYNC pag=
> e</a>=C2=A0recommends=C2=A0using the flag, but doesn't mention netCDF-4=
> /HDF5file=C2=A0incompatibility; mentioning the incompatibility=C2=A0is save=
> d for the <a href=3D"http://www.unidata.ucar.edu/software/netcdf/docs/netcd=
> f-f90/NF90_005fCREATE.html">NF90_CREATE page</a>.)</div>
> <div><br></div><div>Thanks.</div><div>-Leon</div>
>
> --f46d0438914b0a7a0704d62ae7bc--
>
>
> --===============1900850759==
> Content-Type: text/plain; charset="us-ascii"
> MIME-Version: 1.0
> Content-Transfer-Encoding: 7bit
> Content-Disposition: inline
>
> _______________________________________________
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/m
> ailing_lists/
> --===============1900850759==--
