Hi,
NFS is not meant for the sort of concurrent access that you're
attempting. There is no coordination of the caching on one client with
another. The sync() operation pushes data from the writing client back
to the server, but there's no good way to ensure that the reader sees
the up-to-date data. You could *try* closing and re-opening on the
client, but there are no guarantees.
This is not a netCDF issue, but an NFS one.
Regards,
Rob
---
Rob Ross
Mathematics and Computer Science Division
Argonne National Laboratory
On Oct 28, 2009, at 5:17 AM, Thomas Orgis wrote:
Hi!
I doing some simple run-time visualization of a model run that
creates a NetCDF file, adding records to state variables as time
advances (time is the unlimited dimension). The model runs on one
machine, the visualization on another one, both share the data via
NFS (from another machine).
Now, the visualization is simply this: Look at the NetCDF header if
the time dimension grew, if it did, extract and plot the last
available record. It is a shell script that does simple busy-waiting
with intervals of a second (or 10, which is the same, basically;-),
calling a Fortran program to look at the header and then another one
to extract the data.
I have experienced issues with the NFS shares (or even the local
file system) before (p.ex. builds fail because a following
compilation step cannot read all of the object data written in an
earlier step), so I am suspicious about any data inconsistencies and
funky caching issues. But I want to first make sure that the issue I
am having with my simple run-time visualization is not simply rooted
in the way NetCDF writes its data.
The problem is this: My script sees that a new time is there, but
the subsequent data extraction does not yield a complete data set.
There can be several seconds between writing of the data and the
attempt to extract it... so I suspect some nasty caching at some
point. It is not that much data, in the megabyte range -- it does
take a fraction of a second to write it.
I was about to ask about the order of write operations in NetCDF,
like, if it first writes the new record of a variable and _then_
updates the header to increase the record dimension length -- but
this is really not my problem here (an in addition, it is rather
moot since I have to write my data and the "time" variable itself,
too, so there are always times with inconsistent header/variable
values when not using some locking for file access). My problem is
that there simply is a long delay (approaching the minute range,
instead of seconds) before the vizualization machine sees the
written bytes from the model run, and then it even doesn't see full
records.
I do call nf90_sync() in the writing program, just after appending a
record. From its documentation I gather that it should exactly do
what I expect: Make sure any buffers are written do disk... make the
file on disk consistent.
But, well... I see now that this does not work with NFS. Or should it?
I observe that I can trigger an apparent update of the data
available for visualization (also getting a consistent data excerpt)
when running 'sync' on the machine that runs the model code -- where
nf90_sync() has been called before.
So, perhaps it boils down to this: What kind of syncing is implied
in nf90_sync()? It is just internal NetCDF buffers to operating
system buffers, I presume... so there is no call to the C fsync()
function in NetCDF, for example? Or is there such a call and our NFS
(with ZFS behind) setup is simply broken? But then, 'sync' on the
command line works...
Can someone enlighten me on the caching/synchronization strategy
there?
Alrighty then,
Thomas.
PS: I may quickly check what kind of syncing NetCDF does by looking
at the code, but it may also be a good idea to have a bit of
discussion about this... or have me quickly pointed to the FAQ entry
that explains it all;-)
--
Dipl. Phys. Thomas Orgis
Atmospheric Modelling
Alfred-Wegener-Institute for Polar and Marine Research
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
