Re: [netcdfgroup] Wondering about when NetCDF data hits the disk...

  • To: Thomas Orgis <thomas.orgis@xxxxxx>
  • Subject: Re: [netcdfgroup] Wondering about when NetCDF data hits the disk...
  • From: Rob Ross <rross@xxxxxxxxxxx>
  • Date: Wed, 28 Oct 2009 08:17:34 -0500
Hi,

NFS is not meant for the sort of concurrent access that you're attempting. There is no coordination of the caching on one client with another. The sync() operation pushes data from the writing client back to the server, but there's no good way to ensure that the reader sees the up-to-date data. You could *try* closing and re-opening on the client, but there are no guarantees.

This is not a netCDF issue, but an NFS one.

Regards,

Rob
---
Rob Ross
Mathematics and Computer Science Division
Argonne National Laboratory

On Oct 28, 2009, at 5:17 AM, Thomas Orgis wrote:

Hi!

I doing some simple run-time visualization of a model run that creates a NetCDF file, adding records to state variables as time advances (time is the unlimited dimension). The model runs on one machine, the visualization on another one, both share the data via NFS (from another machine). Now, the visualization is simply this: Look at the NetCDF header if the time dimension grew, if it did, extract and plot the last available record. It is a shell script that does simple busy-waiting with intervals of a second (or 10, which is the same, basically;-), calling a Fortran program to look at the header and then another one to extract the data.

I have experienced issues with the NFS shares (or even the local file system) before (p.ex. builds fail because a following compilation step cannot read all of the object data written in an earlier step), so I am suspicious about any data inconsistencies and funky caching issues. But I want to first make sure that the issue I am having with my simple run-time visualization is not simply rooted in the way NetCDF writes its data.

The problem is this: My script sees that a new time is there, but the subsequent data extraction does not yield a complete data set. There can be several seconds between writing of the data and the attempt to extract it... so I suspect some nasty caching at some point. It is not that much data, in the megabyte range -- it does take a fraction of a second to write it.

I was about to ask about the order of write operations in NetCDF, like, if it first writes the new record of a variable and _then_ updates the header to increase the record dimension length -- but this is really not my problem here (an in addition, it is rather moot since I have to write my data and the "time" variable itself, too, so there are always times with inconsistent header/variable values when not using some locking for file access). My problem is that there simply is a long delay (approaching the minute range, instead of seconds) before the vizualization machine sees the written bytes from the model run, and then it even doesn't see full records.

I do call nf90_sync() in the writing program, just after appending a record. From its documentation I gather that it should exactly do what I expect: Make sure any buffers are written do disk... make the file on disk consistent.
But, well... I see now that this does not work with NFS. Or should it?
I observe that I can trigger an apparent update of the data available for visualization (also getting a consistent data excerpt) when running 'sync' on the machine that runs the model code -- where nf90_sync() has been called before.

So, perhaps it boils down to this: What kind of syncing is implied in nf90_sync()? It is just internal NetCDF buffers to operating system buffers, I presume... so there is no call to the C fsync() function in NetCDF, for example? Or is there such a call and our NFS (with ZFS behind) setup is simply broken? But then, 'sync' on the command line works...

Can someone enlighten me on the caching/synchronization strategy there?


Alrighty then,

Thomas.


PS: I may quickly check what kind of syncing NetCDF does by looking at the code, but it may also be a good idea to have a bit of discussion about this... or have me quickly pointed to the FAQ entry that explains it all;-)

--
Dipl. Phys. Thomas Orgis
Atmospheric Modelling
Alfred-Wegener-Institute for Polar and Marine Research

_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/



  • 2009 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: