Re: [netcdfgroup] Wondering about when NetCDF data hits the disk...

  • To: Thomas Orgis <thomas.orgis@xxxxxx>
  • Subject: Re: [netcdfgroup] Wondering about when NetCDF data hits the disk...
  • From: Rob Ross <rross@xxxxxxxxxxx>
  • Date: Wed, 28 Oct 2009 10:23:25 -0500
Also, I stand corrected -- you are right that there don't appear to be any calls to fsync() in netCDF (at least version 3.6.3 that I just looked through). That, obviously, surprises me.

Rob

On Oct 28, 2009, at 10:09 AM, Thomas Orgis wrote:

Am Wed, 28 Oct 2009 09:28:36 -0500
schrieb Rob Ross <rross@xxxxxxxxxxx>:

It is a mistake to think that there is any rhyme or reason to the
cache update and replacement policy in NFS. In fact it is ok for a
client implementation to cache the file size and other metadata too,
and return an out-of-date version to a process.

I won't argue about the (non)reliability/consistency when working over NFS. I know there are many agents, caching on every host concerned, etc. All I'm asking for is reducing the time until the data hits the disk... for my use case it has been shown to be "good enough" to make the writer push its writes towards the NFS server (call 'sync' on the shell).

"The function NF90_SYNC offers a way to synchronize the disk copy
of a netCDF dataset with in-memory buffers"
"For a writer, this flushes buffers to disk."

It is supposedly synchronizing that process's in-memory buffers with
the copy on the server (on disk). Actually, what is probably
happening is that the dirty regions are being pushed to the server,

Actually not. I grepped trough netCDF source and did not find an fsync() call. nc_sync() just seems to trigger plain write()s. I may be wrong here, as there is some IO layering going on inside the code and I am not that familiar with it. The situation looks to me like NetCDF only hands data over to the operating system buffers and the NFS server, let alone the client, don't get any chance with the new data since it is held in the writer's system buffers. A call to 'sync' triggers sending of the data over NFS, resulting in an update of the data in the visualization in acceptable time. That is all I'm asking for... I know that this is no substitute for a cluster file system and there is no guarantee, but it supposedly works 99% of times instead of getting a broken plot in 10% of cases, and the plot very late in all cases.

My guess is that the issue isn't with the writer (who would call
fsync()) at all, but the reader.

Of course there are delays and possible inconsistencies on the reader side, but practice shows that our NFS setup seems to be "good enough" for my purpose if it gets the data at all from the writer.

readers cache data and hand that data back to processes without
bothering to check if the data is up-to-date with respect to the
data on the server.

Getting old data would be a different issue: I simply would not get updates of the plot. The plotter needs updated header data to see that there is a new record appended. Once it has figured out that much, it needs to access newly written data which has not been read before on that machine -- the file is being grown just now, where should the cache come from? Anyhow, I said I won't argue about how NFS should behave here, let's focus on what kind of sync()ing NetCDF does / should do.

In any case, one should clarify the documentation...

How would you propose clarifying it? Something like:

 "Note, the view of the file relative to other processes is file
system dependent, so this call is not adequate to ensure that the
most up-to-date file state is available at all processes."

As I see it -- I would still like to see a comment from someone who really knows what NetCDF I/O code does and what it does not -- it would be more accurate like this:

"This does flush any NetCDF-internal buffers of the calling process, handing the data over to the operating system. This should (on any sane operating system) make the data immediately available to other processes on the same machine, but it does not guarantee that any actual write operation takes place on the hard disk or network share the data file is residing on."

And I still hold my point that adding some hook to actually call fsync() (or similar functionality on non-UNIX) on the underlying file handle would be nice. When writing a file in plain C one has the option to tell the system to actually commit the file changes using fsync(). With the NetCDF API one currently does not have that option since the underlying file is hidden from the user code, while the demand for it is valid, IMHO*. But since it can have severe performance drawbacks, fsync() should not be called implicitly -- only on specific request!


Alrighty then,

Thomas.

* Like, having worked on a document over night, duly saving it along the way, but still loosing hours of work during power outage because your text editor did _not_ use fsync() on saves and the kernel filesystem buffering was setup to be quite patient...

--
Dipl. Phys. Thomas Orgis
Atmospheric Modelling
Alfred-Wegener-Institute for Polar and Marine Research

_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/



  • 2009 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: