On Tue, May 29, 2012 at 02:29:12PM -0600, Russ Rew wrote:
> Hi Kristopher,
>
> > I am processing a large volume of satellite data where multiple
> > processes could be simultaneously writing data to the same netcdf
> > file. This has not been supported in previous NetCDF versions and
> > I've gotten fatal errors when two simultaneous writes conflicted. I
> > now understand that recent NetCDF versions do support this
> > functionality. Could someone tell me or provide an example of what I
> > need to do (i.e. new
> > function calls, options in netcdf open, etc...) to make this work for
> > me? I've tried the pnetcdf package does not support chunking which
> > I need to internally compress these files.
>
> No, sorry, it's not supported in current netCDF versions either.
> NetCDF-4 uses HDF5 as its storage layer, and HDF5 does not support
> compression with parallel access, as explained here:
Is there any chance you can compress at the application layer? Each
processor takes it's local hunk of data, compresses it, then writes to
the file.
I admit, you will quickly find out why parallel writes with
compression is not already implemented in these parallel I/O
libraries!
However, it's possible that at your application level, there may be
ways to simplify the parallel, compressed writes problem that a
general purpose library cannot use.
==rob
--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA
