If you are only using 15 processors, I would suggest using a 'round robin'
approach with non-parallel, chunked and compressed output. Essentially, each
processor writes in succession to the file (open, write, close, next
processor). This works really well for me for smallish numbers of processors
(less than 60, say). If the chunking is set up such that each processor writes
just its own chunks, this method works well.
good luck!
-- Ted
On Jun 5, 2012, at 10:23 AM, Kristopher Bedka wrote:
> I'm not quite sure what you mean by the "application layer"? My goal was to
> have 15 different processors process 15 segments of a satellite orbit, where
> each processor would write to the same NetCDF file in the most disk space
> efficient manner possible without any problems with simultaneous NetCDF
> writes. I had previously done the compression with the "nf_def_var_chunking"
> function call in non-parallel NetCDF. As this function does not seem to be
> available in parallel NetCDF, I'd be interested in alternative suggestions to
> accomplish my goal. Sorry I am more of the scientist type and am not a
> software engineer, so I may absorb some of these concepts a little slower
> than others.
>
> Thanks for the help,
> Kris
>
> On Jun 5, 2012, at 11:13 AM, Rob Latham wrote:
>
>> On Tue, May 29, 2012 at 02:29:12PM -0600, Russ Rew wrote:
>>> Hi Kristopher,
>>>
>>>> I am processing a large volume of satellite data where multiple
>>>> processes could be simultaneously writing data to the same netcdf
>>>> file. This has not been supported in previous NetCDF versions and
>>>> I've gotten fatal errors when two simultaneous writes conflicted. I
>>>> now understand that recent NetCDF versions do support this
>>>> functionality. Could someone tell me or provide an example of what I
>>>> need to do (i.e. new
>>>> function calls, options in netcdf open, etc...) to make this work for
>>>> me? I've tried the pnetcdf package does not support chunking which
>>>> I need to internally compress these files.
>>>
>>> No, sorry, it's not supported in current netCDF versions either.
>>> NetCDF-4 uses HDF5 as its storage layer, and HDF5 does not support
>>> compression with parallel access, as explained here:
>>
>> Is there any chance you can compress at the application layer? Each
>> processor takes it's local hunk of data, compresses it, then writes to
>> the file.
>>
>> I admit, you will quickly find out why parallel writes with
>> compression is not already implemented in these parallel I/O
>> libraries!
>>
>> However, it's possible that at your application level, there may be
>> ways to simplify the parallel, compressed writes problem that a
>> general purpose library cannot use.
>>
>> ==rob
>>
>> --
>> Rob Latham
>> Mathematics and Computer Science Division
>> Argonne National Lab, IL USA
>
> =========================================================
> Kristopher Bedka
> Science Systems & Applications, Inc. @ NASA Langley Research Center
> Climate Science Branch
> 1 Enterprise Parkway, Suite 200
> Hampton, VA 23666
> Phone: (757) 951-1920
> Fax: (757) 951-1902
> Kristopher.m.bedka@xxxxxxxx
> =========================================================
>
>
>
>
>
>
> _______________________________________________
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
