Hi, Heiko
In that case, you can use independent mode.
I.e. nc_var_par_access(ncid, varid, NC_INDEPENDENT)
It still allows you to write to a shared file from multiple
MPI processes independently, at different time.
However, the performance will not be as good as the collective mode.
Wei-keng
On Sep 22, 2015, at 1:45 AM, Heiko Klein wrote:
> Hei Wei-keng,
>
> thanks for your tip about using pnetcdf. I've worked with MPI, but only
> for modeling, i.e. when all processes do approximately the same thing at
> the same time.
>
> The problem here is that the 10 input-files don't appear on my machines
> at the same time. They are ensemble members and downloaded from
> different machines with different processors, so the first file might
> appear 30s before the last file (within a total time-step time of 2
> minutes). I would like to start as soon as the first file appears, but
> this sounds very difficult with MPI, isn't it? (I'm more familiar with
> OpenMP, and there exist task-based parallelization (what I would use
> here), and loop-base parallelization (which is more like MPI?))
>
> Best regards,
>
> Heiko
>
> On 2015-09-22 03:24, Wei-keng Liao wrote:
>> Hi, Heiko
>>
>> Parallel I/O to the classical netCDF format is supported by netCDF through
>> PnetCDF underneath.
>> It allows you to write concurrently to a single shared file from multiple
>> MPI processes.
>> Of course, you will have to build PnetCDF first and then build netCDF with
>> --enable-pnetcdf configure option.
>>
>> Your netCDF program does not need much changes to make use this feature. All
>> you have to
>> do is the followings.
>> 1. call nc_create_par() instead of nc_create()
>> 2. add NC_PNETCDF to the create mode argument of nc_create_par
>> 3. call nc_var_par_access(ncid, varid, NC_COLLECTIVE) after nc_enddef to
>> enable collective I/O mode
>>
>> There are a couple example codes available in this URL.
>> http://cucis.ece.northwestern.edu/projects/PnetCDF/#InteroperabilityWithNetCDF4
>>
>> There are instructions in each example file for building netCDF with PnetCDF.
>> For downloading PnetCDF, please see
>> http://cucis.ece.northwestern.edu/projects/PnetCDF/download.html
>>
>> Wei-keng
>>
>> On Sep 21, 2015, at 9:14 AM, Heiko Klein wrote:
>>
>>> Hi Nick,
>>>
>>> yes, they are all writing to the same file - we want to have one file at
>>> the end.
>>>
>>> I've been scanning through the source-code of netcdf3. I guess the
>>> problem of the partly written sections is caused by the translation of
>>> the nc_put_vara calls to internal pages, and the from the internal pages
>>> to disk. And eventually, the internal pages are not aligned with my
>>> nc_put_vara calls, so even when the region of nc_put_vara doesn't
>>> overlap between concurrent calls, the internal pages do? Is there a way
>>> to enforce proper alignment? I see nc__enddef has several align parameters.
>>>
>>>
>>> I'm aware that concurrent writes are not officially supported by the
>>> netcdf-library. But IT-infrastructure has changed a lot since the start
>>> of the netcdf-library and systems are nowadays highly parallelized, both
>>> on CPU and also in IO/filesystems. I'm trying to find a way to allow for
>>> simple parallelization. Having many output-files from a model is risky
>>> for data-consistency - so I would like to avoid it without sacrificing
>>> to much speed.
>>>
>>> Best regards,
>>>
>>> Heiko
>>>
>>>
>>> On 2015-09-21 15:18, Nick Papior wrote:
>>>> So, are they writing to the same files?
>>>>
>>>> I.e. job1 writes a(:,1) to test.nc <http://test.nc> and job2 writes
>>>> a(:,2) to test.nc <http://test.nc>?
>>>> Because that is not allowed.
>>>>
>>>> 2015-09-21 15:13 GMT+02:00 Heiko Klein <Heiko.Klein@xxxxxx
>>>> <mailto:Heiko.Klein@xxxxxx>>:
>>>>
>>>> Hi,
>>>>
>>>> I'm trying to convert about 90GB of NWP data 4 times daily from grib to
>>>> netcdf. The grib-files arrive as fast as the data can be downloaded from
>>>> the HPC machines. They come by 10 files/forecast timestep.
>>>>
>>>> Currently, I manage to convert 1 file/forecast timestep and I would like
>>>> to parallelize the conversion into independent jobs (i.e. neither MPI or
>>>> OpenMP), with a theoretical performance increase of 10. The underlying
>>>> IO system is fast enough to handle 10 jobs, and I have enough CPUs, but
>>>> the concurrently written netcdf-files show data which is only written
>>>> half to the disk, or mixed with other slices.
>>>>
>>>> What I do is create a _FILL_VALUE 'template' file, containing all
>>>> definitions before the NWP job runs. When a new set of files arrives,
>>>> the data is put to the respective data-slices which don't have any
>>>> overlap, there is never a redefine, only functions like: nc_put_vara_*
>>>> with different slices.
>>>>
>>>> Since the nc_put_vara_* calls are non-overlapping, I hoped that this
>>>> type of concurrent write would work - but it doesn't. Is my idea really
>>>> so bad to write data in parallel (e.g. there are internal buffers which
>>>> are rewritten)? Any ideas how to improve the conversion process?
>>>>
>>>> Best regards,
>>>>
>>>> Heiko
>>>>
>>>> _______________________________________________
>>>> netcdfgroup mailing list
>>>> netcdfgroup@xxxxxxxxxxxxxxxx <mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
>>>> For list information or to unsubscribe, visit:
>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Kind regards Nick
>>>
>>> --
>>> Dr. Heiko Klein Norwegian Meteorological Institute
>>> Tel. + 47 22 96 32 58 P.O. Box 43 Blindern
>>> http://www.met.no 0313 Oslo NORWAY
>>>
>>> _______________________________________________
>>> netcdfgroup mailing list
>>> netcdfgroup@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit:
>>> http://www.unidata.ucar.edu/mailing_lists/
>>
>
> --
> Dr. Heiko Klein Norwegian Meteorological Institute
> Tel. + 47 22 96 32 58 P.O. Box 43 Blindern
> http://www.met.no 0313 Oslo NORWAY
