Hei Wei-keng,
thanks for your tip about using pnetcdf. I've worked with MPI, but only
for modeling, i.e. when all processes do approximately the same thing at
the same time.
The problem here is that the 10 input-files don't appear on my machines
at the same time. They are ensemble members and downloaded from
different machines with different processors, so the first file might
appear 30s before the last file (within a total time-step time of 2
minutes). I would like to start as soon as the first file appears, but
this sounds very difficult with MPI, isn't it? (I'm more familiar with
OpenMP, and there exist task-based parallelization (what I would use
here), and loop-base parallelization (which is more like MPI?))
Best regards,
Heiko
On 2015-09-22 03:24, Wei-keng Liao wrote:
> Hi, Heiko
>
> Parallel I/O to the classical netCDF format is supported by netCDF through
> PnetCDF underneath.
> It allows you to write concurrently to a single shared file from multiple MPI
> processes.
> Of course, you will have to build PnetCDF first and then build netCDF with
> --enable-pnetcdf configure option.
>
> Your netCDF program does not need much changes to make use this feature. All
> you have to
> do is the followings.
> 1. call nc_create_par() instead of nc_create()
> 2. add NC_PNETCDF to the create mode argument of nc_create_par
> 3. call nc_var_par_access(ncid, varid, NC_COLLECTIVE) after nc_enddef to
> enable collective I/O mode
>
> There are a couple example codes available in this URL.
> http://cucis.ece.northwestern.edu/projects/PnetCDF/#InteroperabilityWithNetCDF4
>
> There are instructions in each example file for building netCDF with PnetCDF.
> For downloading PnetCDF, please see
> http://cucis.ece.northwestern.edu/projects/PnetCDF/download.html
>
> Wei-keng
>
> On Sep 21, 2015, at 9:14 AM, Heiko Klein wrote:
>
>> Hi Nick,
>>
>> yes, they are all writing to the same file - we want to have one file at
>> the end.
>>
>> I've been scanning through the source-code of netcdf3. I guess the
>> problem of the partly written sections is caused by the translation of
>> the nc_put_vara calls to internal pages, and the from the internal pages
>> to disk. And eventually, the internal pages are not aligned with my
>> nc_put_vara calls, so even when the region of nc_put_vara doesn't
>> overlap between concurrent calls, the internal pages do? Is there a way
>> to enforce proper alignment? I see nc__enddef has several align parameters.
>>
>>
>> I'm aware that concurrent writes are not officially supported by the
>> netcdf-library. But IT-infrastructure has changed a lot since the start
>> of the netcdf-library and systems are nowadays highly parallelized, both
>> on CPU and also in IO/filesystems. I'm trying to find a way to allow for
>> simple parallelization. Having many output-files from a model is risky
>> for data-consistency - so I would like to avoid it without sacrificing
>> to much speed.
>>
>> Best regards,
>>
>> Heiko
>>
>>
>> On 2015-09-21 15:18, Nick Papior wrote:
>>> So, are they writing to the same files?
>>>
>>> I.e. job1 writes a(:,1) to test.nc <http://test.nc> and job2 writes
>>> a(:,2) to test.nc <http://test.nc>?
>>> Because that is not allowed.
>>>
>>> 2015-09-21 15:13 GMT+02:00 Heiko Klein <Heiko.Klein@xxxxxx
>>> <mailto:Heiko.Klein@xxxxxx>>:
>>>
>>> Hi,
>>>
>>> I'm trying to convert about 90GB of NWP data 4 times daily from grib to
>>> netcdf. The grib-files arrive as fast as the data can be downloaded from
>>> the HPC machines. They come by 10 files/forecast timestep.
>>>
>>> Currently, I manage to convert 1 file/forecast timestep and I would like
>>> to parallelize the conversion into independent jobs (i.e. neither MPI or
>>> OpenMP), with a theoretical performance increase of 10. The underlying
>>> IO system is fast enough to handle 10 jobs, and I have enough CPUs, but
>>> the concurrently written netcdf-files show data which is only written
>>> half to the disk, or mixed with other slices.
>>>
>>> What I do is create a _FILL_VALUE 'template' file, containing all
>>> definitions before the NWP job runs. When a new set of files arrives,
>>> the data is put to the respective data-slices which don't have any
>>> overlap, there is never a redefine, only functions like: nc_put_vara_*
>>> with different slices.
>>>
>>> Since the nc_put_vara_* calls are non-overlapping, I hoped that this
>>> type of concurrent write would work - but it doesn't. Is my idea really
>>> so bad to write data in parallel (e.g. there are internal buffers which
>>> are rewritten)? Any ideas how to improve the conversion process?
>>>
>>> Best regards,
>>>
>>> Heiko
>>>
>>> _______________________________________________
>>> netcdfgroup mailing list
>>> netcdfgroup@xxxxxxxxxxxxxxxx <mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
>>> For list information or to unsubscribe, visit:
>>> http://www.unidata.ucar.edu/mailing_lists/
>>>
>>>
>>>
>>>
>>> --
>>> Kind regards Nick
>>
>> --
>> Dr. Heiko Klein Norwegian Meteorological Institute
>> Tel. + 47 22 96 32 58 P.O. Box 43 Blindern
>> http://www.met.no 0313 Oslo NORWAY
>>
>> _______________________________________________
>> netcdfgroup mailing list
>> netcdfgroup@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>
--
Dr. Heiko Klein Norwegian Meteorological Institute
Tel. + 47 22 96 32 58 P.O. Box 43 Blindern
http://www.met.no 0313 Oslo NORWAY
