I would like to add one feature of PnetCDF that it supports I/O request
aggregation
through PnetCDF nonblocking APIs. This feature can combine multiple (small)
requests
into large one, so to achieve a better performance. This is particularly useful
if
you have many variables to write and/or read.
Wei-keng
On Mar 2, 2016, at 4:25 PM, dmh@xxxxxxxx wrote:
> That is correct. Pnetcdf only supports the netcdf-3 format and it offshoot
> CDF5. They do not support chunking or compression.
> =Dennis Heimbigner
> Unidata
>
> On 3/2/2016 3:01 PM, Kent Yang wrote:
>> I don't think pnetcdf from ANL uses the chunking technique as the HDF5 does.
>> That may lead to bigger performance difference when some subset patterns get
>> involved.
>>
>> Kent
>>
>> -----Original Message-----
>> From: netcdfgroup-bounces@xxxxxxxxxxxxxxxx
>> [mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Sjaardema,
>> Gregory D
>> Sent: Wednesday, March 02, 2016 3:55 PM
>> To: Latham, Robert J.; seanb@xxxxxxxx
>> Cc: netcdfgroup@xxxxxxxxxxxxxxxx
>> Subject: Re: [netcdfgroup] [EXTERNAL] Re: NetCDF parallel I/O configurations
>>
>>
>> On 3/2/16, 2:49 PM, "netcdfgroup-bounces@xxxxxxxxxxxxxxxx on behalf of
>> Latham, Robert J." <netcdfgroup-bounces@xxxxxxxxxxxxxxxx on behalf of
>> robl@xxxxxxxxxxx> wrote:
>>
>>> On Tue, 2016-02-23 at 17:13 +0000, Sean Byland wrote:
>>>> Hello,
>>>> I¹m not particularly knowledge on NetCDF but know that it can do
>>>> parallel I/O via parallel HDF5 or ANL¹s/NU's pNetCDF? What would be
>>>> the pros and cons of each configuration?
>>>>
>>> The HDF5 backend ("new netcdf") allows for some nice features: VLEN
>>> arrays, compression, multiple dimensions of NC_UNLIMITED. Those
>>> features come at some cost of metadata.
>> Note that compression can¹t be used in HDF5 backend if doing parallel io.
>>
>> ŠGreg
>>
>>> ANL/Northwestern (thank you for mentioning both institutions!) pnetcdf
>>> implements the much simpler classic NetCDF format (CDF-1, CDF-2 and
>>> CDF-5), and takes advantage of the older, more restrictive constraints.
>>>
>>> If you have very large datasets, you're unlikely to see much difference
>>> between the two approaches, as data movement costs will dominate.
>>>
>>> One could construct datasets impossible to implement in ANL/NU pnetcdf,
>>> and one could likewise construct pathological datasets (e.g. a thousand
>>> datasets, each with 4k of data in them) that would perform
>>> exceptionally poorly under Unidada NetCDF.
>>>
>>> Here's a fun game you can play: let's say you've got a representative
>>> benchmark that shows Unidata NetCDF outperforming ANL/Northwestern
>>> pnetcdf. Wei-keng and I will defend our professional pride and tune
>>> the heck out of pnetcdf to meet or beat our good-natured competitor.
>>> Likewise, Ward and team would do the same if the results were reversed.
>>> You can get decades worth of experience looking at your workload for
>>> free!
>>>
>>> =rob
>>>
>>>> Thanks,
>>>> Sean
>>>> _______________________________________________
>>>> netcdfgroup mailing list
>>>> netcdfgroup@xxxxxxxxxxxxxxxx
>>>> For list information or to unsubscribe, visit:
>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>
>>> _______________________________________________
>>> netcdfgroup mailing list
>>> netcdfgroup@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit:
>>> http://www.unidata.ucar.edu/mailing_lists/
>> _______________________________________________
>> netcdfgroup mailing list
>> netcdfgroup@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>>
>> _______________________________________________
>> netcdfgroup mailing list
>> netcdfgroup@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>
> _______________________________________________
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
