I don't think pnetcdf from ANL uses the chunking technique as the HDF5 does.
That may lead to bigger performance difference when some subset patterns get
involved.
Kent
-----Original Message-----
From: netcdfgroup-bounces@xxxxxxxxxxxxxxxx
[mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Sjaardema, Gregory D
Sent: Wednesday, March 02, 2016 3:55 PM
To: Latham, Robert J.; seanb@xxxxxxxx
Cc: netcdfgroup@xxxxxxxxxxxxxxxx
Subject: Re: [netcdfgroup] [EXTERNAL] Re: NetCDF parallel I/O configurations
On 3/2/16, 2:49 PM, "netcdfgroup-bounces@xxxxxxxxxxxxxxxx on behalf of Latham,
Robert J." <netcdfgroup-bounces@xxxxxxxxxxxxxxxx on behalf of robl@xxxxxxxxxxx>
wrote:
>On Tue, 2016-02-23 at 17:13 +0000, Sean Byland wrote:
>> Hello,
>> I¹m not particularly knowledge on NetCDF but know that it can do
>>parallel I/O via parallel HDF5 or ANL¹s/NU's pNetCDF? What would be
>>the pros and cons of each configuration?
>>
>
>The HDF5 backend ("new netcdf") allows for some nice features: VLEN
>arrays, compression, multiple dimensions of NC_UNLIMITED. Those
>features come at some cost of metadata.
Note that compression can¹t be used in HDF5 backend if doing parallel io.
ŠGreg
>
>ANL/Northwestern (thank you for mentioning both institutions!) pnetcdf
>implements the much simpler classic NetCDF format (CDF-1, CDF-2 and
>CDF-5), and takes advantage of the older, more restrictive constraints.
>
>If you have very large datasets, you're unlikely to see much difference
>between the two approaches, as data movement costs will dominate.
>
>One could construct datasets impossible to implement in ANL/NU pnetcdf,
>and one could likewise construct pathological datasets (e.g. a thousand
>datasets, each with 4k of data in them) that would perform
>exceptionally poorly under Unidada NetCDF.
>
>Here's a fun game you can play: let's say you've got a representative
>benchmark that shows Unidata NetCDF outperforming ANL/Northwestern
>pnetcdf. Wei-keng and I will defend our professional pride and tune
>the heck out of pnetcdf to meet or beat our good-natured competitor.
>Likewise, Ward and team would do the same if the results were reversed.
>You can get decades worth of experience looking at your workload for
>free!
>
>=rob
>
>> Thanks,
>> Sean
>> _______________________________________________
>> netcdfgroup mailing list
>> netcdfgroup@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>>http://www.unidata.ucar.edu/mailing_lists/
>
>
>_______________________________________________
>netcdfgroup mailing list
>netcdfgroup@xxxxxxxxxxxxxxxx
>For list information or to unsubscribe, visit:
>http://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
