That is correct. Pnetcdf only supports the netcdf-3 format and it offshoot
CDF5. They do not support chunking or compression.
=Dennis Heimbigner
Unidata
On 3/2/2016 3:01 PM, Kent Yang wrote:
I don't think pnetcdf from ANL uses the chunking technique as the HDF5 does.
That may lead to bigger performance difference when some subset patterns get
involved.
Kent
-----Original Message-----
From: netcdfgroup-bounces@xxxxxxxxxxxxxxxx
[mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Sjaardema, Gregory D
Sent: Wednesday, March 02, 2016 3:55 PM
To: Latham, Robert J.; seanb@xxxxxxxx
Cc: netcdfgroup@xxxxxxxxxxxxxxxx
Subject: Re: [netcdfgroup] [EXTERNAL] Re: NetCDF parallel I/O configurations
On 3/2/16, 2:49 PM, "netcdfgroup-bounces@xxxxxxxxxxxxxxxx on behalf of Latham, Robert
J." <netcdfgroup-bounces@xxxxxxxxxxxxxxxx on behalf of robl@xxxxxxxxxxx> wrote:
On Tue, 2016-02-23 at 17:13 +0000, Sean Byland wrote:
Hello,
I¹m not particularly knowledge on NetCDF but know that it can do
parallel I/O via parallel HDF5 or ANL¹s/NU's pNetCDF? What would be
the pros and cons of each configuration?
The HDF5 backend ("new netcdf") allows for some nice features: VLEN
arrays, compression, multiple dimensions of NC_UNLIMITED. Those
features come at some cost of metadata.
Note that compression can¹t be used in HDF5 backend if doing parallel io.
ŠGreg
ANL/Northwestern (thank you for mentioning both institutions!) pnetcdf
implements the much simpler classic NetCDF format (CDF-1, CDF-2 and
CDF-5), and takes advantage of the older, more restrictive constraints.
If you have very large datasets, you're unlikely to see much difference
between the two approaches, as data movement costs will dominate.
One could construct datasets impossible to implement in ANL/NU pnetcdf,
and one could likewise construct pathological datasets (e.g. a thousand
datasets, each with 4k of data in them) that would perform
exceptionally poorly under Unidada NetCDF.
Here's a fun game you can play: let's say you've got a representative
benchmark that shows Unidata NetCDF outperforming ANL/Northwestern
pnetcdf. Wei-keng and I will defend our professional pride and tune
the heck out of pnetcdf to meet or beat our good-natured competitor.
Likewise, Ward and team would do the same if the results were reversed.
You can get decades worth of experience looking at your workload for
free!
=rob
Thanks,
Sean
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
