Re: [netcdfgroup] [EXTERNAL] Re: NetCDF parallel I/O configurations

I would like to add one feature of PnetCDF that it supports I/O request 
aggregation
through PnetCDF nonblocking APIs. This feature can combine multiple (small) 
requests
into large one, so to achieve a better performance. This is particularly useful 
if
you have many variables to write and/or read.


Wei-keng

On Mar 2, 2016, at 4:25 PM, dmh@xxxxxxxx wrote:

> That is correct. Pnetcdf only supports the netcdf-3 format and it offshoot
> CDF5. They do not support chunking or compression.
> =Dennis Heimbigner
>  Unidata
> 
> On 3/2/2016 3:01 PM, Kent Yang wrote:
>> I don't think pnetcdf from ANL uses the chunking technique as the HDF5 does. 
>> That may lead to bigger performance difference when some subset patterns get 
>> involved.
>> 
>> Kent
>> 
>> -----Original Message-----
>> From: netcdfgroup-bounces@xxxxxxxxxxxxxxxx 
>> [mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Sjaardema, 
>> Gregory D
>> Sent: Wednesday, March 02, 2016 3:55 PM
>> To: Latham, Robert J.; seanb@xxxxxxxx
>> Cc: netcdfgroup@xxxxxxxxxxxxxxxx
>> Subject: Re: [netcdfgroup] [EXTERNAL] Re: NetCDF parallel I/O configurations
>> 
>> 
>> On 3/2/16, 2:49 PM, "netcdfgroup-bounces@xxxxxxxxxxxxxxxx on behalf of 
>> Latham, Robert J." <netcdfgroup-bounces@xxxxxxxxxxxxxxxx on behalf of 
>> robl@xxxxxxxxxxx> wrote:
>> 
>>> On Tue, 2016-02-23 at 17:13 +0000, Sean Byland wrote:
>>>> Hello,
>>>> I¹m not particularly knowledge on NetCDF but know that it can do
>>>> parallel I/O via parallel HDF5 or ANL¹s/NU's pNetCDF? What would be
>>>> the pros and cons of each configuration?
>>>> 
>>> The HDF5 backend ("new netcdf") allows for some nice features:  VLEN
>>> arrays, compression, multiple dimensions of NC_UNLIMITED.  Those
>>> features come at some cost of metadata.
>> Note that compression can¹t be used in HDF5 backend if doing parallel io.
>> 
>> ŠGreg
>> 
>>> ANL/Northwestern (thank you for mentioning both institutions!) pnetcdf
>>> implements the much simpler classic NetCDF format (CDF-1, CDF-2 and
>>> CDF-5), and takes advantage of the older, more restrictive constraints.
>>> 
>>> If you have very large datasets, you're unlikely to see much difference
>>> between the two approaches, as data movement costs will dominate.
>>> 
>>> One could construct datasets impossible to implement in ANL/NU pnetcdf,
>>> and one could likewise construct pathological datasets (e.g. a thousand
>>> datasets, each with 4k of data in them) that would perform
>>> exceptionally poorly under Unidada NetCDF.
>>> 
>>> Here's a fun game you can play:  let's say you've got a representative
>>> benchmark that shows Unidata NetCDF outperforming ANL/Northwestern
>>> pnetcdf.  Wei-keng and I will defend our professional pride and tune
>>> the heck out of pnetcdf to meet or beat our good-natured competitor.
>>> Likewise, Ward and team would do the same if the results were reversed.
>>> You can get decades worth of experience looking at your workload for
>>> free!
>>> 
>>> =rob
>>> 
>>>> Thanks,
>>>> Sean
>>>> _______________________________________________
>>>> netcdfgroup mailing list
>>>> netcdfgroup@xxxxxxxxxxxxxxxx
>>>> For list information or to unsubscribe,  visit:
>>>> http://www.unidata.ucar.edu/mailing_lists/
>>> 
>>> _______________________________________________
>>> netcdfgroup mailing list
>>> netcdfgroup@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe,  visit:
>>> http://www.unidata.ucar.edu/mailing_lists/
>> _______________________________________________
>> netcdfgroup mailing list
>> netcdfgroup@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe,  visit: 
>> http://www.unidata.ucar.edu/mailing_lists/
>> 
>> _______________________________________________
>> netcdfgroup mailing list
>> netcdfgroup@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe,  visit: 
>> http://www.unidata.ucar.edu/mailing_lists/
> 
> _______________________________________________
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit: 
> http://www.unidata.ucar.edu/mailing_lists/



  • 2016 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: