Re: [netcdfgroup] Informations about parallel netCDF4



On 05/07/2014 09:39 AM, Kent Yang wrote:

-- There should be a paper that listed the flash benchmark comparison between 
parallel NetCDF from Northwest(or parallel netcdf-3) and parallel HDF5. 
However, it is an unfair comparison. It used collective IO for parallel 
NetCDF-3 but independent IO for parallel HDF5.  You can find more detailed 
about the fair comparison with the collective IO for these two packages from 
http://www.spscicomp.org/ScicomP12/Presentations/User/Yang.pdf

netcdf has define mode and data mode separate. this restricts what the user can do, but it also means once you are out of define mode, the metadata will not change.

HDF5's metadata book keeping code means writes require not only a bulk data update, but also mean updating a bit of metadata. not a huge deal if you are moving tons of data, but if you are working with many small datasets, it can be a factor

Be aware this was also a bit old. Don't know what's the current status between 
these two packages.

for most people and most workloads, the simple fact that either pnetcdf or HDF5 is being used is great. I think both libraries have pain points (like when parallel-netcdf tries to read or write one of several record variables ) where performance can suffer.

==rob

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA



  • 2014 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: