Re: [netcdfgroup] read performance slow compared to netCDF on other systems

On 11/10/16 4:01 PM, Liam Forbes wrote:
Hello! We are installing netCDF 4.4.1 w/ HDF5 1.8.17 on our new Intel
based cluster. We've noticed the read performance on this cluster using
ncks is extremely slow compared to a couple other systems. For example,
parsing a file on our lustre 2.1 based filesystem takes less 8 seconds
on our Cray XK6-200m. Parsing the same file on the same filesystem on
our new cluster is taking 30+ seconds, with most of that time apparently
spent reading in the file.

Cray (hostname fish):
fish1:lforbes$ time ncks test.nc <http://test.nc> out.nc <http://out.nc>

real    0m4.804s
user    0m3.180s
sys    0m1.300s

Cluster (hostname chinook):
n0:loforbes$ time ncks mod.nc <http://mod.nc> out.nc <http://out.nc>

real    0m32.435s
user    0m29.240s
sys    0m1.936s


A question: are you using the same version of NCO in both cases? A while back (year or two) I encountered a performance issue that was due to NCO and not netCDF (I tried many a combination). Charlie fixed whatever was causing the slowdown a while back, so if you just built NCO it's probably not that, but I wanted to make sure.

Matt

--
Matt Thompson, SSAI, Sr Scientific Programmer/Analyst
NASA GSFC,    Global Modeling and Assimilation Office
Code 610.1,  8800 Greenbelt Rd,  Greenbelt,  MD 20771
Phone: 301-614-6712                 Fax: 301-614-6246
http://science.gsfc.nasa.gov/sed/bio/matthew.thompson



  • 2016 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: