On 11/10/16 4:01 PM, Liam Forbes wrote:
Hello! We are installing netCDF 4.4.1 w/ HDF5 1.8.17 on our new Intel
based cluster. We've noticed the read performance on this cluster using
ncks is extremely slow compared to a couple other systems. For example,
parsing a file on our lustre 2.1 based filesystem takes less 8 seconds
on our Cray XK6-200m. Parsing the same file on the same filesystem on
our new cluster is taking 30+ seconds, with most of that time apparently
spent reading in the file.
Cray (hostname fish):
fish1:lforbes$ time ncks test.nc <http://test.nc> out.nc <http://out.nc>
real 0m4.804s
user 0m3.180s
sys 0m1.300s
Cluster (hostname chinook):
n0:loforbes$ time ncks mod.nc <http://mod.nc> out.nc <http://out.nc>
real 0m32.435s
user 0m29.240s
sys 0m1.936s
A question: are you using the same version of NCO in both cases? A while
back (year or two) I encountered a performance issue that was due to NCO
and not netCDF (I tried many a combination). Charlie fixed whatever was
causing the slowdown a while back, so if you just built NCO it's
probably not that, but I wanted to make sure.
Matt
--
Matt Thompson, SSAI, Sr Scientific Programmer/Analyst
NASA GSFC, Global Modeling and Assimilation Office
Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771
Phone: 301-614-6712 Fax: 301-614-6246
http://science.gsfc.nasa.gov/sed/bio/matthew.thompson
