On 10/28/2015 09:57 AM, Matthew Jones wrote:
I am running some tests on an HPC cluster, altering the size of reads to
test the performance of the file system.
I am using python, and for sequential reads not using netCDF4 the read
rate is pretty constant across different read sizes. However, when I
introduce the netCDF4 library the smaller and larger reads see a dip in
performance with a peak on the medium sized reads (creating a hill-like
profile). The peak in the netCDF4 performance is at about the same read
rate as the non-netCDF4 reads. The peak is at reads of about 1MB.
We think this could be to do with buffering somewhere in the NetCDF
library. Does anyone know of such buffering that we should be aware of?
Your software stack is pretty deep here. 1 MB might be a perfectly
reasonable rate, but some additional experiments (or information from
your HPC consultants) would help.
Your message regards sequential access. It's possible the file system
has been tuned for parallel access. Is peak parallel performance
something you might care about some day soon? The tuning for a parallel
access will be a bit different from serial access.
==rob
Many thanks
Matt
----------------------------------------
Matthew Jones
PhD Student
Atmosphere, Oceans and Climate
Department of Meteorology,
University of Reading
Room 288, ESSC, Harry Pitt Building,
3 Earley Gate, Reading, RG6 6AL, UK
Ext: 5214
https://www.linkedin.com/pub/matthew-jones/8b/b81/25a
http://www.met.reading.ac.uk/users/users/1887
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA
