Hi Rob
1 MB/s compares with more like 400 MB/s from the native file system from a
shallower stack … so we’re pretty sure there is some sort of buffering
involved somewhere (everywhere) in the netcdf/hdf/python stack. We’re after
some pointers to understand what buffers are where, particularly in the
netcdf stack itself.
Cheers
Bryan
On 30 October 2015 at 14:41, <robl@xxxxxxxxxxx> wrote:
>
>
> On 10/28/2015 09:57 AM, Matthew Jones wrote:
> > I am running some tests on an HPC cluster, altering the size of reads to
> > test the performance of the file system.
> >
> >
> > I am using python, and for sequential reads not using netCDF4 the read
> > rate is pretty constant across different read sizes. However, when I
> > introduce the netCDF4 library the smaller and larger reads see a dip in
> > performance with a peak on the medium sized reads (creating a hill-like
> > profile). The peak in the netCDF4 performance is at about the same read
> > rate as the non-netCDF4 reads. The peak is at reads of about 1MB.
> >
> >
> > We think this could be to do with buffering somewhere in the NetCDF
> > library. Does anyone know of such buffering that we should be aware of?
>
> Your software stack is pretty deep here. 1 MB might be a perfectly
> reasonable rate, but some additional experiments (or information from
> your HPC consultants) would help.
>
> Your message regards sequential access. It's possible the file system
> has been tuned for parallel access. Is peak parallel performance
> something you might care about some day soon? The tuning for a parallel
> access will be a bit different from serial access.
>
> ==rob
>
>
>
> >
> >
> > Many thanks
> >
> > Matt
> >
> >
> >
> > ----------------------------------------
> > Matthew Jones
> > PhD Student
> > Atmosphere, Oceans and Climate
> > Department of Meteorology,
> > University of Reading
> >
> > Room 288, ESSC, Harry Pitt Building,
> > 3 Earley Gate, Reading, RG6 6AL, UK
> >
> > Ext: 5214
> >
> > https://www.linkedin.com/pub/matthew-jones/8b/b81/25a
> > http://www.met.reading.ac.uk/users/users/1887
> >
> >
> >
> >
> > _______________________________________________
> > netcdfgroup mailing list
> > netcdfgroup@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
> >
>
> --
> Rob Latham
> Mathematics and Computer Science Division
> Argonne National Lab, IL USA
>
> _______________________________________________
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
--
Bryan Lawrence
University of Reading: Professor of Weather and Climate Computing.
National Centre for Atmospheric Science: Director of Models and Data.
STFC: Director of the Centre for Environmental Data Analysis.
Ph: +44 118 3786507 or 1235 445012; Web:home.badc.rl.ac.uk/lawrence
