Hi John,
I've been doing some more tests on this. Joe was absolutely right, on
the first read of data from a file the situation is completely i/o
bound, but subsequent reads come from disk cache and performance is
limited by the speed of conversion to Java types.
I've also found out that using GridDatatype.makeSubset() is inefficient
for my use case of "many small read operations" since it creates a lot
of georeferencing objects with each invocation (profiling shows that
most of the time is spent creating the LatLonRect, strangely). I've
found it much more efficient to use the lower-level objects like
Variable.read() to do the reading, having used the GridDatatype to
identify which axis is which. This means that there is very little
penalty for each read operation, particularly if the read operations are
sorted in order of increasing offset (this means that buffering in the
RAF does its job well). It actually turns out to be quite efficient to
read data point-by-point to satisfy the use case of reading sparse
points from a data file.
This may ultimately bubble up to simplifications in THREDDS-WMS, which
I'll let you know about after more experimentation. It's possible that
the WMS will no longer need the capability to open datasets with
Enhance.ScaleMissingDefer.
Thanks for your help,
Jon
-----Original Message-----
From: John Caron [mailto:caron@xxxxxxxxxxxxxxxx]
Sent: 19 July 2010 15:23
To: Jon Blower
Cc: netcdf-java@xxxxxxxxxxxxxxxx
Subject: Re: [netcdf-java] Reading contiguous data in NetCDF files
Hi Jon:
Jon Blower wrote:
> Thanks John and Joe. Yes, I do know that disk I/O is the limiting
> factor, but optimising it isn't easy due to all the buffers and disk
> caches (as you and Joe have pointed out). Interestingly, I can "see"
> these caches. When I read random chunks of data from a file,
sometimes
> a read takes ~1ms, sometimes ~5ms and sometimes ~10ms, with not much
in
> between these values (a trimodal distribution). I think these must be
> three levels of caching. Also, if I run the same test multiple times
on
> the same file, the number of 10ms reads drops off, and the number of
1ms
> reads increases. (I'm on a Windows XP laptop with a 5400 rpm hard
drive.)
>
>
>
> I guess the only way to bypass the caches would be to cycle between a
> large set of data files, which are in total bigger than the disk
caches.
> (I'm trying to simulate a busy server environment.)
If your server is running linux or solaris instead of windows XP, your
will have different I/O results.
>
>
>
> By the way, I've been digging in the IOSPs and the ucar
RandomAccessFile
> class. The ucar RAF seems to be the same as java.io.RAF except that
it
> implements an 8k buffer which is supposed to increase performance.
But
> the code of N3raf (which extends N3iosp and I assume is the default
> class used for data reading) uses raf.readToByteChannel(), which
> bypasses the 8k buffer. So could a java.io.RAF have been used in this
case?
>
ucar.RAF forked java.RAF to add buffering back in Java 1.0 days. It has
accumulated various other conveniences since then, i think byte ordering
is one (?). also, java.RAF is a final class so cant be sublassed by
HTTPRandomAccessFile. For these and various reasons, one could not
revert to use java.RAF except by forking the CDM code.
raf.readToByteChannel() is an experiment to try to allow "streamed
reading", ie direct file to network transfer. It can be used in such a
restricted manner that its not very useful generally. Its not used by
the Grid/WMS code.
>
>
> To expand a little on my use case: in general, to create a
> low-resolution map of data for a WMS, one has to read only a small
> fraction of the available data in the file. So I'm looking for an
> efficient way to read sparse clouds of data (not evenly-spaced).
> Reading point-by-point is not efficient, but nor is reading lots of
> data, converting it to new types, then throwing most of it away.
What about writing low-resolution versions of the data and using that
when possible?
>
>
>
> Cheers, Jon
