[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Fwd: RE: [netcdf-java] Grib1Data and RandomAccessFile performance questions and Java NIO] (fwd)



Fred,

thanks for the code snippet. when i get some time i'll replace the current code. The float4 routine is only called once in the grib 1 module and not at all in the grib2 module. As John mentioned we will look again at NIO.

Robb...

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================


-----Original Message-----
From: John Caron [mailto:address@hidden]
Sent: Monday, February 25, 2008 11:17 AM
To: Thurber, Fred
Subject: Re: [netcdf-java] Grib1Data and RandomAccessFile performance
questions and Java NIO



Thurber, Fred wrote:
I profiled our Grib code (2.2.18) and found a bottleneck in some of
the
low-level routines such as Grib1BinaryDataSection.bits2UInt(),
ucar.unidata.io.RandomAccessFile.read(), and, to a lesser degree, in
ucar.grib.GribNumbers.float4().



Has the UCar team ever thought of moving to Java NIO?

we measured NIO and RandomAccessFile when NIO first came out, and there
was no improvement. However, java 1.6 has had lots of improvements, so
any new measurements would be welcome.


I rewrote the
ucar.grib.GribNumbers.float4() method in NIO and got 10 fold
performance
boost.  I will post the new float4() if anyone is interested.

yes, we'd like to see it.




As far as RandomAccessFile.read(), the reason our program was spending

so much time in it was because of a multiplier inside the
Grib1BinaryDataSection constructor.  So 2,558 calls to the
Grib1BinaryDataSection constructor resulted in 273,175,727 calls to
RandomAccessFile.read()!  Is there anything I can do about that?

since RandomAccessFile.read() is buffered, these are not actual I/O
calls. Still, there may be improvements in the way
Grib1BinaryDataSection works, and any help you can give would be
welcome.


Increase the defaultBufferSize size?  It looks as if we have it set to

8K, but I was thinking of increasing that substantially.  Can I do
this
without a recompile?

Ive done some experiments with different buffer sizes and havent seen a
big improvements with bigger sizes. OTOH, I havent targeted grib files.
You can experiment by using:

  NetcdfFile.open(String location, int buffer_size,
ucar.nc2.util.CancelTask cancelTask);


Im not sure if you are using NetcdfFile or the grib library as
standalone.