Re: [netcdf-java] Grib1Data and RandomAccessFile performance questions and Java NIO

  • To: "John Caron" <caron@xxxxxxxxxxxxxxxx>
  • Subject: Re: [netcdf-java] Grib1Data and RandomAccessFile performance questions and Java NIO
  • From: "Thurber, Fred" <Fred.Thurber@xxxxxxxx>
  • Date: Mon, 25 Feb 2008 11:50:59 -0500
Here is my re-write of float4() with NIO; its simplicity is another
reason it is so fast (10 to 11x).  I was talking about this to our
resident Grib-ologist, and he was wondering about the complexity of the
original float4.  Could the Grib float be in some sort of non-standard
format?  I should also note that our Grib-ologist does not think that
there are many/any floats in typical Gribs so you will not get much of
an overall speedup.  However if you operate on floats, it could make a
huge different. In any case it is a nice demonstration of NIO Buffers:


    import java.nio.ByteBuffer;

    static ByteBuffer bb = ByteBuffer.allocate(4);

    private static final float float4 (final byte a, final byte b, final
byte c, final byte d)
    {
        bb.clear();
        bb.put(a).put(b).put(c).put(d);
        return bb.getFloat(0);
    }    


I have attached my whole test program, but sometime attachments get
stripped before being sent out.  It is ready to compile and run.






-----Original Message-----
From: John Caron [mailto:caron@xxxxxxxxxxxxxxxx] 
Sent: Monday, February 25, 2008 11:17 AM
To: Thurber, Fred
Cc: netcdf-java@xxxxxxxxxxxxxxxx
Subject: Re: [netcdf-java] Grib1Data and RandomAccessFile performance
questions and Java NIO



Thurber, Fred wrote:
> I profiled our Grib code (2.2.18) and found a bottleneck in some of
the 
> low-level routines such as Grib1BinaryDataSection.bits2UInt(), 
> ucar.unidata.io.RandomAccessFile.read(), and, to a lesser degree, in 
> ucar.grib.GribNumbers.float4().
> 
>  
> 
> Has the UCar team ever thought of moving to Java NIO?   

we measured NIO and RandomAccessFile when NIO first came out, and there
was no improvement. However, java 1.6 has had lots of improvements, so
any new measurements would be welcome.


I rewrote the 
> ucar.grib.GribNumbers.float4() method in NIO and got 10 fold
performance 
> boost.  I will post the new float4() if anyone is interested.

yes, we'd like to see it.

> 
>  
> 
> As far as RandomAccessFile.read(), the reason our program was spending

> so much time in it was because of a multiplier inside the 
> Grib1BinaryDataSection constructor.  So 2,558 calls to the 
> Grib1BinaryDataSection constructor resulted in 273,175,727 calls to 
> RandomAccessFile.read()!  Is there anything I can do about that?  

since RandomAccessFile.read() is buffered, these are not actual I/O
calls. Still, there may be improvements in the way
Grib1BinaryDataSection works, and any help you can give would be
welcome.


> Increase the defaultBufferSize size?  It looks as if we have it set to

> 8K, but I was thinking of increasing that substantially.  Can I do
this 
> without a recompile? 

Ive done some experiments with different buffer sizes and havent seen a
big improvements with bigger sizes. OTOH, I havent targeted grib files.
You can experiment by using:

  NetcdfFile.open(String location, int buffer_size,
ucar.nc2.util.CancelTask cancelTask);


Im not sure if you are using NetcdfFile or the grib library as
standalone.

Attachment: FourFloat.java
Description: FourFloat.java

  • 2008 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: