Re: [netcdf-java] I/O perfomance

Hi Antonio,

I just checked something similar with the netcdf-c library and I manage
to read > 1600MB/s (the file is cached, and memory bandwidth is 6GB/s).
Your 20MB/s are far too low.

That said, java virtual machines are beasts concerning performance
measurements due to JIT compilation and object-handling. As a rule of
thumb, to get realistic performance of a java routine, run it first 1000
times, then wait for the compiler to finish and then start measuring
your routine. Otherwise, you only measure java-overhead.

Heiko



On 2016-09-21 22:40, Ryan May wrote:
> Antonio,
> 
> If you're just reading the entire variable, try using the read() call
> with no parameters, which will read all of the data. You can also pass
> null for origin and shape.
> 
> Ryan
> 
> On Wed, Sep 21, 2016 at 2:07 PM, Antonio Rodriges <antonio.rrz@xxxxxxxxx
> <mailto:antonio.rrz@xxxxxxxxx>> wrote:
> 
>     Hello,
> 
>     I have written a simple test that opens NetCDF3 file (50.2 MB),
>     reads its whole array time x lat x lon = 1460 x 94 x 192
> 
>     Open file: 838 ms
>     Find var: 12 ms
>     Read array: 2393 ms
>     (code is below)
> 
>     Numbers are for SSD with approx 222 MB/sec speed
> 
>     This is almost 3 sec while during the same time C++ code (NCO, CDO)
>     may do a lot more.
> 
>     My questions are:
>     - could this be improved? (e.g. link to native lib or smth else)
>     - is this the fastest way to read array (code below)?
> 
>     Thanks
> 
>             long millis = System.currentTimeMillis();
> 
>             NetcdfFile FILE = NetcdfFile.open(filepath, null);
>             System.out.println("Open file: " +
>     (System.currentTimeMillis() - millis));
> 
>             millis = System.currentTimeMillis();
>             Variable var = FILE.findVariable("uwnd");
>             System.out.println("Find var: " +
>     (System.currentTimeMillis() - millis));
> 
>             int[] origin ;
>             int[] shape ;
> 
>             millis = System.currentTimeMillis();
>             shape = var.getShape();
>             System.out.println("Get shape: " +
>     (System.currentTimeMillis() - millis));
> 
>             origin = new int[shape.length];
> 
>             millis = System.currentTimeMillis();
>             Array values = var.read(origin, shape);
>             System.out.println("Read array: " +
>     (System.currentTimeMillis() - millis));
> 
> 
>     _______________________________________________
>     NOTE: All exchanges posted to Unidata maintained email lists are
>     recorded in the Unidata inquiry tracking system and made publicly
>     available through the web.  Users who post to any of the lists we
>     maintain are reminded to remove any personal information that they
>     do not want to be made public.
> 
> 
>     netcdf-java mailing list
>     netcdf-java@xxxxxxxxxxxxxxxx <mailto:netcdf-java@xxxxxxxxxxxxxxxx>
>     For list information or to unsubscribe, visit:
>     http://www.unidata.ucar.edu/mailing_lists/
>     <http://www.unidata.ucar.edu/mailing_lists/>
> 
> 
> 
> 
> -- 
> Ryan May, Ph.D.
> Software Engineer
> UCAR/Unidata
> Boulder, CO
> 
> 
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
> 



  • 2016 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: