[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDFJava #WHR-919836]: Problem reading grib2 files in memory

Hi Adrian:

Yes, Im thinking it may work when opening an individual grib file, but not for 
a collection, which would require special processing.

Can you say why you are trying to use these files from memory? Generally we 
assume that datasets will not fit into memory.

John

> Hi,
> 
> Thanks very much for the quick response.
> 
> The fixed worked in as far as I can open Grib1 and 2 files in memory - but 
> the class ucar.nc2.grib.GribCollection creates a new RandomAccessFile when 
> the method getDataRaf is called.  See the included stack trace (please ignore 
> the version name that's just a quick renamed to get the artifacts into nexus).
> 
> at 
> ucar.nc2.grib.GribCollection.getDataRaf(GribCollection.java:345)[grib-4.3.16-gribinmemoryfix.jar:4.3.16-gribinmemoryfix]
> at 
> ucar.nc2.grib.grib2.Grib2Iosp$DataReader.read(Grib2Iosp.java:974)[grib-4.3.16-gribinmemoryfix.jar:4.3.16-gribinmemoryfix]
> at 
> ucar.nc2.grib.grib2.Grib2Iosp.readDataFromCollection(Grib2Iosp.java:913)[grib-4.3.16-gribinmemoryfix.jar:4.3.16-gribinmemoryfix]
> at 
> ucar.nc2.grib.grib2.Grib2Iosp.readData(Grib2Iosp.java:723)[grib-4.3.16-gribinmemoryfix.jar:4.3.16-gribinmemoryfix]
> at 
> ucar.nc2.NetcdfFile.readData(NetcdfFile.java:1930)[netcdf-4.3.16-gribinmemoryfix.jar:4.3.16-gribinmemoryfix]
> at 
> ucar.nc2.Variable.reallyRead(Variable.java:894)[netcdf-4.3.16-gribinmemoryfix.jar:4.3.16-gribinmemoryfix]
> at 
> ucar.nc2.Variable._read(Variable.java:880)[netcdf-4.3.16-gribinmemoryfix.jar:4.3.16-gribinmemoryfix]
> at 
> ucar.nc2.Variable.read(Variable.java:692)[netcdf-4.3.16-gribinmemoryfix.jar:4.3.16-gribinmemoryfix]
> at 
> ucar.nc2.dataset.VariableDS.reallyRead(VariableDS.java:553)[netcdf-4.3.16-gribinmemoryfix.jar:4.3.16-gribinmemoryfix]
> at 
> ucar.nc2.dataset.VariableDS._read(VariableDS.java:533)[netcdf-4.3.16-gribinmemoryfix.jar:4.3.16-gribinmemoryfix]
> at 
> ucar.nc2.Variable.read(Variable.java:692)[netcdf-4.3.16-gribinmemoryfix.jar:4.3.16-gribinmemoryfix]
> at 
> ucar.nc2.Variable.read(Variable.java:638)[netcdf-4.3.16-gribinmemoryfix.jar:4.3.16-gribinmemoryfix]
> at 
> ucar.nc2.dt.grid.GeoGrid.readDataSlice(GeoGrid.java:607)[netcdf-4.3.16-gribinmemoryfix.jar:4.3.16-gribinmemoryfix]
> at 
> ucar.nc2.dt.grid.GeoGrid.readDataSlice(GeoGrid.java:518)[netcdf-4.3.16-gribinmemoryfix.jar:4.3.16-gribinmemoryfix]
> 
> Cheers
> 
> Adrian
> 
> -----Original Message-----
> From: Unidata netCDF Java Support [mailto:address@hidden]
> Sent: Saturday, 16 March 2013 10:49 a.m.
> To: Adrian Jarvis
> Cc: address@hidden
> Subject: [netCDFJava #WHR-919836]: Problem reading grib2 files in memory
> 
> Hi Adrian,
> 
> Thank you for bringing this issue to our attention.  We believe we have 
> implemented a solution in our latest release (4.3.16), though it is not 
> tested, and we would appreciate your feedback on it.  Note that, due to the 
> nature of grib files and their associated tables (as opposed to a 
> self-contained dataset such as NetCDF), the netcdf-java code will still need 
> to maintain ncx and gbx9 files on disk.
> 
> Cheers,
> Lansing Madry
> Unidata
> Boulder, Colorado
> 
> > Hi,
> >
> > I have run into a bug while trying to read a grib2 files from a byte
> > array.
> >
> > I am trying to use the static method
> > NetcdfFile openInMemory(String name, byte[] data)
> >
> > But the Grib2Iosp clas is calling the makeIndex method in the
> > Grib2Index cl= ass which is using the location to try and open a
> > RandomAccessFile using th= e filename.  It seems the Grib2Index class
> > is written based on files on dis= k and does not support data in memory.
> >
> > This all fails in a heap since there isn't a file on disk.
> >
> > Here is the stack trace I'm getting:
> > Exception in thread "main" java.io.FileNotFoundException: test-file
> > (The system cannot find the file specified) at
> > java.io.RandomAccessFile.open(Native Method) at
> > java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:118)
> > at ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:277)
> > at ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
> > at ucar.nc2.grib.grib2.Grib2Index.makeIndex(Grib2Index.java:234)
> > at
> > ucar.nc2.grib.GribIndex.makeGribCollectionFromSingleFile(GribIndex.jav
> > a:112) at ucar.nc2.grib.grib2.Grib2Iosp.open(Grib2Iosp.java:310)
> > at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:1499)
> > at ucar.nc2.NetcdfFile.open(NetcdfFile.java:887)
> > at ucar.nc2.NetcdfFile.openInMemory(NetcdfFile.java:793)
> > at griberror.TestGribReading.main(TestGribReading.java:22)
> >
> > Cheers
> >
> > Adrian Jarvis
> > Senior Java Developer,
> > MetService New Zealand Ltd
> >
> >
> >
> 
> 
> Ticket Details
> ===================
> Ticket ID: WHR-919836
> Department: Support netCDF Java
> Priority: High
> Status: Open
> 
> 


Ticket Details
===================
Ticket ID: WHR-919836
Department: Support netCDF Java
Priority: High
Status: Open