Hi Antonio:
Unfortunately, GRIB is a difficult file format to get right. It has not
been possible to issue bug fixes, rather we have had to rewrite the
library, so far 3 times now. So I consider any versions before 4.6 as
broken, and unfixable. While earlier versions do work sometimes, they also
often appear to work correctly, but are wrong.
I am hopeful that 4.6 is the last rewrite and from now on we can issue
backwards-compatible bug fixes going forward. Meanwhile we have to go
through the painful process of upgrading and reporting the problems.
4.3 does create indexes, they would by default be in the directory
"~user/.unidata/cache" (in the home directory of the user running the
application). You might want to have a look in there. Its possible that I
didnt get that same directory correctly wired up in 4.6, i will check on
that.
John
On Tue, Mar 10, 2015 at 10:31 AM, Antonio Rodriges <antonio.rrz@xxxxxxxxx>
wrote:
> John,
> >
> > 1) The Netcdf-Java library creates indices in the same directory as the
> data
> > file. If that directory is not writeable, it will put it into a "disk
> cache"
> > directory. This behavior can be modified by
> > ucar.nc2.grib.GribCollection.setDiskCache2(). Are you calling that?
>
> That's it: I do not touch
> "ucar.nc2.grib.GribCollection.setDiskCache2()" nor in 4.3 neither in
> 4.5 while 4.3 does not create indices and 4.5 does. I call this
> "backward compatibility issue" when I change the library, do not
> modify my code and it stops working (or works differently) as it used
> to be with previous version
>
> > 2) The Netcdf-Java library can only (easily) create a grib collection
> from a
> > single file. Is that what you are doing?
>
> No, I do not create a collection, I simply read variable data from a grib2
> file.
>
> >
> > 3) I will have a look at your CFSR files.
>
> Thank you
>
> > 4) I would advise you to try 4.6 for GRIB files. Its the only branch that
> > GRIB problems are being addressed. We will do simple bug fixes on 4.5,
> but
> > 4.6 is another rewrite, so it cant be easily backported.
>
> I am not sure about backward comp. of 4.6. I update NetCDF java only
> for bugfixes I encounter but I would be happy if there is a branch
> only for bugfixes without any fuctionality updates/rewrites...
>
> > 5) interesting idea to make a test on files at a user site, maybe make a
> > report that could be sent to us. We will look at that, thanks for the
> > suggestion.
>
> You are welcome. Hope this will help someone else.
>
> Antonio
>
> >
> > John
> >
> > On Tue, Mar 10, 2015 at 6:17 AM, Antonio Rodriges <antonio.rrz@xxxxxxxxx
> >
> > wrote:
> >>
> >> Hello,
> >>
> >> 2015-03-10 2:57 GMT+03:00 John Caron <caron@xxxxxxxx>:
> >> > Hi Antonio:
> >> >
> >> > Some answers embedded below:
> >> >
> >> > On Mon, Mar 9, 2015 at 1:43 PM, Antonio Rodriges <
> antonio.rrz@xxxxxxxxx>
> >> > wrote:
> >> >>
> >> >> Hello,
> >> >>
> >> >> Thank you for the 4.5 release.
> >> >>
> >> >> However, there are troubles migrating from 4.3 to 4.5 (I am trying to
> >> >> migrate from 4.3 to a higher version for quite long now).
> >> >>
> >> >> I have used a pre-release snapshot kindly provided at the 17th of
> >> >> January but I suppose the following issues may stlll hold for the
> >> >> final release. They are mainly concerned with IOSP grib2 backward
> >> >> compatibility.
> >> >>
> >> >> 1) Creation of index files
> >> >> version 4.3 does not create in the same directory where original
> grib2
> >> >> files are located indexes for each date I access. But 4.5 does so for
> >> >> each date:
> >> >> 2015-02-18/06:09:50.118/GMT-00:00
> [ucar.nc2.grib.collection.Grib2Iosp]
> >> >> INFO GribCollectionBuilder write
> >> >>
> >> >>
> >> >>
> D:/RS_DATA/worker/symlinks/CFSR/windspeed/wnd10m.gdas.200508.grb2-20050818-180000.ncx2
> >> >> ok=true
> >> >> 2015-02-18/06:09:50.120/GMT-00:00
> [ucar.nc2.grib.collection.Grib2Iosp]
> >> >> DEBUG createIndex for
> >> >>
> >> >>
> >> >>
> D:\RS_DATA\worker\symlinks\CFSR\windspeed\wnd10m.gdas.200508.grb2-20050815-000000.ncx2
> >> >> 2015-02-18/06:09:50.120/GMT-00:00
> [ucar.nc2.grib.collection.Grib2Iosp]
> >> >> DEBUG write RecordMaps: bytes = 218 record = 14 bytesPerRecord=15
> >> >> 2015-02-18/06:09:50.121/GMT-00:00
> [ucar.nc2.grib.collection.Grib2Iosp]
> >> >> DEBUG write GribCollectionIndex= 551 bytes
> >> >>
> >> >> At the end of traversing the dates from a given interval one arrives
> >> >> at a point where the directory has files that are not anticipated to
> >> >> emerge (for one previously used 4.3).
> >> >>
> >> >> Even if index creation time is negligible and it significantly
> >> >> accelerates further operation and etc. etc. it is quite logical to
> >> >> keep this feature off since it was off or not present in 4.3.
> >> >
> >> >
> >> > If you want to put index files in a sperate directory from the data
> >> > files,
> >> > use following in threddsConfig.xml :
> >> >
> >> > <GribIndex>
> >> > <alwaysUse>true</alwaysUse>
> >> > <dir>/tomcat_home/content/thredds/cache/grib/</dir>
> >> > </GribIndex>
> >> >
> >>
> >> I do not use THREDDS, I use NetCDF-Java library
> >>
> >> > see
> >> >
> >> >
> >> >
> http://www.unidata.ucar.edu/software/thredds/v4.5/tds/reference/ThreddsConfigXMLFile.html#GribIndexWriting
> >> >
> >> > I dont think this has changed since 4.3 (?)
> >>
> >> And I have 4.3 and tried 4.5 so I can't update
> >>
> >> >
> >> >>
> >> >> You may argue that it is not so critical but it depends. The
> following
> >> >> issue is more severe.
> >> >>
> >> >> 2) Consider grib2 files of CFSR reanalysis, for example:
> >> >> http://nomads.ncdc.noaa.gov/data/cfsr/198209/wnd10m.gdas.198209.grb2
> >> >>
> >> >> If I would like to read "u-component_of_wind_height_above_ground" in
> >> >> version 4.3 it is OK since it finds only a single variable with this
> >> >> name:
> >> >>
> >> >> float u-component_of_wind_height_above_ground(time=745,
> >> >> height_above_ground=1, lat=576, lon=1152);
> >> >> :long_name = "u-component of wind @ Specified height level above
> >> >> ground";
> >> >> :units = "m/s";
> >> >> :missing_value = NaNf; // float
> >> >> :abbreviation = "UGRD";
> >> >> :Grib_Variable_Id = "VAR_0-2-2_L103";
> >> >> :Grib2_Parameter = 0, 2, 2; // int
> >> >> :Grib2_Parameter_Discipline = "Meteorological products";
> >> >> :Grib2_Parameter_Category = "Momentum";
> >> >> :Grib2_Parameter_Name = "u-component of wind";
> >> >> :Grib2_Level_Type = 103; // int
> >> >> :Grib2_Generating_Process_Type = "Forecast";
> >> >>
> >> >> However, if I switch to 4.5 the program crashes since 4.5 finds 2
> >> >> variables of the same name and findVariable returns null
> >> >>
> >> >> float u-component_of_wind_height_above_ground(time=745,
> >> >> height_above_ground=1, lat=576, lon=1152);
> >> >> :long_name = "u-component of wind @ Specified height level above
> >> >> ground";
> >> >> :units = "m/s";
> >> >> :missing_value = NaNf; // float
> >> >> :abbreviation = "UGRD";
> >> >> :coordinates = "time height_above_ground ";
> >> >> :Grib_Variable_Id = "VAR_0-2-2_L103";
> >> >> :Grib2_Parameter = 0, 2, 2; // int
> >> >> :Grib2_Parameter_Discipline = "Meteorological products";
> >> >> :Grib2_Parameter_Category = "Momentum";
> >> >> :Grib2_Parameter_Name = "u-component of wind";
> >> >> :Grib2_Level_Type = "Specified height level above ground";
> >> >> :Grib2_Generating_Process_Type = "Forecast";
> >> >
> >> >
> >> >>
> >> >> float u-component_of_wind_height_above_ground(reftime=124, time=7,
> >> >> height_above_ground=1, lat=576, lon=1152);
> >> >> :long_name = "u-component of wind @ Specified height level above
> >> >> ground";
> >> >> :units = "m/s";
> >> >> :missing_value = NaNf; // float
> >> >> :abbreviation = "UGRD";
> >> >> :coordinates = "reftime time height_above_ground ";
> >> >> :Grib_Variable_Id = "VAR_0-2-2_L103";
> >> >> :Grib2_Parameter = 0, 2, 2; // int
> >> >> :Grib2_Parameter_Discipline = "Meteorological products";
> >> >> :Grib2_Parameter_Category = "Momentum";
> >> >> :Grib2_Parameter_Name = "u-component of wind";
> >> >> :Grib2_Level_Type = "Specified height level above ground";
> >> >> :Grib2_Generating_Process_Type = "Forecast";
> >> >>
> >> >> Again, it is all about the backward compatibility.
> >> >
> >> >
> >> > CFSR encoding is notoriously messed up. Its likely there is some
> glitch
> >> > like
> >> > table version that differs between two records. If you can send me a
> >> > minimum
> >> > set of files that reproduces the problem, i will see if theres a
> >> > workaround.
> >> >
> >> > Also, you could consider moving to version 4.6, which will once again
> >> > write
> >> > new CDM index files for GRIB. The advantage is that it scales to large
> >> > collections. The disadvantage is that it is currently alpha. But if
> you
> >> > are
> >> > interested, I can point you to the current snapshot.
> >>
> >> I have included the link in my previous e-mail, here it is again:
> >> http://nomads.ncdc.noaa.gov/data/cfsr/198209/wnd10m.gdas.198209.grb2
> >> and you can find there a lot of other files for other dates
> >>
> >>
> >> >
> >> >>
> >> >>
> >> >> 3) Would it be helpful to maintain a collective repository of files
> >> >> from diverse domains and different formats that are read by
> >> >> NetCDF-Java users and test all subsequent releases against that
> >> >> repository?
> >> >>
> >> >> It could reveal differences/changes and document them at the early
> >> >> stage and avoid things like described (also recall HDF issues like
> >> >>
> >> >>
> http://www.unidata.ucar.edu/support/help/MailArchives/netcdf/msg12946.html
> >> >> )
> >> >
> >> >
> >> > yes, that would be great. we have a set of files that we automatically
> >> > read
> >> > in our unit tests, which would have caught this problem. It can be
> hard
> >> > to
> >> > whittle the files down to a manageable size (a few Mbytes would be
> >> > nice).
> >>
> >>
> >> Maybe there should be tests embedded in the library itself so everyone
> >> could run against their own collection and report the problems?
> >
> >
>
