Hi Antonio:
1) The Netcdf-Java library creates indices in the same directory as the
data file. If that directory is not writeable, it will put it into a "disk
cache" directory. This behavior can be modified
by ucar.nc2.grib.GribCollection.setDiskCache2(). Are you calling that?
2) The Netcdf-Java library can only (easily) create a grib collection from
a single file. Is that what you are doing?
3) I will have a look at your CFSR files.
4) I would advise you to try 4.6 for GRIB files. Its the only branch that
GRIB problems are being addressed. We will do simple bug fixes on 4.5, but
4.6 is another rewrite, so it cant be easily backported.
5) interesting idea to make a test on files at a user site, maybe make a
report that could be sent to us. We will look at that, thanks for the
suggestion.
John
On Tue, Mar 10, 2015 at 6:17 AM, Antonio Rodriges <antonio.rrz@xxxxxxxxx>
wrote:
> Hello,
>
> 2015-03-10 2:57 GMT+03:00 John Caron <caron@xxxxxxxx>:
> > Hi Antonio:
> >
> > Some answers embedded below:
> >
> > On Mon, Mar 9, 2015 at 1:43 PM, Antonio Rodriges <antonio.rrz@xxxxxxxxx>
> > wrote:
> >>
> >> Hello,
> >>
> >> Thank you for the 4.5 release.
> >>
> >> However, there are troubles migrating from 4.3 to 4.5 (I am trying to
> >> migrate from 4.3 to a higher version for quite long now).
> >>
> >> I have used a pre-release snapshot kindly provided at the 17th of
> >> January but I suppose the following issues may stlll hold for the
> >> final release. They are mainly concerned with IOSP grib2 backward
> >> compatibility.
> >>
> >> 1) Creation of index files
> >> version 4.3 does not create in the same directory where original grib2
> >> files are located indexes for each date I access. But 4.5 does so for
> >> each date:
> >> 2015-02-18/06:09:50.118/GMT-00:00 [ucar.nc2.grib.collection.Grib2Iosp]
> >> INFO GribCollectionBuilder write
> >>
> >>
> D:/RS_DATA/worker/symlinks/CFSR/windspeed/wnd10m.gdas.200508.grb2-20050818-180000.ncx2
> >> ok=true
> >> 2015-02-18/06:09:50.120/GMT-00:00 [ucar.nc2.grib.collection.Grib2Iosp]
> >> DEBUG createIndex for
> >>
> >>
> D:\RS_DATA\worker\symlinks\CFSR\windspeed\wnd10m.gdas.200508.grb2-20050815-000000.ncx2
> >> 2015-02-18/06:09:50.120/GMT-00:00 [ucar.nc2.grib.collection.Grib2Iosp]
> >> DEBUG write RecordMaps: bytes = 218 record = 14 bytesPerRecord=15
> >> 2015-02-18/06:09:50.121/GMT-00:00 [ucar.nc2.grib.collection.Grib2Iosp]
> >> DEBUG write GribCollectionIndex= 551 bytes
> >>
> >> At the end of traversing the dates from a given interval one arrives
> >> at a point where the directory has files that are not anticipated to
> >> emerge (for one previously used 4.3).
> >>
> >> Even if index creation time is negligible and it significantly
> >> accelerates further operation and etc. etc. it is quite logical to
> >> keep this feature off since it was off or not present in 4.3.
> >
> >
> > If you want to put index files in a sperate directory from the data
> files,
> > use following in threddsConfig.xml :
> >
> > <GribIndex>
> > <alwaysUse>true</alwaysUse>
> > <dir>/tomcat_home/content/thredds/cache/grib/</dir>
> > </GribIndex>
> >
>
> I do not use THREDDS, I use NetCDF-Java library
>
> > see
> >
> >
> http://www.unidata.ucar.edu/software/thredds/v4.5/tds/reference/ThreddsConfigXMLFile.html#GribIndexWriting
> >
> > I dont think this has changed since 4.3 (?)
>
> And I have 4.3 and tried 4.5 so I can't update
>
> >
> >>
> >> You may argue that it is not so critical but it depends. The following
> >> issue is more severe.
> >>
> >> 2) Consider grib2 files of CFSR reanalysis, for example:
> >> http://nomads.ncdc.noaa.gov/data/cfsr/198209/wnd10m.gdas.198209.grb2
> >>
> >> If I would like to read "u-component_of_wind_height_above_ground" in
> >> version 4.3 it is OK since it finds only a single variable with this
> >> name:
> >>
> >> float u-component_of_wind_height_above_ground(time=745,
> >> height_above_ground=1, lat=576, lon=1152);
> >> :long_name = "u-component of wind @ Specified height level above
> >> ground";
> >> :units = "m/s";
> >> :missing_value = NaNf; // float
> >> :abbreviation = "UGRD";
> >> :Grib_Variable_Id = "VAR_0-2-2_L103";
> >> :Grib2_Parameter = 0, 2, 2; // int
> >> :Grib2_Parameter_Discipline = "Meteorological products";
> >> :Grib2_Parameter_Category = "Momentum";
> >> :Grib2_Parameter_Name = "u-component of wind";
> >> :Grib2_Level_Type = 103; // int
> >> :Grib2_Generating_Process_Type = "Forecast";
> >>
> >> However, if I switch to 4.5 the program crashes since 4.5 finds 2
> >> variables of the same name and findVariable returns null
> >>
> >> float u-component_of_wind_height_above_ground(time=745,
> >> height_above_ground=1, lat=576, lon=1152);
> >> :long_name = "u-component of wind @ Specified height level above
> >> ground";
> >> :units = "m/s";
> >> :missing_value = NaNf; // float
> >> :abbreviation = "UGRD";
> >> :coordinates = "time height_above_ground ";
> >> :Grib_Variable_Id = "VAR_0-2-2_L103";
> >> :Grib2_Parameter = 0, 2, 2; // int
> >> :Grib2_Parameter_Discipline = "Meteorological products";
> >> :Grib2_Parameter_Category = "Momentum";
> >> :Grib2_Parameter_Name = "u-component of wind";
> >> :Grib2_Level_Type = "Specified height level above ground";
> >> :Grib2_Generating_Process_Type = "Forecast";
> >
> >
> >>
> >> float u-component_of_wind_height_above_ground(reftime=124, time=7,
> >> height_above_ground=1, lat=576, lon=1152);
> >> :long_name = "u-component of wind @ Specified height level above
> >> ground";
> >> :units = "m/s";
> >> :missing_value = NaNf; // float
> >> :abbreviation = "UGRD";
> >> :coordinates = "reftime time height_above_ground ";
> >> :Grib_Variable_Id = "VAR_0-2-2_L103";
> >> :Grib2_Parameter = 0, 2, 2; // int
> >> :Grib2_Parameter_Discipline = "Meteorological products";
> >> :Grib2_Parameter_Category = "Momentum";
> >> :Grib2_Parameter_Name = "u-component of wind";
> >> :Grib2_Level_Type = "Specified height level above ground";
> >> :Grib2_Generating_Process_Type = "Forecast";
> >>
> >> Again, it is all about the backward compatibility.
> >
> >
> > CFSR encoding is notoriously messed up. Its likely there is some glitch
> like
> > table version that differs between two records. If you can send me a
> minimum
> > set of files that reproduces the problem, i will see if theres a
> workaround.
> >
> > Also, you could consider moving to version 4.6, which will once again
> write
> > new CDM index files for GRIB. The advantage is that it scales to large
> > collections. The disadvantage is that it is currently alpha. But if you
> are
> > interested, I can point you to the current snapshot.
>
> I have included the link in my previous e-mail, here it is again:
> http://nomads.ncdc.noaa.gov/data/cfsr/198209/wnd10m.gdas.198209.grb2
> and you can find there a lot of other files for other dates
>
>
> >
> >>
> >>
> >> 3) Would it be helpful to maintain a collective repository of files
> >> from diverse domains and different formats that are read by
> >> NetCDF-Java users and test all subsequent releases against that
> >> repository?
> >>
> >> It could reveal differences/changes and document them at the early
> >> stage and avoid things like described (also recall HDF issues like
> >>
> http://www.unidata.ucar.edu/support/help/MailArchives/netcdf/msg12946.html
> >> )
> >
> >
> > yes, that would be great. we have a set of files that we automatically
> read
> > in our unit tests, which would have caught this problem. It can be hard
> to
> > whittle the files down to a manageable size (a few Mbytes would be nice).
>
>
> Maybe there should be tests embedded in the library itself so everyone
> could run against their own collection and report the problems?
>
