John,
>
> 1) The Netcdf-Java library creates indices in the same directory as the data
> file. If that directory is not writeable, it will put it into a "disk cache"
> directory. This behavior can be modified by
> ucar.nc2.grib.GribCollection.setDiskCache2(). Are you calling that?
That's it: I do not touch
"ucar.nc2.grib.GribCollection.setDiskCache2()" nor in 4.3 neither in
4.5 while 4.3 does not create indices and 4.5 does. I call this
"backward compatibility issue" when I change the library, do not
modify my code and it stops working (or works differently) as it used
to be with previous version
> 2) The Netcdf-Java library can only (easily) create a grib collection from a
> single file. Is that what you are doing?
No, I do not create a collection, I simply read variable data from a grib2 file.
>
> 3) I will have a look at your CFSR files.
Thank you
> 4) I would advise you to try 4.6 for GRIB files. Its the only branch that
> GRIB problems are being addressed. We will do simple bug fixes on 4.5, but
> 4.6 is another rewrite, so it cant be easily backported.
I am not sure about backward comp. of 4.6. I update NetCDF java only
for bugfixes I encounter but I would be happy if there is a branch
only for bugfixes without any fuctionality updates/rewrites...
> 5) interesting idea to make a test on files at a user site, maybe make a
> report that could be sent to us. We will look at that, thanks for the
> suggestion.
You are welcome. Hope this will help someone else.
Antonio
>
> John
>
> On Tue, Mar 10, 2015 at 6:17 AM, Antonio Rodriges <antonio.rrz@xxxxxxxxx>
> wrote:
>>
>> Hello,
>>
>> 2015-03-10 2:57 GMT+03:00 John Caron <caron@xxxxxxxx>:
>> > Hi Antonio:
>> >
>> > Some answers embedded below:
>> >
>> > On Mon, Mar 9, 2015 at 1:43 PM, Antonio Rodriges <antonio.rrz@xxxxxxxxx>
>> > wrote:
>> >>
>> >> Hello,
>> >>
>> >> Thank you for the 4.5 release.
>> >>
>> >> However, there are troubles migrating from 4.3 to 4.5 (I am trying to
>> >> migrate from 4.3 to a higher version for quite long now).
>> >>
>> >> I have used a pre-release snapshot kindly provided at the 17th of
>> >> January but I suppose the following issues may stlll hold for the
>> >> final release. They are mainly concerned with IOSP grib2 backward
>> >> compatibility.
>> >>
>> >> 1) Creation of index files
>> >> version 4.3 does not create in the same directory where original grib2
>> >> files are located indexes for each date I access. But 4.5 does so for
>> >> each date:
>> >> 2015-02-18/06:09:50.118/GMT-00:00 [ucar.nc2.grib.collection.Grib2Iosp]
>> >> INFO GribCollectionBuilder write
>> >>
>> >>
>> >> D:/RS_DATA/worker/symlinks/CFSR/windspeed/wnd10m.gdas.200508.grb2-20050818-180000.ncx2
>> >> ok=true
>> >> 2015-02-18/06:09:50.120/GMT-00:00 [ucar.nc2.grib.collection.Grib2Iosp]
>> >> DEBUG createIndex for
>> >>
>> >>
>> >> D:\RS_DATA\worker\symlinks\CFSR\windspeed\wnd10m.gdas.200508.grb2-20050815-000000.ncx2
>> >> 2015-02-18/06:09:50.120/GMT-00:00 [ucar.nc2.grib.collection.Grib2Iosp]
>> >> DEBUG write RecordMaps: bytes = 218 record = 14 bytesPerRecord=15
>> >> 2015-02-18/06:09:50.121/GMT-00:00 [ucar.nc2.grib.collection.Grib2Iosp]
>> >> DEBUG write GribCollectionIndex= 551 bytes
>> >>
>> >> At the end of traversing the dates from a given interval one arrives
>> >> at a point where the directory has files that are not anticipated to
>> >> emerge (for one previously used 4.3).
>> >>
>> >> Even if index creation time is negligible and it significantly
>> >> accelerates further operation and etc. etc. it is quite logical to
>> >> keep this feature off since it was off or not present in 4.3.
>> >
>> >
>> > If you want to put index files in a sperate directory from the data
>> > files,
>> > use following in threddsConfig.xml :
>> >
>> > <GribIndex>
>> > <alwaysUse>true</alwaysUse>
>> > <dir>/tomcat_home/content/thredds/cache/grib/</dir>
>> > </GribIndex>
>> >
>>
>> I do not use THREDDS, I use NetCDF-Java library
>>
>> > see
>> >
>> >
>> > http://www.unidata.ucar.edu/software/thredds/v4.5/tds/reference/ThreddsConfigXMLFile.html#GribIndexWriting
>> >
>> > I dont think this has changed since 4.3 (?)
>>
>> And I have 4.3 and tried 4.5 so I can't update
>>
>> >
>> >>
>> >> You may argue that it is not so critical but it depends. The following
>> >> issue is more severe.
>> >>
>> >> 2) Consider grib2 files of CFSR reanalysis, for example:
>> >> http://nomads.ncdc.noaa.gov/data/cfsr/198209/wnd10m.gdas.198209.grb2
>> >>
>> >> If I would like to read "u-component_of_wind_height_above_ground" in
>> >> version 4.3 it is OK since it finds only a single variable with this
>> >> name:
>> >>
>> >> float u-component_of_wind_height_above_ground(time=745,
>> >> height_above_ground=1, lat=576, lon=1152);
>> >> :long_name = "u-component of wind @ Specified height level above
>> >> ground";
>> >> :units = "m/s";
>> >> :missing_value = NaNf; // float
>> >> :abbreviation = "UGRD";
>> >> :Grib_Variable_Id = "VAR_0-2-2_L103";
>> >> :Grib2_Parameter = 0, 2, 2; // int
>> >> :Grib2_Parameter_Discipline = "Meteorological products";
>> >> :Grib2_Parameter_Category = "Momentum";
>> >> :Grib2_Parameter_Name = "u-component of wind";
>> >> :Grib2_Level_Type = 103; // int
>> >> :Grib2_Generating_Process_Type = "Forecast";
>> >>
>> >> However, if I switch to 4.5 the program crashes since 4.5 finds 2
>> >> variables of the same name and findVariable returns null
>> >>
>> >> float u-component_of_wind_height_above_ground(time=745,
>> >> height_above_ground=1, lat=576, lon=1152);
>> >> :long_name = "u-component of wind @ Specified height level above
>> >> ground";
>> >> :units = "m/s";
>> >> :missing_value = NaNf; // float
>> >> :abbreviation = "UGRD";
>> >> :coordinates = "time height_above_ground ";
>> >> :Grib_Variable_Id = "VAR_0-2-2_L103";
>> >> :Grib2_Parameter = 0, 2, 2; // int
>> >> :Grib2_Parameter_Discipline = "Meteorological products";
>> >> :Grib2_Parameter_Category = "Momentum";
>> >> :Grib2_Parameter_Name = "u-component of wind";
>> >> :Grib2_Level_Type = "Specified height level above ground";
>> >> :Grib2_Generating_Process_Type = "Forecast";
>> >
>> >
>> >>
>> >> float u-component_of_wind_height_above_ground(reftime=124, time=7,
>> >> height_above_ground=1, lat=576, lon=1152);
>> >> :long_name = "u-component of wind @ Specified height level above
>> >> ground";
>> >> :units = "m/s";
>> >> :missing_value = NaNf; // float
>> >> :abbreviation = "UGRD";
>> >> :coordinates = "reftime time height_above_ground ";
>> >> :Grib_Variable_Id = "VAR_0-2-2_L103";
>> >> :Grib2_Parameter = 0, 2, 2; // int
>> >> :Grib2_Parameter_Discipline = "Meteorological products";
>> >> :Grib2_Parameter_Category = "Momentum";
>> >> :Grib2_Parameter_Name = "u-component of wind";
>> >> :Grib2_Level_Type = "Specified height level above ground";
>> >> :Grib2_Generating_Process_Type = "Forecast";
>> >>
>> >> Again, it is all about the backward compatibility.
>> >
>> >
>> > CFSR encoding is notoriously messed up. Its likely there is some glitch
>> > like
>> > table version that differs between two records. If you can send me a
>> > minimum
>> > set of files that reproduces the problem, i will see if theres a
>> > workaround.
>> >
>> > Also, you could consider moving to version 4.6, which will once again
>> > write
>> > new CDM index files for GRIB. The advantage is that it scales to large
>> > collections. The disadvantage is that it is currently alpha. But if you
>> > are
>> > interested, I can point you to the current snapshot.
>>
>> I have included the link in my previous e-mail, here it is again:
>> http://nomads.ncdc.noaa.gov/data/cfsr/198209/wnd10m.gdas.198209.grb2
>> and you can find there a lot of other files for other dates
>>
>>
>> >
>> >>
>> >>
>> >> 3) Would it be helpful to maintain a collective repository of files
>> >> from diverse domains and different formats that are read by
>> >> NetCDF-Java users and test all subsequent releases against that
>> >> repository?
>> >>
>> >> It could reveal differences/changes and document them at the early
>> >> stage and avoid things like described (also recall HDF issues like
>> >>
>> >> http://www.unidata.ucar.edu/support/help/MailArchives/netcdf/msg12946.html
>> >> )
>> >
>> >
>> > yes, that would be great. we have a set of files that we automatically
>> > read
>> > in our unit tests, which would have caught this problem. It can be hard
>> > to
>> > whittle the files down to a manageable size (a few Mbytes would be
>> > nice).
>>
>>
>> Maybe there should be tests embedded in the library itself so everyone
>> could run against their own collection and report the problems?
>
>
