John,
The "~user/.unidata/cache" is empty on my PC.
Instead I have
wnd10m.gdas.200508.grb2.ncx
wnd10m.gdas.200508.grb2.gbx8
wnd10m.gdas.200508.grb2.gbx9
created by 4.3 for the whole file
and
wnd10m.gdas.200508.grb2-20050801-000000.ncx2
........
wnd10m.gdas.200508.grb2-20050831-120000.ncx2
wnd10m.gdas.200508.grb2-20050831-180000.ncx2
created by 4.5 for each time step in that file
So, indexing format has changed from 4.3 to 4.5?
2015-03-10 19:58 GMT+03:00 John Caron <caron@xxxxxxxx>:
> Hi Antonio:
>
> Unfortunately, GRIB is a difficult file format to get right. It has not been
> possible to issue bug fixes, rather we have had to rewrite the library, so
> far 3 times now. So I consider any versions before 4.6 as broken, and
> unfixable. While earlier versions do work sometimes, they also often appear
> to work correctly, but are wrong.
>
> I am hopeful that 4.6 is the last rewrite and from now on we can issue
> backwards-compatible bug fixes going forward. Meanwhile we have to go
> through the painful process of upgrading and reporting the problems.
>
> 4.3 does create indexes, they would by default be in the directory
> "~user/.unidata/cache" (in the home directory of the user running the
> application). You might want to have a look in there. Its possible that I
> didnt get that same directory correctly wired up in 4.6, i will check on
> that.
>
> John
>
> On Tue, Mar 10, 2015 at 10:31 AM, Antonio Rodriges <antonio.rrz@xxxxxxxxx>
> wrote:
>>
>> John,
>> >
>> > 1) The Netcdf-Java library creates indices in the same directory as the
>> > data
>> > file. If that directory is not writeable, it will put it into a "disk
>> > cache"
>> > directory. This behavior can be modified by
>> > ucar.nc2.grib.GribCollection.setDiskCache2(). Are you calling that?
>>
>> That's it: I do not touch
>> "ucar.nc2.grib.GribCollection.setDiskCache2()" nor in 4.3 neither in
>> 4.5 while 4.3 does not create indices and 4.5 does. I call this
>> "backward compatibility issue" when I change the library, do not
>> modify my code and it stops working (or works differently) as it used
>> to be with previous version
>>
>> > 2) The Netcdf-Java library can only (easily) create a grib collection
>> > from a
>> > single file. Is that what you are doing?
>>
>> No, I do not create a collection, I simply read variable data from a grib2
>> file.
>>
>> >
>> > 3) I will have a look at your CFSR files.
>>
>> Thank you
>>
>> > 4) I would advise you to try 4.6 for GRIB files. Its the only branch
>> > that
>> > GRIB problems are being addressed. We will do simple bug fixes on 4.5,
>> > but
>> > 4.6 is another rewrite, so it cant be easily backported.
>>
>> I am not sure about backward comp. of 4.6. I update NetCDF java only
>> for bugfixes I encounter but I would be happy if there is a branch
>> only for bugfixes without any fuctionality updates/rewrites...
>>
>> > 5) interesting idea to make a test on files at a user site, maybe make a
>> > report that could be sent to us. We will look at that, thanks for the
>> > suggestion.
>>
>> You are welcome. Hope this will help someone else.
>>
>> Antonio
>>
>> >
>> > John
>> >
>> > On Tue, Mar 10, 2015 at 6:17 AM, Antonio Rodriges
>> > <antonio.rrz@xxxxxxxxx>
>> > wrote:
>> >>
>> >> Hello,
>> >>
>> >> 2015-03-10 2:57 GMT+03:00 John Caron <caron@xxxxxxxx>:
>> >> > Hi Antonio:
>> >> >
>> >> > Some answers embedded below:
>> >> >
>> >> > On Mon, Mar 9, 2015 at 1:43 PM, Antonio Rodriges
>> >> > <antonio.rrz@xxxxxxxxx>
>> >> > wrote:
>> >> >>
>> >> >> Hello,
>> >> >>
>> >> >> Thank you for the 4.5 release.
>> >> >>
>> >> >> However, there are troubles migrating from 4.3 to 4.5 (I am trying
>> >> >> to
>> >> >> migrate from 4.3 to a higher version for quite long now).
>> >> >>
>> >> >> I have used a pre-release snapshot kindly provided at the 17th of
>> >> >> January but I suppose the following issues may stlll hold for the
>> >> >> final release. They are mainly concerned with IOSP grib2 backward
>> >> >> compatibility.
>> >> >>
>> >> >> 1) Creation of index files
>> >> >> version 4.3 does not create in the same directory where original
>> >> >> grib2
>> >> >> files are located indexes for each date I access. But 4.5 does so
>> >> >> for
>> >> >> each date:
>> >> >> 2015-02-18/06:09:50.118/GMT-00:00
>> >> >> [ucar.nc2.grib.collection.Grib2Iosp]
>> >> >> INFO GribCollectionBuilder write
>> >> >>
>> >> >>
>> >> >>
>> >> >> D:/RS_DATA/worker/symlinks/CFSR/windspeed/wnd10m.gdas.200508.grb2-20050818-180000.ncx2
>> >> >> ok=true
>> >> >> 2015-02-18/06:09:50.120/GMT-00:00
>> >> >> [ucar.nc2.grib.collection.Grib2Iosp]
>> >> >> DEBUG createIndex for
>> >> >>
>> >> >>
>> >> >>
>> >> >> D:\RS_DATA\worker\symlinks\CFSR\windspeed\wnd10m.gdas.200508.grb2-20050815-000000.ncx2
>> >> >> 2015-02-18/06:09:50.120/GMT-00:00
>> >> >> [ucar.nc2.grib.collection.Grib2Iosp]
>> >> >> DEBUG write RecordMaps: bytes = 218 record = 14 bytesPerRecord=15
>> >> >> 2015-02-18/06:09:50.121/GMT-00:00
>> >> >> [ucar.nc2.grib.collection.Grib2Iosp]
>> >> >> DEBUG write GribCollectionIndex= 551 bytes
>> >> >>
>> >> >> At the end of traversing the dates from a given interval one arrives
>> >> >> at a point where the directory has files that are not anticipated to
>> >> >> emerge (for one previously used 4.3).
>> >> >>
>> >> >> Even if index creation time is negligible and it significantly
>> >> >> accelerates further operation and etc. etc. it is quite logical to
>> >> >> keep this feature off since it was off or not present in 4.3.
>> >> >
>> >> >
>> >> > If you want to put index files in a sperate directory from the data
>> >> > files,
>> >> > use following in threddsConfig.xml :
>> >> >
>> >> > <GribIndex>
>> >> > <alwaysUse>true</alwaysUse>
>> >> > <dir>/tomcat_home/content/thredds/cache/grib/</dir>
>> >> > </GribIndex>
>> >> >
>> >>
>> >> I do not use THREDDS, I use NetCDF-Java library
>> >>
>> >> > see
>> >> >
>> >> >
>> >> >
>> >> > http://www.unidata.ucar.edu/software/thredds/v4.5/tds/reference/ThreddsConfigXMLFile.html#GribIndexWriting
>> >> >
>> >> > I dont think this has changed since 4.3 (?)
>> >>
>> >> And I have 4.3 and tried 4.5 so I can't update
>> >>
>> >> >
>> >> >>
>> >> >> You may argue that it is not so critical but it depends. The
>> >> >> following
>> >> >> issue is more severe.
>> >> >>
>> >> >> 2) Consider grib2 files of CFSR reanalysis, for example:
>> >> >> http://nomads.ncdc.noaa.gov/data/cfsr/198209/wnd10m.gdas.198209.grb2
>> >> >>
>> >> >> If I would like to read "u-component_of_wind_height_above_ground" in
>> >> >> version 4.3 it is OK since it finds only a single variable with this
>> >> >> name:
>> >> >>
>> >> >> float u-component_of_wind_height_above_ground(time=745,
>> >> >> height_above_ground=1, lat=576, lon=1152);
>> >> >> :long_name = "u-component of wind @ Specified height level above
>> >> >> ground";
>> >> >> :units = "m/s";
>> >> >> :missing_value = NaNf; // float
>> >> >> :abbreviation = "UGRD";
>> >> >> :Grib_Variable_Id = "VAR_0-2-2_L103";
>> >> >> :Grib2_Parameter = 0, 2, 2; // int
>> >> >> :Grib2_Parameter_Discipline = "Meteorological products";
>> >> >> :Grib2_Parameter_Category = "Momentum";
>> >> >> :Grib2_Parameter_Name = "u-component of wind";
>> >> >> :Grib2_Level_Type = 103; // int
>> >> >> :Grib2_Generating_Process_Type = "Forecast";
>> >> >>
>> >> >> However, if I switch to 4.5 the program crashes since 4.5 finds 2
>> >> >> variables of the same name and findVariable returns null
>> >> >>
>> >> >> float u-component_of_wind_height_above_ground(time=745,
>> >> >> height_above_ground=1, lat=576, lon=1152);
>> >> >> :long_name = "u-component of wind @ Specified height level above
>> >> >> ground";
>> >> >> :units = "m/s";
>> >> >> :missing_value = NaNf; // float
>> >> >> :abbreviation = "UGRD";
>> >> >> :coordinates = "time height_above_ground ";
>> >> >> :Grib_Variable_Id = "VAR_0-2-2_L103";
>> >> >> :Grib2_Parameter = 0, 2, 2; // int
>> >> >> :Grib2_Parameter_Discipline = "Meteorological products";
>> >> >> :Grib2_Parameter_Category = "Momentum";
>> >> >> :Grib2_Parameter_Name = "u-component of wind";
>> >> >> :Grib2_Level_Type = "Specified height level above ground";
>> >> >> :Grib2_Generating_Process_Type = "Forecast";
>> >> >
>> >> >
>> >> >>
>> >> >> float u-component_of_wind_height_above_ground(reftime=124, time=7,
>> >> >> height_above_ground=1, lat=576, lon=1152);
>> >> >> :long_name = "u-component of wind @ Specified height level above
>> >> >> ground";
>> >> >> :units = "m/s";
>> >> >> :missing_value = NaNf; // float
>> >> >> :abbreviation = "UGRD";
>> >> >> :coordinates = "reftime time height_above_ground ";
>> >> >> :Grib_Variable_Id = "VAR_0-2-2_L103";
>> >> >> :Grib2_Parameter = 0, 2, 2; // int
>> >> >> :Grib2_Parameter_Discipline = "Meteorological products";
>> >> >> :Grib2_Parameter_Category = "Momentum";
>> >> >> :Grib2_Parameter_Name = "u-component of wind";
>> >> >> :Grib2_Level_Type = "Specified height level above ground";
>> >> >> :Grib2_Generating_Process_Type = "Forecast";
>> >> >>
>> >> >> Again, it is all about the backward compatibility.
>> >> >
>> >> >
>> >> > CFSR encoding is notoriously messed up. Its likely there is some
>> >> > glitch
>> >> > like
>> >> > table version that differs between two records. If you can send me a
>> >> > minimum
>> >> > set of files that reproduces the problem, i will see if theres a
>> >> > workaround.
>> >> >
>> >> > Also, you could consider moving to version 4.6, which will once again
>> >> > write
>> >> > new CDM index files for GRIB. The advantage is that it scales to
>> >> > large
>> >> > collections. The disadvantage is that it is currently alpha. But if
>> >> > you
>> >> > are
>> >> > interested, I can point you to the current snapshot.
>> >>
>> >> I have included the link in my previous e-mail, here it is again:
>> >> http://nomads.ncdc.noaa.gov/data/cfsr/198209/wnd10m.gdas.198209.grb2
>> >> and you can find there a lot of other files for other dates
>> >>
>> >>
>> >> >
>> >> >>
>> >> >>
>> >> >> 3) Would it be helpful to maintain a collective repository of files
>> >> >> from diverse domains and different formats that are read by
>> >> >> NetCDF-Java users and test all subsequent releases against that
>> >> >> repository?
>> >> >>
>> >> >> It could reveal differences/changes and document them at the early
>> >> >> stage and avoid things like described (also recall HDF issues like
>> >> >>
>> >> >>
>> >> >> http://www.unidata.ucar.edu/support/help/MailArchives/netcdf/msg12946.html
>> >> >> )
>> >> >
>> >> >
>> >> > yes, that would be great. we have a set of files that we
>> >> > automatically
>> >> > read
>> >> > in our unit tests, which would have caught this problem. It can be
>> >> > hard
>> >> > to
>> >> > whittle the files down to a manageable size (a few Mbytes would be
>> >> > nice).
>> >>
>> >>
>> >> Maybe there should be tests embedded in the library itself so everyone
>> >> could run against their own collection and report the problems?
>> >
>> >
>
>
