Yes, correct. And 4.6 will write .ncx3 files, so that the different
versions can coexist.
It only uses "~user/.unidata/cache" is you tell it to, or the data
directory in not writeable.
On Tue, Mar 10, 2015 at 12:57 PM, Antonio Rodriges <antonio.rrz@xxxxxxxxx>
wrote:
> John,
>
> The "~user/.unidata/cache" is empty on my PC.
>
> Instead I have
> wnd10m.gdas.200508.grb2.ncx
> wnd10m.gdas.200508.grb2.gbx8
> wnd10m.gdas.200508.grb2.gbx9
>
> created by 4.3 for the whole file
>
> and
> wnd10m.gdas.200508.grb2-20050801-000000.ncx2
> ........
> wnd10m.gdas.200508.grb2-20050831-120000.ncx2
> wnd10m.gdas.200508.grb2-20050831-180000.ncx2
> created by 4.5 for each time step in that file
>
> So, indexing format has changed from 4.3 to 4.5?
>
> 2015-03-10 19:58 GMT+03:00 John Caron <caron@xxxxxxxx>:
> > Hi Antonio:
> >
> > Unfortunately, GRIB is a difficult file format to get right. It has not
> been
> > possible to issue bug fixes, rather we have had to rewrite the library,
> so
> > far 3 times now. So I consider any versions before 4.6 as broken, and
> > unfixable. While earlier versions do work sometimes, they also often
> appear
> > to work correctly, but are wrong.
> >
> > I am hopeful that 4.6 is the last rewrite and from now on we can issue
> > backwards-compatible bug fixes going forward. Meanwhile we have to go
> > through the painful process of upgrading and reporting the problems.
> >
> > 4.3 does create indexes, they would by default be in the directory
> > "~user/.unidata/cache" (in the home directory of the user running the
> > application). You might want to have a look in there. Its possible that I
> > didnt get that same directory correctly wired up in 4.6, i will check on
> > that.
> >
> > John
> >
> > On Tue, Mar 10, 2015 at 10:31 AM, Antonio Rodriges <
> antonio.rrz@xxxxxxxxx>
> > wrote:
> >>
> >> John,
> >> >
> >> > 1) The Netcdf-Java library creates indices in the same directory as
> the
> >> > data
> >> > file. If that directory is not writeable, it will put it into a "disk
> >> > cache"
> >> > directory. This behavior can be modified by
> >> > ucar.nc2.grib.GribCollection.setDiskCache2(). Are you calling that?
> >>
> >> That's it: I do not touch
> >> "ucar.nc2.grib.GribCollection.setDiskCache2()" nor in 4.3 neither in
> >> 4.5 while 4.3 does not create indices and 4.5 does. I call this
> >> "backward compatibility issue" when I change the library, do not
> >> modify my code and it stops working (or works differently) as it used
> >> to be with previous version
> >>
> >> > 2) The Netcdf-Java library can only (easily) create a grib collection
> >> > from a
> >> > single file. Is that what you are doing?
> >>
> >> No, I do not create a collection, I simply read variable data from a
> grib2
> >> file.
> >>
> >> >
> >> > 3) I will have a look at your CFSR files.
> >>
> >> Thank you
> >>
> >> > 4) I would advise you to try 4.6 for GRIB files. Its the only branch
> >> > that
> >> > GRIB problems are being addressed. We will do simple bug fixes on 4.5,
> >> > but
> >> > 4.6 is another rewrite, so it cant be easily backported.
> >>
> >> I am not sure about backward comp. of 4.6. I update NetCDF java only
> >> for bugfixes I encounter but I would be happy if there is a branch
> >> only for bugfixes without any fuctionality updates/rewrites...
> >>
> >> > 5) interesting idea to make a test on files at a user site, maybe
> make a
> >> > report that could be sent to us. We will look at that, thanks for the
> >> > suggestion.
> >>
> >> You are welcome. Hope this will help someone else.
> >>
> >> Antonio
> >>
> >> >
> >> > John
> >> >
> >> > On Tue, Mar 10, 2015 at 6:17 AM, Antonio Rodriges
> >> > <antonio.rrz@xxxxxxxxx>
> >> > wrote:
> >> >>
> >> >> Hello,
> >> >>
> >> >> 2015-03-10 2:57 GMT+03:00 John Caron <caron@xxxxxxxx>:
> >> >> > Hi Antonio:
> >> >> >
> >> >> > Some answers embedded below:
> >> >> >
> >> >> > On Mon, Mar 9, 2015 at 1:43 PM, Antonio Rodriges
> >> >> > <antonio.rrz@xxxxxxxxx>
> >> >> > wrote:
> >> >> >>
> >> >> >> Hello,
> >> >> >>
> >> >> >> Thank you for the 4.5 release.
> >> >> >>
> >> >> >> However, there are troubles migrating from 4.3 to 4.5 (I am trying
> >> >> >> to
> >> >> >> migrate from 4.3 to a higher version for quite long now).
> >> >> >>
> >> >> >> I have used a pre-release snapshot kindly provided at the 17th of
> >> >> >> January but I suppose the following issues may stlll hold for the
> >> >> >> final release. They are mainly concerned with IOSP grib2 backward
> >> >> >> compatibility.
> >> >> >>
> >> >> >> 1) Creation of index files
> >> >> >> version 4.3 does not create in the same directory where original
> >> >> >> grib2
> >> >> >> files are located indexes for each date I access. But 4.5 does so
> >> >> >> for
> >> >> >> each date:
> >> >> >> 2015-02-18/06:09:50.118/GMT-00:00
> >> >> >> [ucar.nc2.grib.collection.Grib2Iosp]
> >> >> >> INFO GribCollectionBuilder write
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> D:/RS_DATA/worker/symlinks/CFSR/windspeed/wnd10m.gdas.200508.grb2-20050818-180000.ncx2
> >> >> >> ok=true
> >> >> >> 2015-02-18/06:09:50.120/GMT-00:00
> >> >> >> [ucar.nc2.grib.collection.Grib2Iosp]
> >> >> >> DEBUG createIndex for
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> D:\RS_DATA\worker\symlinks\CFSR\windspeed\wnd10m.gdas.200508.grb2-20050815-000000.ncx2
> >> >> >> 2015-02-18/06:09:50.120/GMT-00:00
> >> >> >> [ucar.nc2.grib.collection.Grib2Iosp]
> >> >> >> DEBUG write RecordMaps: bytes = 218 record = 14
> bytesPerRecord=15
> >> >> >> 2015-02-18/06:09:50.121/GMT-00:00
> >> >> >> [ucar.nc2.grib.collection.Grib2Iosp]
> >> >> >> DEBUG write GribCollectionIndex= 551 bytes
> >> >> >>
> >> >> >> At the end of traversing the dates from a given interval one
> arrives
> >> >> >> at a point where the directory has files that are not anticipated
> to
> >> >> >> emerge (for one previously used 4.3).
> >> >> >>
> >> >> >> Even if index creation time is negligible and it significantly
> >> >> >> accelerates further operation and etc. etc. it is quite logical to
> >> >> >> keep this feature off since it was off or not present in 4.3.
> >> >> >
> >> >> >
> >> >> > If you want to put index files in a sperate directory from the data
> >> >> > files,
> >> >> > use following in threddsConfig.xml :
> >> >> >
> >> >> > <GribIndex>
> >> >> > <alwaysUse>true</alwaysUse>
> >> >> > <dir>/tomcat_home/content/thredds/cache/grib/</dir>
> >> >> > </GribIndex>
> >> >> >
> >> >>
> >> >> I do not use THREDDS, I use NetCDF-Java library
> >> >>
> >> >> > see
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> http://www.unidata.ucar.edu/software/thredds/v4.5/tds/reference/ThreddsConfigXMLFile.html#GribIndexWriting
> >> >> >
> >> >> > I dont think this has changed since 4.3 (?)
> >> >>
> >> >> And I have 4.3 and tried 4.5 so I can't update
> >> >>
> >> >> >
> >> >> >>
> >> >> >> You may argue that it is not so critical but it depends. The
> >> >> >> following
> >> >> >> issue is more severe.
> >> >> >>
> >> >> >> 2) Consider grib2 files of CFSR reanalysis, for example:
> >> >> >>
> http://nomads.ncdc.noaa.gov/data/cfsr/198209/wnd10m.gdas.198209.grb2
> >> >> >>
> >> >> >> If I would like to read "u-component_of_wind_height_above_ground"
> in
> >> >> >> version 4.3 it is OK since it finds only a single variable with
> this
> >> >> >> name:
> >> >> >>
> >> >> >> float u-component_of_wind_height_above_ground(time=745,
> >> >> >> height_above_ground=1, lat=576, lon=1152);
> >> >> >> :long_name = "u-component of wind @ Specified height level above
> >> >> >> ground";
> >> >> >> :units = "m/s";
> >> >> >> :missing_value = NaNf; // float
> >> >> >> :abbreviation = "UGRD";
> >> >> >> :Grib_Variable_Id = "VAR_0-2-2_L103";
> >> >> >> :Grib2_Parameter = 0, 2, 2; // int
> >> >> >> :Grib2_Parameter_Discipline = "Meteorological products";
> >> >> >> :Grib2_Parameter_Category = "Momentum";
> >> >> >> :Grib2_Parameter_Name = "u-component of wind";
> >> >> >> :Grib2_Level_Type = 103; // int
> >> >> >> :Grib2_Generating_Process_Type = "Forecast";
> >> >> >>
> >> >> >> However, if I switch to 4.5 the program crashes since 4.5 finds 2
> >> >> >> variables of the same name and findVariable returns null
> >> >> >>
> >> >> >> float u-component_of_wind_height_above_ground(time=745,
> >> >> >> height_above_ground=1, lat=576, lon=1152);
> >> >> >> :long_name = "u-component of wind @ Specified height level above
> >> >> >> ground";
> >> >> >> :units = "m/s";
> >> >> >> :missing_value = NaNf; // float
> >> >> >> :abbreviation = "UGRD";
> >> >> >> :coordinates = "time height_above_ground ";
> >> >> >> :Grib_Variable_Id = "VAR_0-2-2_L103";
> >> >> >> :Grib2_Parameter = 0, 2, 2; // int
> >> >> >> :Grib2_Parameter_Discipline = "Meteorological products";
> >> >> >> :Grib2_Parameter_Category = "Momentum";
> >> >> >> :Grib2_Parameter_Name = "u-component of wind";
> >> >> >> :Grib2_Level_Type = "Specified height level above ground";
> >> >> >> :Grib2_Generating_Process_Type = "Forecast";
> >> >> >
> >> >> >
> >> >> >>
> >> >> >> float u-component_of_wind_height_above_ground(reftime=124, time=7,
> >> >> >> height_above_ground=1, lat=576, lon=1152);
> >> >> >> :long_name = "u-component of wind @ Specified height level above
> >> >> >> ground";
> >> >> >> :units = "m/s";
> >> >> >> :missing_value = NaNf; // float
> >> >> >> :abbreviation = "UGRD";
> >> >> >> :coordinates = "reftime time height_above_ground ";
> >> >> >> :Grib_Variable_Id = "VAR_0-2-2_L103";
> >> >> >> :Grib2_Parameter = 0, 2, 2; // int
> >> >> >> :Grib2_Parameter_Discipline = "Meteorological products";
> >> >> >> :Grib2_Parameter_Category = "Momentum";
> >> >> >> :Grib2_Parameter_Name = "u-component of wind";
> >> >> >> :Grib2_Level_Type = "Specified height level above ground";
> >> >> >> :Grib2_Generating_Process_Type = "Forecast";
> >> >> >>
> >> >> >> Again, it is all about the backward compatibility.
> >> >> >
> >> >> >
> >> >> > CFSR encoding is notoriously messed up. Its likely there is some
> >> >> > glitch
> >> >> > like
> >> >> > table version that differs between two records. If you can send me
> a
> >> >> > minimum
> >> >> > set of files that reproduces the problem, i will see if theres a
> >> >> > workaround.
> >> >> >
> >> >> > Also, you could consider moving to version 4.6, which will once
> again
> >> >> > write
> >> >> > new CDM index files for GRIB. The advantage is that it scales to
> >> >> > large
> >> >> > collections. The disadvantage is that it is currently alpha. But if
> >> >> > you
> >> >> > are
> >> >> > interested, I can point you to the current snapshot.
> >> >>
> >> >> I have included the link in my previous e-mail, here it is again:
> >> >> http://nomads.ncdc.noaa.gov/data/cfsr/198209/wnd10m.gdas.198209.grb2
> >> >> and you can find there a lot of other files for other dates
> >> >>
> >> >>
> >> >> >
> >> >> >>
> >> >> >>
> >> >> >> 3) Would it be helpful to maintain a collective repository of
> files
> >> >> >> from diverse domains and different formats that are read by
> >> >> >> NetCDF-Java users and test all subsequent releases against that
> >> >> >> repository?
> >> >> >>
> >> >> >> It could reveal differences/changes and document them at the early
> >> >> >> stage and avoid things like described (also recall HDF issues like
> >> >> >>
> >> >> >>
> >> >> >>
> http://www.unidata.ucar.edu/support/help/MailArchives/netcdf/msg12946.html
> >> >> >> )
> >> >> >
> >> >> >
> >> >> > yes, that would be great. we have a set of files that we
> >> >> > automatically
> >> >> > read
> >> >> > in our unit tests, which would have caught this problem. It can be
> >> >> > hard
> >> >> > to
> >> >> > whittle the files down to a manageable size (a few Mbytes would be
> >> >> > nice).
> >> >>
> >> >>
> >> >> Maybe there should be tests embedded in the library itself so
> everyone
> >> >> could run against their own collection and report the problems?
> >> >
> >> >
> >
> >
>
