Hi Antonio:
Some answers embedded below:
On Mon, Mar 9, 2015 at 1:43 PM, Antonio Rodriges <antonio.rrz@xxxxxxxxx>
wrote:
> Hello,
>
> Thank you for the 4.5 release.
>
> However, there are troubles migrating from 4.3 to 4.5 (I am trying to
> migrate from 4.3 to a higher version for quite long now).
>
> I have used a pre-release snapshot kindly provided at the 17th of
> January but I suppose the following issues may stlll hold for the
> final release. They are mainly concerned with IOSP grib2 backward
> compatibility.
>
> 1) Creation of index files
> version 4.3 does not create in the same directory where original grib2
> files are located indexes for each date I access. But 4.5 does so for
> each date:
> 2015-02-18/06:09:50.118/GMT-00:00 [ucar.nc2.grib.collection.Grib2Iosp]
> INFO GribCollectionBuilder write
>
> D:/RS_DATA/worker/symlinks/CFSR/windspeed/wnd10m.gdas.200508.grb2-20050818-180000.ncx2
> ok=true
> 2015-02-18/06:09:50.120/GMT-00:00 [ucar.nc2.grib.collection.Grib2Iosp]
> DEBUG createIndex for
>
> D:\RS_DATA\worker\symlinks\CFSR\windspeed\wnd10m.gdas.200508.grb2-20050815-000000.ncx2
> 2015-02-18/06:09:50.120/GMT-00:00 [ucar.nc2.grib.collection.Grib2Iosp]
> DEBUG write RecordMaps: bytes = 218 record = 14 bytesPerRecord=15
> 2015-02-18/06:09:50.121/GMT-00:00 [ucar.nc2.grib.collection.Grib2Iosp]
> DEBUG write GribCollectionIndex= 551 bytes
>
> At the end of traversing the dates from a given interval one arrives
> at a point where the directory has files that are not anticipated to
> emerge (for one previously used 4.3).
>
> Even if index creation time is negligible and it significantly
> accelerates further operation and etc. etc. it is quite logical to
> keep this feature off since it was off or not present in 4.3.
>
If you want to put index files in a sperate directory from the data files,
use following in threddsConfig.xml :
<GribIndex>
<alwaysUse>true</alwaysUse>
<dir>/tomcat_home/content/thredds/cache/grib/</dir>
</GribIndex>
see
http://www.unidata.ucar.edu/software/thredds/v4.5/tds/reference/ThreddsConfigXMLFile.html#GribIndexWriting
I dont think this has changed since 4.3 (?)
> You may argue that it is not so critical but it depends. The following
> issue is more severe.
>
> 2) Consider grib2 files of CFSR reanalysis, for example:
> http://nomads.ncdc.noaa.gov/data/cfsr/198209/wnd10m.gdas.198209.grb2
>
> If I would like to read "u-component_of_wind_height_above_ground" in
> version 4.3 it is OK since it finds only a single variable with this
> name:
>
> float u-component_of_wind_height_above_ground(time=745,
> height_above_ground=1, lat=576, lon=1152);
> :long_name = "u-component of wind @ Specified height level above ground";
> :units = "m/s";
> :missing_value = NaNf; // float
> :abbreviation = "UGRD";
> :Grib_Variable_Id = "VAR_0-2-2_L103";
> :Grib2_Parameter = 0, 2, 2; // int
> :Grib2_Parameter_Discipline = "Meteorological products";
> :Grib2_Parameter_Category = "Momentum";
> :Grib2_Parameter_Name = "u-component of wind";
> :Grib2_Level_Type = 103; // int
> :Grib2_Generating_Process_Type = "Forecast";
>
> However, if I switch to 4.5 the program crashes since 4.5 finds 2
> variables of the same name and findVariable returns null
>
> float u-component_of_wind_height_above_ground(time=745,
> height_above_ground=1, lat=576, lon=1152);
> :long_name = "u-component of wind @ Specified height level above ground";
> :units = "m/s";
> :missing_value = NaNf; // float
> :abbreviation = "UGRD";
> :coordinates = "time height_above_ground ";
> :Grib_Variable_Id = "VAR_0-2-2_L103";
> :Grib2_Parameter = 0, 2, 2; // int
> :Grib2_Parameter_Discipline = "Meteorological products";
> :Grib2_Parameter_Category = "Momentum";
> :Grib2_Parameter_Name = "u-component of wind";
> :Grib2_Level_Type = "Specified height level above ground";
> :Grib2_Generating_Process_Type = "Forecast";
>
> float u-component_of_wind_height_above_ground(reftime=124, time=7,
> height_above_ground=1, lat=576, lon=1152);
> :long_name = "u-component of wind @ Specified height level above ground";
> :units = "m/s";
> :missing_value = NaNf; // float
> :abbreviation = "UGRD";
> :coordinates = "reftime time height_above_ground ";
> :Grib_Variable_Id = "VAR_0-2-2_L103";
> :Grib2_Parameter = 0, 2, 2; // int
> :Grib2_Parameter_Discipline = "Meteorological products";
> :Grib2_Parameter_Category = "Momentum";
> :Grib2_Parameter_Name = "u-component of wind";
> :Grib2_Level_Type = "Specified height level above ground";
> :Grib2_Generating_Process_Type = "Forecast";
>
> Again, it is all about the backward compatibility.
>
CFSR encoding is notoriously messed up. Its likely there is some glitch
like table version that differs between two records. If you can send me a
minimum set of files that reproduces the problem, i will see if theres a
workaround.
Also, you could consider moving to version 4.6, which will once again write
new CDM index files for GRIB. The advantage is that it scales to large
collections. The disadvantage is that it is currently alpha. But if you are
interested, I can point you to the current snapshot.
>
> 3) Would it be helpful to maintain a collective repository of files
> from diverse domains and different formats that are read by
> NetCDF-Java users and test all subsequent releases against that
> repository?
>
> It could reveal differences/changes and document them at the early
> stage and avoid things like described (also recall HDF issues like
> http://www.unidata.ucar.edu/support/help/MailArchives/netcdf/msg12946.html
> )
>
yes, that would be great. we have a set of files that we automatically read
in our unit tests, which would have caught this problem. It can be hard to
whittle the files down to a manageable size (a few Mbytes would be nice).
sorry for the trouble, thanks for reporting.
John
>
>
