Hello,
2015-03-10 2:57 GMT+03:00 John Caron <caron@xxxxxxxx>:
> Hi Antonio:
>
> Some answers embedded below:
>
> On Mon, Mar 9, 2015 at 1:43 PM, Antonio Rodriges <antonio.rrz@xxxxxxxxx>
> wrote:
>>
>> Hello,
>>
>> Thank you for the 4.5 release.
>>
>> However, there are troubles migrating from 4.3 to 4.5 (I am trying to
>> migrate from 4.3 to a higher version for quite long now).
>>
>> I have used a pre-release snapshot kindly provided at the 17th of
>> January but I suppose the following issues may stlll hold for the
>> final release. They are mainly concerned with IOSP grib2 backward
>> compatibility.
>>
>> 1) Creation of index files
>> version 4.3 does not create in the same directory where original grib2
>> files are located indexes for each date I access. But 4.5 does so for
>> each date:
>> 2015-02-18/06:09:50.118/GMT-00:00 [ucar.nc2.grib.collection.Grib2Iosp]
>> INFO GribCollectionBuilder write
>>
>> D:/RS_DATA/worker/symlinks/CFSR/windspeed/wnd10m.gdas.200508.grb2-20050818-180000.ncx2
>> ok=true
>> 2015-02-18/06:09:50.120/GMT-00:00 [ucar.nc2.grib.collection.Grib2Iosp]
>> DEBUG createIndex for
>>
>> D:\RS_DATA\worker\symlinks\CFSR\windspeed\wnd10m.gdas.200508.grb2-20050815-000000.ncx2
>> 2015-02-18/06:09:50.120/GMT-00:00 [ucar.nc2.grib.collection.Grib2Iosp]
>> DEBUG write RecordMaps: bytes = 218 record = 14 bytesPerRecord=15
>> 2015-02-18/06:09:50.121/GMT-00:00 [ucar.nc2.grib.collection.Grib2Iosp]
>> DEBUG write GribCollectionIndex= 551 bytes
>>
>> At the end of traversing the dates from a given interval one arrives
>> at a point where the directory has files that are not anticipated to
>> emerge (for one previously used 4.3).
>>
>> Even if index creation time is negligible and it significantly
>> accelerates further operation and etc. etc. it is quite logical to
>> keep this feature off since it was off or not present in 4.3.
>
>
> If you want to put index files in a sperate directory from the data files,
> use following in threddsConfig.xml :
>
> <GribIndex>
> <alwaysUse>true</alwaysUse>
> <dir>/tomcat_home/content/thredds/cache/grib/</dir>
> </GribIndex>
>
I do not use THREDDS, I use NetCDF-Java library
> see
>
> http://www.unidata.ucar.edu/software/thredds/v4.5/tds/reference/ThreddsConfigXMLFile.html#GribIndexWriting
>
> I dont think this has changed since 4.3 (?)
And I have 4.3 and tried 4.5 so I can't update
>
>>
>> You may argue that it is not so critical but it depends. The following
>> issue is more severe.
>>
>> 2) Consider grib2 files of CFSR reanalysis, for example:
>> http://nomads.ncdc.noaa.gov/data/cfsr/198209/wnd10m.gdas.198209.grb2
>>
>> If I would like to read "u-component_of_wind_height_above_ground" in
>> version 4.3 it is OK since it finds only a single variable with this
>> name:
>>
>> float u-component_of_wind_height_above_ground(time=745,
>> height_above_ground=1, lat=576, lon=1152);
>> :long_name = "u-component of wind @ Specified height level above
>> ground";
>> :units = "m/s";
>> :missing_value = NaNf; // float
>> :abbreviation = "UGRD";
>> :Grib_Variable_Id = "VAR_0-2-2_L103";
>> :Grib2_Parameter = 0, 2, 2; // int
>> :Grib2_Parameter_Discipline = "Meteorological products";
>> :Grib2_Parameter_Category = "Momentum";
>> :Grib2_Parameter_Name = "u-component of wind";
>> :Grib2_Level_Type = 103; // int
>> :Grib2_Generating_Process_Type = "Forecast";
>>
>> However, if I switch to 4.5 the program crashes since 4.5 finds 2
>> variables of the same name and findVariable returns null
>>
>> float u-component_of_wind_height_above_ground(time=745,
>> height_above_ground=1, lat=576, lon=1152);
>> :long_name = "u-component of wind @ Specified height level above
>> ground";
>> :units = "m/s";
>> :missing_value = NaNf; // float
>> :abbreviation = "UGRD";
>> :coordinates = "time height_above_ground ";
>> :Grib_Variable_Id = "VAR_0-2-2_L103";
>> :Grib2_Parameter = 0, 2, 2; // int
>> :Grib2_Parameter_Discipline = "Meteorological products";
>> :Grib2_Parameter_Category = "Momentum";
>> :Grib2_Parameter_Name = "u-component of wind";
>> :Grib2_Level_Type = "Specified height level above ground";
>> :Grib2_Generating_Process_Type = "Forecast";
>
>
>>
>> float u-component_of_wind_height_above_ground(reftime=124, time=7,
>> height_above_ground=1, lat=576, lon=1152);
>> :long_name = "u-component of wind @ Specified height level above
>> ground";
>> :units = "m/s";
>> :missing_value = NaNf; // float
>> :abbreviation = "UGRD";
>> :coordinates = "reftime time height_above_ground ";
>> :Grib_Variable_Id = "VAR_0-2-2_L103";
>> :Grib2_Parameter = 0, 2, 2; // int
>> :Grib2_Parameter_Discipline = "Meteorological products";
>> :Grib2_Parameter_Category = "Momentum";
>> :Grib2_Parameter_Name = "u-component of wind";
>> :Grib2_Level_Type = "Specified height level above ground";
>> :Grib2_Generating_Process_Type = "Forecast";
>>
>> Again, it is all about the backward compatibility.
>
>
> CFSR encoding is notoriously messed up. Its likely there is some glitch like
> table version that differs between two records. If you can send me a minimum
> set of files that reproduces the problem, i will see if theres a workaround.
>
> Also, you could consider moving to version 4.6, which will once again write
> new CDM index files for GRIB. The advantage is that it scales to large
> collections. The disadvantage is that it is currently alpha. But if you are
> interested, I can point you to the current snapshot.
I have included the link in my previous e-mail, here it is again:
http://nomads.ncdc.noaa.gov/data/cfsr/198209/wnd10m.gdas.198209.grb2
and you can find there a lot of other files for other dates
>
>>
>>
>> 3) Would it be helpful to maintain a collective repository of files
>> from diverse domains and different formats that are read by
>> NetCDF-Java users and test all subsequent releases against that
>> repository?
>>
>> It could reveal differences/changes and document them at the early
>> stage and avoid things like described (also recall HDF issues like
>> http://www.unidata.ucar.edu/support/help/MailArchives/netcdf/msg12946.html
>> )
>
>
> yes, that would be great. we have a set of files that we automatically read
> in our unit tests, which would have caught this problem. It can be hard to
> whittle the files down to a manageable size (a few Mbytes would be nice).
Maybe there should be tests embedded in the library itself so everyone
could run against their own collection and report the problems?
