Thanks Sean!
Stuck in meetings all Friday so I didn't get to reply sooner, but really this
is a big help and the linked gist got me through the steps.
Thanks again for all your hard work!
-Tom
________________________________
From: Sean Arms <sarms@xxxxxxxx>
Sent: Thursday, January 5, 2017 4:55 PM
To: Hagman,Tom
Cc: netcdf-java@xxxxxxxxxxxxxxxx
Subject: Re: [netcdf-java] AggregateExisting with multiple Time dimensions
present
CAUTION - EXTERNAL EMAIL
Greetings Tom!
Ah GRIB...good times, or something :-)
Here is a mailing list thread discussing some of the issues you
outlined about the name of the time coordinate:
http://www.unidata.ucar.edu/mailing_lists/archives/netcdf-java/2015/msg00133.html
TL;DR; The GRIB files you've downloaded are essentially a collection
of GRIB records concatenated into a single file, and the order of
those GRIB records can change from file to file. The naming of the
dimensions netcdf-java creates is based on when they are first
encountered in a file, and thus can change from file to file.
Unfortunately, NcML aggregation with GRIB files won't really work
anymore (as of netcdf-java 4.3, I do believe).
However, there is a way to aggregate GRIB files using a
GribFeatureCollection. The code is a bit lower level, but it is sort
of what the THREDDS Data Server uses when serving out collections of
GRIB files. Here is a code example:
https://gist.github.com/lesserwhirls/5eea2ffa0f1e66b50fb87b971fa111b5
The code example in the gist is using non-public parts of the
netCDF-Java API, but should work. It is my hope to extend NcML to be
able to do GRIB aggregations in the next version of netCDF-Java (v5),
but this is what we have currently.
Finally, while we just released the 4.6.7 version of netCDF-Java, I
forgot one small but critical step in linking to the NCEP GRIB2 tables
after doing a table update, and as a result this code won't work
smoothly with 4.6.7 (some variable names will look like
VAR0-7-192_FROM_7-0--1_surface...yuck...). We are in the process of
cutting a new release that has the problem fixed. In the meantime, you
may use the 4.6.6 version of the netCDF-All.jar file from here:
https://github.com/Unidata/thredds/releases/tag/v4.6.6
Cheers,
Sean
On Thu, Jan 5, 2017 at 1:29 PM, Hagman,Tom <Tom.Hagman@xxxxxxxxxxxxxx> wrote:
> Hi all,
>
>
> Firstly, thank you all so much for your hard work on this library.
>
>
> For my brain teaser, I've searched through the docs and mailing list
> archives awhile and am having a hard time putting together the steps I need
> to handle this aggregation.
>
> CFSR 1 hour data files data from here :
> http://rda.ucar.edu/datasets/ds094.0/
>
> cdas_20161215_0000_f00000_G4.grib2
> cdas_20161215_0000_f00100_G4
> cdas_20161215_0000_f00200_G4
> cdas_20161215_0000_f00300_G4
> etc...
> The hourly files declare 2 time dimensions, one with bounds set and another
> without.
>
> cdas_20161215_0000_f00300_G4.grib2
> double time(time=1);
> :units = "Hour since 2016-12-15T00:00:00Z";
> :standard_name = "time";
> :long_name = "GRIB forecast or observation time";
> :calendar = "proleptic_gregorian";
> :bounds = "time_bounds";
> double time_bounds(time=1, 2);
> :units = "Hour since 2016-12-15T00:00:00Z";
> :long_name = "bounds for time";
> double time1(time1=1);
> :units = "Hour since 2016-12-15T00:00:00Z";
> :standard_name = "time";
> :long_name = "GRIB forecast or observation time";
> :calendar = "proleptic_gregorian";
>
> The problem is that when I step through each dataset creation, different
> hourly files will swap names for the 2 time dimension names. So then
> AggregationExisting is unable to find the dimension name 'time' for certain
> files, e.g. on the u-component_of_wind_isobaric variable in the 0300 file
> because it was swapped to time1 instead.
>
>
> Time dimension name issue illustrated in 2 files:
>
> //cdas_20161215_0000_f00300_G4.grib2
>
> float u-component_of_wind_isobaric(time1=1, isobaric3=37, lat=361, lon=720);
> :long_name = "u-component of wind @ Isobaric surface";
> :units = "m/s";
> :abbreviation = "UGRD";
> :missing_value = NaNf; // float
> :grid_mapping = "LatLon_Projection";
> :coordinates = "reftime time1 isobaric3 lat lon ";
> :Grib_Variable_Id = "VAR_0-2-2_L100";
> :Grib2_Parameter = 0, 2, 2; // int
> :Grib2_Parameter_Discipline = "Meteorological products";
> :Grib2_Parameter_Category = "Momentum";
> :Grib2_Parameter_Name = "u-component of wind";
> :Grib2_Level_Type = "Isobaric surface";
> :Grib2_Generating_Process_Type = "Forecast";
>
>
> //cdas_20161215_0000_f00200_G4.grib2
>
> float u-component_of_wind_isobaric(time=1, isobaric3=37, lat=361, lon=720);
> :long_name = "u-component of wind @ Isobaric surface";
> :units = "m/s";
> :abbreviation = "UGRD";
> :missing_value = NaNf; // float
> :grid_mapping = "LatLon_Projection";
> :coordinates = "reftime time isobaric3 lat lon ";
> :Grib_Variable_Id = "VAR_0-2-2_L100";
> :Grib2_Parameter = 0, 2, 2; // int
> :Grib2_Parameter_Discipline = "Meteorological products";
> :Grib2_Parameter_Category = "Momentum";
> :Grib2_Parameter_Name = "u-component of wind";
> :Grib2_Level_Type = "Isobaric surface";
> :Grib2_Generating_Process_Type = "Forecast";
>
> Code I'm calling:
>
> List<String> variableNames =
> Arrays.asList("u-component_of_wind_isobaric","u-component_of_wind_height_above_ground","v-component_of_wind_isobaric","v-component_of_wind_height_above_ground","Pressure_reduced_to_MSL_msl","Geopotential_height_isobaric");
> NetcdfDataset netcdfDataset = new NetcdfDataset();
> //here i'm trying to aggregate on a dimension called 'time'
> AggregationExisting aggregationExisting = new
> AggregationExisting(netcdfDataset, "time", null);
> aggregationExisting.addDatasetScan(null,
> "/cfsr-gribs/201612/",
> "G4.grib2",
> null,
> null,
> NetcdfDataset.getDefaultEnhanceMode(),
> "false",
> null);
> aggregationExisting.persistWrite();
> aggregationExisting.finish(new CancelTaskImpl());
> GridDataset gridDataset = new GridDataset(netcdfDataset);
> //....spline data in merged dataset in memory
> //writer is the NetcdfFileWriter with the merged file path
> writer.setRedefineMode(true);
> CFGridWriter2.writeFile(gridDataset, variableNames,
> gridDataset.getBoundingBox(), null, 1, null, null, 1, true, writer);
>
> This is my first NetCDF library use so I'm shopping for some preprocessing
> tools to get these datasets merged that have this quirk. Could I move all
> the variables into the same time dimension and rename it, for instance? Even
> a link to an example I missed would be helpful. Otherwise I'm guessing I
> will look into manually declaring dimensions and using readDataSlice() to
> manually copy in the data into a new merged file.
>
>
>
> Thanks!
> -Tom Hagman
>
>
> ________________________________
>
> CONFIDENTIALITY NOTICE: This e-mail message is for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information. Any unauthorized review, use, disclosure or distribution of any
> kind is strictly prohibited. If you are not the intended recipient, please
> contact the sender via reply e-mail and destroy all copies of the original
> message. Thank you.
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> netcdf-java mailing list
> netcdf-java@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
