[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDFJava #IUX-310433]: Aggregating with Groups in NCML



If he extracted the records he wants and put them in a single file, that would 
work. 

> Hi John...
> 
> Many thanks for looking into this.  I'm not sure about the projection
> issues, since HP was only working with the
> "proj1/Instantaneous_Rain_Rate" and this display just fine in the IDV.
> His problem is he has 96 files and wants to show
> them in a loop (without having to select each of them....).
> 
> So, I must conclude from your comments that the 2 GDS's are the real
> issue with the aggregation, and because of that he is out-of-luck for
> now, unless your "stab" works out...?  I'll keep my fingers crossed!
> 
> Thank you again...let us know if you have any success!
> 
> Best wishes,
> 
> tom
> 
> On Mon, Jun 20, 2011 at 2:43 PM, Unidata netCDF Java Support
> <address@hidden> wrote:
> > Hi Tom, HP:
> >
> > I have two files, each with 3 records.
> >
> > 1) 1 record uses "space view perspective" (attached MSG1.png), 2 records 
> > use a lat/lon GDS (attached MSG2.png). The first has an interesting banded 
> > pattern. The second has -9999 as the first lat/lon point (see GDS dump 
> > below). Im guessing that this might mean "calculate the domain using only 
> > the second lat/lon point and nx/ny/dx/dy". But theres nothing in the GRIB 
> > spec that seems to allow this. HP, is this the case? Is "use -9999 to mean 
> > calculate the grid from the other info" part of GRIB standard or is it some 
> > local convention from EUmetsat?
> >
> > 2) the 2 lat/lon have unknown parameters 3-1-98 and 3-1-99 (3=discipline, 
> > 1=category,98/99=parameter). These are not in the latest WMO parameter 
> > tables, and they are not in the "local parameter" area of the table (>191). 
> > HP, can you comment on what these are, and if EUM is using WMO parameter 
> > numbers for local parameters ?
> >
> > 3) theres some interesting metadata in the local use section that we are 
> > not picking up. Ive copied a complete dump of one of the records below.
> >
> > 4) because there are 2 GDS, there are 2 groups. the aggregation is failing 
> > with groups. im not sure if i can fix this quickly, although im taking a 
> > stab at it.
> >
> > 4) TDS 4.3 has a complete refactoring of GRIB-2, with a new feature to 
> > specify a collection of grib2 files, which are then correctly aggregated. 
> > So it works correctly because NcML aggregation is not needed. I can give 
> > you an advance copy, but this wont be released for a few months, and then 
> > theres the question of IDV integration.
> >
> > John
> >
> > ----
> >
> > File=0 
> > E:/work/whittaker/MSG2-SEVI-MSGMPEG-0100-0100-20090830163000.000000000Z-1002820.grb
> > Header="(0)(2)(1)(6)(1)(2)(0)(0)(234)u(10)(20)2(4)(5)(1)(0)!V(9)(13)(10)HeaderVersionNo
> >                    0(13)(10)FileType                          
> > 2(13)(10)SubHeaderType                     1(13)(10)SourceFacilityID        
> >           6(13)(10)SourceEnvId                       
> > 1(13)(10)SourceInstanceId                  2(13)(10)SourceSUId              
> >       60021(13)(10)SourceCPUId          10  20  50   4(13)(10)DestFaciliyId 
> >                     5(13)(10)DestEnvId                         
> > 1(13)(10)DataFieldLength             
> > 2184713(13)(10)(0)(171)(24)I(181)(3)(153)(10) 
> > (1)B(13)(10)SubHeaderVersionNo                0(13)(10)ServiceType          
> >            171(13)(10)ServiceSubType                   24(13)(10)FileTime   
> >           18869:60361248(13)(10)SpacecraftId                    
> > 322(13)(10)(0)(16)(0)(0)(1)(1)I(181)(3)(138)
 
_@ddd(1)(0)(0)I(181)(3)(138)_@(1)(0)I(181)(3)(138)_@(0)(0)I(181)(3)(138)_@(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0
> >  
> > )(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(1)(0)(0)(0)(1)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)(0)"
> >
> > Grib2IndicatorSection
> >  Discipline = (3) Space products
> >  Length     = 2184244
> >
> > Grib2IdentificationSection
> >  Center        = (254) EUMETSAT Operation Centre
> >  SubCenter     = (0) null
> >  Master Table  = 3
> >  Local Table   = 0
> >  RefTimeSignif = 3 (Observation time)
> >  RefTime       = 2009-08-30T16:30:00.000Z
> >  RefTime Fields = 2009-8-30 16:30:0
> >  ProductionStatus      = 1 (Operational test products)
> >  TypeOfProcessedData   = 6 (Processed satellite observations)
> >
> > Grib2GridDefinitionSection hash=-1919541760 crc=3711080673
> >  Length             = 72
> >  Source  (3.0)      = 0 (Specified in Code table 3.1)
> >  Npts               = 576
> >  Template (3.1)     = 0
> >
> > (3.0) Grid definition template 3.0 - latitude/longitude (or equidistant 
> > cylindrical, or Plate Carrée)
> >  1:                                                                         
> >         GDS length == 72
> >  5:                                                                         
> >            Section == 3
> >  6:                                             Source of Grid Definition 
> > (see code table 3.0) == 0 (table 3.0: Specified in Code table 3.1)
> >  7:                                                                      
> > Number of data points == 576
> >  11:                                             Number of octects for 
> > optional list of numbers == 0
> >  12:                                                          
> > Interpretation of list of numbers == 0 (table 3.11: There is no appended 
> > list)
> >  13:                                                            Grid 
> > Definition Template Number == 0
> >  15:                                                                        
> >  Shape of the Earth == 3 (table 3.2: Earth assumed oblate spheroid with 
> > major and minor axes specified (in km) by data producer)
> >  16:                                                  Scale factor of 
> > radius of spherical Earth == 255
> >  17:                                                  Scaled value of 
> > radius of spherical Earth == -9999
> >  21:                                        Scale factor of major axis of 
> > oblate spheroid Earth == 4
> >  22:                                        Scaled value of major axis of 
> > oblate spheroid Earth == 63781400
> >  26:                                        Scale factor of minor axis of 
> > oblate spheroid Earth == 4
> >  27:                                        Scaled value of minor axis of 
> > oblate spheroid Earth == 63567550
> >  31:                                                     Ni - number of 
> > points along a parallel == 24
> >  35:                                                     Nj - number of 
> > points along a meridian == 24
> >  39:                                               Basic angle of the 
> > initial production domain == 0
> >  43: Subdivisions of basic angle used to define extreme longitudes and 
> > latitudes, and direction increments == -9999
> >  47:                                                         La1 - latitude 
> > of first grid point == -9999
> >  51:                                                        Lo1 - longitude 
> > of first grid point == -9999
> >  55:                                                             Resolution 
> > and component flags == 56
> >  56:                                                          La2 - 
> > latitude of last grid point == 60000000
> >  60:                                                         Lo2 - 
> > longitude of last grid point == 60000000
> >  64:                                                                 Di - i 
> > direction increment == 5000000
> >  68:                                                                 Dj - j 
> > direction increment == 5000000
> >  72:                                                                        
> >       Scanning mode == 64
> >  73:                                  List of number of points along each 
> > meridian or parallel. == -9999
> >
> > Grib2ProductDefinitionSection
> >
> > (4.30) Product definition template 4.30 - satellite product
> >  1:                                                                         
> >         PDS length == 24
> >  5:                                                                         
> >            Section == 4
> >  6:                                                Number of coordinates 
> > values after Template == 0
> >  8:                                                         Product 
> > Definition Template Number == 30
> >  10:                                                                        
> >  Parameter category == 1
> >  11:                                                                        
> >    Parameter number == 98
> >  12:                                                                 Type 
> > of generating process == 8 (table 4.3: Observation)
> >  13:                  Observation generating process identifier (defined by 
> > originating centre) == 7
> >  14:                                                 Number of contributing 
> > spectral bands (NB) == 1
> >
> > Grib2SectionDataRepresentation
> >  Template           = 0 (Grid point data - simple packing)
> >  NPoints            = 576
> >
> > Grib2SectionData
> >  Starting Pos       = 2183854
> >  Data Length        = 581
> >
> >
> >> Hi John....
> >>
> >> Did the data files sample get through okay?  If not, I have also posted 
> >> this at:
> >> <ftp://ftp.ssec.wisc.edu/pub/ssec/tomw/hpgrids.zip>
> >>
> >> Thanks again for looking into this...
> >>
> >> tom
> >>
> >> On Mon, Jun 13, 2011 at 2:32 PM, Tom Whittaker <address@hidden> wrote:
> >> > Hi John...
> >> >
> >> > Thanks....I had simply mis-typed it (guess I should use
> >> > cut-and-paste)....but since it does not work for me, I'm attaching a
> >> > zip file with the 2 nc files and the attempt at ncml...
> >> >
> >> > Thanks ahead for your help!!
> >> >
> >> > tom
> >> >
> >> > On Fri, Jun 10, 2011 at 4:11 PM, Unidata netCDF Java Support
> >> > <address@hidden> wrote:
> >> >> hi tom:
> >> >>
> >> >> jointExisting should be joinExisting.
> >> >>
> >> >> if thats not the actual problem, can you send me 2 files to check?
> >> >>
> >> >> john
> >> >>
> >> >>> I have several files that I want to aggregate over time.  Each file
> >> >>> has two Groups, with a dimension "time" which is used for the desired
> >> >>> variable...for example:
> >> >>>
> >> >>> Group proj1 {
> >> >>> dimensions:
> >> >>> time = 1;
> >> >>> y = 3712;
> >> >>> x = 3712;
> >> >>> variables:
> >> >>> float Instantaneous_Rain_Rate(time=1, y=3712, x=3712);
> >> >>> ...
> >> >>>
> >> >>> I am trying to use NCML to accomplish this, with an "aggregation" 
> >> >>> element like:
> >> >>>
> >> >>> <aggregation dimName="time" type="jointExisting">
> >> >>> ...
> >> >>>
> >> >>> But apparently "time" is not the right syntax for this.  I've tried
> >> >>> "proj1/time" as well, but to no available.  I'm hoping you can help.
> >> >>>
> >> >>> Thanks!
> >> >>>
> >> >>> tom
> >> >>>
> >> >>> --
> >> >>> Tom Whittaker
> >> >>> University of Wisconsin-Madison
> >> >>> Space Science & Engineering Center (SSEC)
> >> >>> Cooperative Institute for Meteorological Satellite Studies (CIMSS)
> >> >>> 1225 W. Dayton Street
> >> >>> Madison, WI  53706  USA
> >> >>> ph: +1 608 262 2759
> >> >>>
> >> >>>
> >> >>
> >> >>
> >> >> Ticket Details
> >> >> ===================
> >> >> Ticket ID: IUX-310433
> >> >> Department: Support netCDF Java
> >> >> Priority: Normal
> >> >> Status: Open
> >> >>
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Tom Whittaker
> >> > University of Wisconsin-Madison
> >> > Space Science & Engineering Center (SSEC)
> >> > Cooperative Institute for Meteorological Satellite Studies (CIMSS)
> >> > 1225 W. Dayton Street
> >> > Madison, WI  53706  USA
> >> > ph: +1 608 262 2759
> >> >
> >>
> >>
> >>
> >> --
> >> Tom Whittaker
> >> University of Wisconsin-Madison
> >> Space Science & Engineering Center (SSEC)
> >> Cooperative Institute for Meteorological Satellite Studies (CIMSS)
> >> 1225 W. Dayton Street
> >> Madison, WI  53706  USA
> >> ph: +1 608 262 2759
> >>
> >>
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: IUX-310433
> > Department: Support netCDF Java
> > Priority: Normal
> > Status: Open
> >
> >
> 
> 
> 
> --
> Tom Whittaker
> University of Wisconsin-Madison
> Space Science & Engineering Center (SSEC)
> Cooperative Institute for Meteorological Satellite Studies (CIMSS)
> 1225 W. Dayton Street
> Madison, WI  53706  USA
> ph: +1 608 262 2759
> 
> 


Ticket Details
===================
Ticket ID: IUX-310433
Department: Support netCDF Java
Priority: Normal
Status: Closed