[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDFJava #HIP-291542]: Question on NcML aggregation



Matt,

I set up an aggregation using four of the files.  At it's simplest, this works 
to join the datasets along the existing time dimension:

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
  <aggregation dimName="time" type="joinExisting">
    <netcdf id="1" location="DOU001_000_20070530_20070906.nc"/>
    <netcdf id="2" location="DOU001_000_20070906_20071107.nc"/>
    <netcdf id="3" location="DOU001_000_20080807_20081204.nc"/>
    <netcdf id="4" location="DOU001_000_20100803_20101117.nc"/>
  </aggregation>
</netcdf>

Since you already have a good dimension for aggregating (time), then you 
probably don't need to worry about doing a joinNew.  You would use that if 
there were no time data in the files, and you wanted to add it into the dataset 
you were serving, for instance.

Let me know if there is something else you are trying to do, I'm happy to help.

-Lansing

> Lansing,
> 
> When I set it up using the "joinExisting" on the time dimension it seems to
> aggregate them. Below is the ncml:
> <?xml version="1.0" encoding="UTF-8"?>
> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> <variable name="lat">
> <attribute name="standard_name" type="String" value="latitude"/>
> <attribute name="axis" type="string" value="Y"/>
> <attribute name="units" type="string" value="degrees_north"/>
> </variable>
> <variable name="lon">
> <attribute name="standard_name" type="string" value="longitude"/>
> <attribute name="axis" type="string" value="X"/>
> <attribute name="units" type="string" value="degrees_east"/>
> </variable>
> <variable name="time">
> <attribute name="standard_name" type="string" value="time"/>
> <attribute name="axis" type="string" value="T"/>
> <attribute name="units" type="string" value="seconds since 1970-01-01
> 00:00:00"/>
> </variable>
> <aggregation dimName="time" type="joinExisting">
> <netcdf id="1" location="
> http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20070530_20070906.nc
> "/>
> <netcdf id="2" location="
> http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20070906_20071107.nc
> "/>
> <netcdf id="3" location="
> http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20080807_20081204.nc
> "/>
> <netcdf id="4" location="
> http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20100803_20101117.nc
> "/>
> <netcdf id="5" location="
> http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20101117_20110318.nc
> "/>
> </aggregation>
> </netcdf>
> 
> Is there a reason the joinExisting would work, but not the joinNew? I
> thought joinNew would be able to aggregate anything, since your creating a
> new variable to aggregate them with.
> 
> I apologize if this is something simple that I am just missing, I am just
> starting to get my bearings with this.
> 
> Thanks,
> 
> Matt
> 
> 
> address@hidden> wrote:
> 
> > Lansing,
> >
> > Below are a few of the files I am trying to aggregate.
> >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20070530_20070906.nc.html
> >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20070906_20071107.nc.html
> >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20080807_20081204.nc.html
> >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20100803_20101117.nc.html
> >
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20101117_20110318.nc.html
> >
> > Here is the ncml for the joinNew I've been fiddling around with:
> >
> > <?xml version="1.0" encoding="UTF-8"?>
> > <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> >  <!-- <variable name="time" type="int">
> > <attribute name="units" type="string" value="seconds since 1970-01-01
> > 00:00:00"/>
> >  <attribute name="_CoordinateAxisType" value="Time" />
> >   </variable>-->
> >   <aggregation dimName="timeagg" type="joinNew">
> >   <variableAgg name="A"/>
> > <netcdf id="1" location="
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20070530_20070906.nc
> > "/>
> > <netcdf id="2" location="
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20070906_20071107.nc
> > "/>
> > <netcdf id="3" location="
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20080807_20081204.nc
> > "/>
> > <netcdf id="4" location="
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20100803_20101117.nc
> > "/>
> > <netcdf id="5" location="
> > http://data.nodc.noaa.gov/thredds/dodsC/nmsp/bml/DOU001_000_20101117_20110318.nc
> > "/>
> >   </aggregation>
> >  </netcdf>
> >
> > For some reason the Aggregation Variables are not being listed... Am I
> > completely missing something?
> > I truly appreciate your assistance in this matter. :)
> >
> > Thanks,
> >
> > Matt
> >
> >
> > address@hidden> wrote:
> >
> >> Matt,
> >>
> >> If you send me a few of the files, I can set up the aggregation and see
> >> if there are any pitfalls in the data.  It's not always straightforward.
> >>
> >> -Lansing
> >>
> >> > Lansing,
> >> >
> >> > The times "should be" unified, but I should look into that. Probably, to
> >> > avoid complications a JoinNew would suffice in this case.
> >> >
> >> > Matt
> >> >
> >> >
> >> > address@hidden> wrote:
> >> >
> >> > > Li,
> >> > >
> >> > > Are the times in the data sets unified (i.e., all the same), or are
> >> they
> >> > > interlaced?  In other words, if you were to set up a single time axis
> >> for
> >> > > all of the buoys, would the data line up on the axis, or would it be
> >> > > scattered around?  If the times are all the same, then you should be
> >> able
> >> > > to do a simple Union as described here:
> >> > >
> >> > >
> >> http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Aggregation.html
> >> > >
> >> > > Otherwise, you will need to to a JoinNew, wherein you declare a new
> >> > > dimension as the aggregation dimension.  This is described on the
> >> same page.
> >> > >
> >> > > -Lansing
> >> > >
> >> > > > Hi Lansing,
> >> > > >
> >> > > > I think there are group of buoys that are reporting data
> >> simultaneously.
> >> > > > The data sets should be overlapping in space.
> >> > > > I included Matt, who is the data officer for this data set, in our
> >> > > > conversation.
> >> > > >
> >> > > > Hi Matt,
> >> > > > Please correct me if the description about the data is not right.
> >> > > >
> >> > > > Thanks a lot,
> >> > > > Li
> >> > > >
> >> > > >
> >> > > > address@hidden> wrote:
> >> > > >
> >> > > > > Li,
> >> > > > >
> >> > > > > Are the data sets from the buoys overlapping in time, space, or
> >> both?
> >> > > > >  That is, do you have many buoys that are reporting data
> >> > > simultaneously, or
> >> > > > > are the individual data set series generated by each buoy
> >> temporally
> >> > > > > distinct?
> >> > > > >
> >> > > > > Also, are the nc files generated by the buoys, or have they been
> >> > > generated
> >> > > > > through some post-processing from a raw data set?
> >> > > > >
> >> > > > > If the files are not too large, feel free to send me a few
> >> > > representative
> >> > > > > files to work with locally, and I will try to set up an
> >> aggregation.
> >> > > > >
> >> > > > > Regards,
> >> > > > >   Lansing Madry
> >> > > > >   Unidata
> >> > > > >   Boulder, Colorado
> >> > > > >
> >> > > > > > Dear Sir,
> >> > > > > >
> >> > > > > > I am trying to reach the experts about NcML aggregation.
> >> > > > > >
> >> > > > > > I have a group of time series buoy nc files. I tried to
> >> aggregate
> >> > > them
> >> > > > > > by time, but failed since each of them holds a time series
> >> itself.
> >> > > > > >
> >> > > > > > It would be great if anyone could advise me on how should I
> >> aggregate
> >> > > > > > these data.
> >> > > > > >
> >> > > > > > Thanks and Regards,
> >> > > > > > Li
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > > > Ticket Details
> >> > > > > ===================
> >> > > > > Ticket ID: HIP-291542
> >> > > > > Department: Support netCDF Java
> >> > > > > Priority: Normal
> >> > > > > Status: Open
> >> > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> > > Ticket Details
> >> > > ===================
> >> > > Ticket ID: HIP-291542
> >> > > Department: Support netCDF Java
> >> > > Priority: Normal
> >> > > Status: Open
> >> > >
> >> > >
> >> >
> >> >
> >> > --
> >> > Mathew Biddle, Oceanographer
> >> > NOAA/NODC UMD/ESSIC/CICS E/OC1
> >> > 1315 East-West Hwy
> >> > Silver Spring, MD 20910-3282
> >> > Phone: (301) 713-3272 X163
> >> > Email: address@hidden
> >> > http://www.nodc.noaa.gov/
> >> > http://www.facebook.com/noaa.nodc
> >> >
> >> >
> >>
> >>
> >> Ticket Details
> >> ===================
> >> Ticket ID: HIP-291542
> >> Department: Support netCDF Java
> >> Priority: Normal
> >> Status: Open
> >>
> >>
> >
> >
> > --
> > Mathew Biddle, Oceanographer
> > NOAA/NODC UMD/ESSIC/CICS E/OC1
> > 1315 East-West Hwy
> > Silver Spring, MD 20910-3282
> > Phone: (301) 713-3272 X163
> > Email: address@hidden
> > http://www.nodc.noaa.gov/
> > http://www.facebook.com/noaa.nodc
> >
> 
> 
> 
> --
> Mathew Biddle, Oceanographer
> NOAA/NODC UMD/ESSIC/CICS E/OC1
> 1315 East-West Hwy
> Silver Spring, MD 20910-3282
> Phone: (301) 713-3272 X163
> Email: address@hidden
> http://www.nodc.noaa.gov/
> http://www.facebook.com/noaa.nodc
> 
> 


Ticket Details
===================
Ticket ID: HIP-291542
Department: Support netCDF Java
Priority: Normal
Status: Open