[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[THREDDS #XLF-986148]: aggregating ensembles



youll have to do time first, since joins are always on the outer dimension.

these files have to be highly homogeneous - exact same coords, or else you will 
get wrong results. id probably write a program to double check that is true.

eventually i will write some agg code that will do that for you - and do the 
right thing is the coordinates dont match. for now, were stuck with these 
homgeneity constraints.

> Thanks.  I understand about the unzipping - hopefully they can provide
> datasets that are not zipped.
> 
> Aggregating by time will be important to get a view of the entire 21st
> century, not just individual decades.  If you could give a sample of
> what that would look like, I'd appreciate that.  In particular whether
> it's better to aggregate by time first and then ensemble, or ensemble
> then time.
> 
> Don
> 
> On 6/10/11 3:45 PM, Unidata THREDDS Support wrote:
> > You can generalize this:
> >
> > <?xml version='1.0' encoding='UTF-8'?>
> > <netcdf xmlns='http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2'>
> >    <variable name='ens' type='String' shape='ens'>
> >      <attribute name='long_name' value='ensemble coordinate' />
> >      <attribute name='_CoordinateAxisType' value='Ensemble' />
> >    </variable>
> >    <aggregation dimName='ens' type='joinNew'>
> >      <variableAgg name='tasmax'/>
> >      <netcdf 
> > location='E:/work/dmurray/A1B_HadCM3Q3_DM_25km_2001-2010_tasmax.nc.gz' 
> > coordValue='HadCM'/>
> >      <netcdf 
> > location='E:/work/dmurray/A1B_ECHAM5-r3_DM_25km_2001-2010_tasmax.nc.gz' 
> > coordValue='ECHAM5'/>
> >    </aggregation>
> > </netcdf>
> >
> > I would not do the time aggregation if possible, as i think it will slow 
> > things down. CDM will need to uncompressed the files which will be slow the 
> > first time. So it would be better to work on subsets, to minimize space.
> >
> >> So, for the site below, I would think you'd want to aggregate on the
> >> ensemble dimension for:
> >>
> >> *25km_2021-2030_tasmax.nc
> >>
> >> then you'd probably want to aggreggate across time (joinExisting) on
> >> time for:
> >>
> >> *_tasmax.nc
> >>
> >> of those ensembles.  More examples under a TDS are at:
> >>
> >> http://clima-dods.ictp.it/thredds/catalog/aap/ENSEMBLES_AMMA/catalog.html
> >>
> >> with a particular run at:
> >>
> >> http://clima-dods.ictp.it/thredds/catalog/aap/ENSEMBLES_AMMA/ERAIN/Months/catalog.html
> >>
> >> Don
> >>
> >> On 6/6/11 6:32 PM, Unidata THREDDS Support wrote:
> >>> ok, ive looked at some random files. can you give me an example of a list 
> >>> of files for different models that they want to aggregate? they will have 
> >>> to be on the exact same grid, is that true?
> >>>
> >>>> They are netCDF files.  Samples are at:
> >>>>
> >>>> http://clima-dods.ictp.it/repository/entry/show/Top/netCDF/d2/ENSEMBLES/A1B?max=500&entryid=509ba64f-eea3-4d6a-9fee-be908f770a77
> >>>>
> >>>> Don
> >>>>
> >>>> On 6/6/11 9:57 AM, Unidata THREDDS Support wrote:
> >>>>> what kind of files are these? We are working on some special mechanisms 
> >>>>> for GRIB2. Otherwise, ensemble support is pretty sketchy.
> >>>>>
> >>>>>
> >>>>>> Ethan/John-
> >>>>>>
> >>>>>> At the ICTP workshop this past week, the question came up about
> >>>>>> aggregating files into an ensemble.  I think this can be done through
> >>>>>> NcML (and thus TDS/IDV/RAMADDA), but I couldn't find an example in the
> >>>>>> docs (although I didn't look very closely).  They have different model
> >>>>>> runs that cover the same geographic/temporal domains, but they want to
> >>>>>> be able to view the files as one dataset in the IDV to be able to do
> >>>>>> ensemble means, etc.  This is akin to the SREF data accessible under
> >>>>>> this catalog:
> >>>>>>
> >>>>>> http://motherlode.ucar.edu/thredds/catalog/model/NCEP/SREF/CONUS_40km/pgrb_biasc/catalog.xml
> >>>>>>
> >>>>>> where each file contains the data for a particular ensemble member.  If
> >>>>>> I wanted to aggregate these files to look like a single dataset with an
> >>>>>> ensemble dimension, what would the NcML look like?
> >>>>>>
> >>>>>> Thanks for your help.
> >>>>>>
> >>>>>> Don
> >>>>>> --
> >>>>>> Don Murray
> >>>>>> NOAA/ESRL/PSD and CIRES
> >>>>>> 303-497-3596
> >>>>>> http://www.esrl.noaa.gov/psd/people/don.murray/
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> Ticket Details
> >>>>> ===================
> >>>>> Ticket ID: XLF-986148
> >>>>> Department: Support THREDDS
> >>>>> Priority: High
> >>>>> Status: Open
> >>>>>
> >>>>
> >>>> --
> >>>> Don Murray
> >>>> NOAA/ESRL/PSD and CIRES
> >>>> 303-497-3596
> >>>> http://www.esrl.noaa.gov/psd/people/don.murray/
> >>>>
> >>>>
> >>>
> >>>
> >>> Ticket Details
> >>> ===================
> >>> Ticket ID: XLF-986148
> >>> Department: Support THREDDS
> >>> Priority: High
> >>> Status: Open
> >>>
> >>
> >> --
> >> Don Murray
> >> NOAA/ESRL/PSD and CIRES
> >> 303-497-3596
> >> http://www.esrl.noaa.gov/psd/people/don.murray/
> >>
> >>
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: XLF-986148
> > Department: Support THREDDS
> > Priority: High
> > Status: Open
> >
> 
> --
> Don Murray
> NOAA/ESRL/PSD and CIRES
> 303-497-3596
> http://www.esrl.noaa.gov/psd/people/don.murray/
> 
> 


Ticket Details
===================
Ticket ID: XLF-986148
Department: Support THREDDS
Priority: High
Status: Closed