[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[THREDDS #LKY-813484]: Missing variable in aggregation

Hi Jonathan,

Jonathan Wilkins wrote:
> Just to clarify things, do you mean that:
> - this should work with these two level agregations but this is not sure
> for more levels ?
> or
> - this is not reliable when repeating this kind of trick for more than
> one file or with a large number of agregated run/time step files ?

I meant the latter, that it might not scale well as the number of time steps 
increased.

However, the former might also be an issue. We haven't had a lot of use cases 
for nested aggregations beyond two levels so that hasn't been tested very 
thoroughly.
 
> So I think I should wait for a later version of Thredds to consider
> these kind of data.
> 
> I've seen the great work done with FMRC and would be happy to use them.
> 
> The facts are that I can't use them at present and I need to generate
> the datasets with ncml as
> we have many use cases to support, wich are:
> 
> + The different ways data files are archived (stored):
> - one file per run
> - one file per time step
> - one file per run and group of variables
> - one file per time step and group of variables
> + The different ways we want data to be presented on the Thredds server:
> - a file view
> - a run view
> - a full time serie / best estimate view (with data already stored as
> best estimate)
> - a 2D view (run + time)
> 
> I know these views can now be done using FMRC, the problem is the way
> data is archived, especially to agregate
> time steps and/or "group of variables" files.

If your data for a single model is all stored as "one file per run" or is all 
stored as "one file per time step", I think the FMRC aggregation should work. 
I'm not sure an FMRC aggregation could deal with dataset collections where 
different variables are split into separate files. I'll have to look into that.

> I didn't succeed to put union agregation in FMRC and don't know how runs
> could be reconstituted (for run and 2D views)
> when archived in  a time step fashion (having the run date somewhere in
> the files name or somewhere else in the the files path)

If you want to send us information on what you've tried for FMRCs and more 
detail on how your model data archives are organized, we could take a look and 
see if we can make any suggestions.

Ethan
 
> So our catalogs are generated with scripts that handle these use cases
> and do quite the same work as Thredds scans or fmrc do.
> 
> Maybe you could tell me how things could be done in a simpliest manner ?
> 
> (Maybe these new questions should be moved in a new thread ?)
> 
> Thanks for your support.
> 
> Regards
> 
> Jonathan


Ticket Details
===================
Ticket ID: LKY-813484
Department: Support THREDDS
Priority: High
Status: Closed