Hi Dan:
dan.swank wrote:
We have had the NARR on GDS for quite some time :
http://nomads.ncdc.noaa.gov:9091/dods/NCEP_NARR_DAILY
GDS works very well with GRIB data in general.
Our particular goal is to get it on TDS due to its suporior metadata
handling and WCS service.
If it is possible to create hooks that allow TDS to read and import
data/metadata from a source OPeNDAP URL, similar to what the Live Access
Server does... that would be outstanding. May pose an easier task to
implement as well.
this is on our list to do. Can you give us a concrete example of what you would
like to see put into the thredds metadata?
-Dan
Rob Cermak wrote the following on 6/14/2006 6:50 PM:
Dan,
The Grads GDS OPeNDAP server is suppose to work with GRIB files. The
large unknown is if TDS can then aggregate the data across the Grads GDS
server. To help answer that is to see if TDS can aggregate across maybe
one of the NOMADs GFS servers? I'm pretty sure they are using the Grads
GDS server.
http://www.iges.org/grads/gds/
Rob
On Tue, June 13, 2006 2:28 pm, dan.swank wrote:
Ethan:
Understood. consider this a wish-list then :)
-Dan
Ethan Davis wrote the following on 6/13/2006 6:22 PM:
Hi Dan,
Evidently, aggregation of GRIB data may not be working. I thought since
GRIB files are read by the netCDF-java library it would just work but I
just heard that there may be some issues with aggregating GRIB involving
how the underlying code writes indexes into the GRIB files. We'll have
to wait till John is back from vacation for a more detailed
answer/explanation.
Sorry for the mistake.
There currently isn't a way to have a single scan element scan a set of
directories.
I believe the underlying netCDF-java code that reads GRIB files should
result in fairly well-formed, CF (I think) compliant netCDF views of the
dataset. Not sure how this would be affected by aggregation.
Ethan
dan.swank wrote:
Ethan:
The NARR is a reanalysis, so it don't have forecast times. I would be
a
simple 03 hr chain (00 hr fct time) spanning 26 years.
See an existing GDS subset aggregation:
http://nomads.ncdc.noaa.gov:9091/dods/NCEP_NARR_DAILY/narr-a_221_tmpprs.subset.info
This will give a sense for the nature of the beast.
The directory structure is set up as such:
http://nomads.ncdc.noaa.gov/data/narr/
Heres the TDS aggregation I set up while experimenting yesterday, on a
non-related dataset:
<dataset name="OceanWinds Test Daily Aggregation"
ID="test/dailyagg" urlPath="test/agg">
<serviceName>allTest</serviceName>
<netcdf
xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
<aggregation dimName="time" type="joinNew">
<variableAgg name="wind" />
<scan dateFormatMark="#yyyyMMdd"
location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1980s/"
suffix=".nc" />
<scan dateFormatMark="#yyyyMMdd"
location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1990s/"
suffix=".nc" />
<scan dateFormatMark="#yyyyMMdd"
location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/2000s/"
suffix=".nc" />
</aggregation>
<variable name="time" orgName="time">
<attribute name="long_name" value="Days"/>
<attribute name="units" value="days since 1987-07-09" />
</variable>
</netcdf>
</dataset>
Would this automatically detect the source of data were GRIB rather
than
NetCDF? and it seems like you need to set the <scan> on each
individual
directory... Doing so the way NARR is set up would create one chunky
configuration file. Is there anyway to have this scan a pattern
(YYYYMM/YYYYMMDD) of directories?
I understand GRIB requires a certain amount of "supplemented" metadata
for complience. Where do you enter this?
-Dan
Ethan Davis wrote the following on 6/13/2006 1:21 PM:
Hi Dan,
Aggregation should work the same for GRIB as for netCDF files. The
issue
would be how your GRIB files are structured and how you want to
aggregate them. Our GRIB files each contain one full model run (all
parameters, all forecast times). We haven't tried aggregating beyond
that.
We have started tracking what is available for the NCEP models on our
server. This is from the TDS 3.8 announcement (with links updated):
We also are now tracking detailed inventory of NCEP model output,
eg:
http://motherlode.ucar.edu:8080/thredds/modelInventory/model/NCEP/NAM/CONUS_12km/
These are all linked from the "collection dataset" pages; For
example from
http://motherlode.ucar.edu:8080/thredds/catalog/model/NCEP/NAM/CONUS_12km/catalog.html
choose the top "CONUS_12_km" link, then choose "Available
Inventory"
Documentation.
One idea for this work is to eventually provide access to alternate
datasets, for instance, a dataset that contains all the 3hr forecast
times from the different runs, or one that contained all the 12Z valid
times from the different runs. Tracking these detailed inventories is
just the first step but aggregation and alternate groupings of the
data
is pretty interesting to think about.
How are your GRIB files structured and what kind of aggregation where
you thinking about?
Ethan
dan.swank wrote:
Hello,
I've been tinkering with the TDS aggregation capabilities and they
work
quite well for NetCDF data, however, I can't seem to find anything in
the docs regarding aggregating GRIB.
We want to get The NARR dataset which we have here at NCDC-NOMADS on
the
TDS. It consists of hundreds of thousands of 50 Mb + GRIB files in a
YYYYMM/YYYYMMDD tree.
Just scouting for a quick answer here:
Is aggregating the NARR GRIB currently feasable with the current
release
of TDS? If so, do any docs exist which could give me a starting
point?
Converting it to NetCDF will not be possible (volume).
--
Dan Swank <dan.swank@xxxxxxxx>
NOMADS Project: Software & Data Management
Contractor - STG, Incorporated
Veach-Baley Federal Building
151 Patton Avenue
Asheville, NC 28801-5001
Phone: 828-271-4007
==============================================================================
To unsubscribe thredds, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================
==============================================================================
To unsubscribe thredds, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================
