NEXRAD ITR Collaborative Research Project (with University of Iowa, Princeton
University, and National Climatic Data Center)
Progress Report: August 2006-July 2007
Mohan Ramamurthy and Jeff Weber, Unidata
Unidata’s activities related to the NEXRAD ITR project during the past
year involved work in the following areas: 1) Provision of NEXRAD Level II
radar to universities participating in the project; 2) Exploration of methodologies
for distribution of metadata associated with Level II data as a new feed in
the Unidata Internet Data Distribution system; 3) Exploration and deployment
of NEXRAD-ITR MapServer; 4) New methods for accessing and visualizing NEXRAD
Level II and Level III data. Although all of these activities are relevant
to the NEXRAD-ITR project and are of benefit to the hydrology community, some
of them were leveraged using other resources within Unidata.
1. Provision of NEXRAD Level II radar data
Unidata continues to supply the University of Iowa with a top-level feed for
Level II radar data using the LDM/IDD. Originally the data feed included only10
radars of interest but during the last year that number was expanded to 30.
The eventual goal is to provide and process Level II data from all 158 WSR-88D
sites. The topology for data distribution to University of Iowa was changed
to establish a more reliable connection. To insure reliable and timely ingest
of the Level II radar data at the University of Iowa, we moved their site higher
up on the IDD topology, feeding it from the University of Wisconsin (a top-level
relay node). Previously, Iowa had been ingesting from Iowa State, which was
in a lower position in the IDD fan out. This relocation to a higher level position
allows the University of Iowa to request any or all of the 158 sites included
in the Level II feed with greater reliability. With the new topology, the LDM/IDD
has performed well in delivering the data in a timely manner. Since Unidata
also hosts a top level feed for Level II radar data, we plan to implement the
algorithms for metadata generation to accompany the installation and operational
use of the University of Minnesota MapServer (described below).
2. Insertion of Level II metadata
Researchers at the University of Iowa are developing metadata descriptions
as well as the algorithms for metadata generation. During the past year, Jeff
Weber, an associate scientist working on this project, explored the methodologies
for the insertion and distribution of metadata associated with Level II radar
data through a new LDM/IDD feed. The metadata generated from the Level II radar
feed is temporarily being distributed via the experimental (EXP) channel of
the IDD data stream. We are currently working to create a more permanent HYDRO
feed of the data stream for the benefit of the hydrologic community. To that
end, we are recommending file naming conventions to allow for uniqueness, and
therefore ease of access via regular expressions, which are utilized by the
LDM for matching data requests.
Jeff is exploring two options for accessing the NEXRAD Level II metadata, ingesting
the metadata via the LDM/IDD from the University of Iowa, and generating it
locally at Unidata from the Level II feed. In addition to distributing the
rich metadata to the community via the IDD, Jeff is exploring ways to integrate
the resulting metadata into THREDDS catalogs and provide via the THREDDS servers and Mapservers.
Below is a schematic of the process:
The PHP preprocessor parses the URL arguments (extent=us&huc=8&radar=yes)
which in this case means: generate a map for the U.S., show 8-digit USGS hydrologic
unit codes (HUC), and also show the NEXRAD radar locations on the map. The
PHP preprocessor uses this information and generates an appropriate MapServer
configuration file and invokes MapServer and finally serves up the map to the
client. In other words, PHP generates MapServer configuration files dynamically
before it invokes MapServer.
During the past year, we developed and tested the above setup at The University
of Iowa and are now in the process of deploying a duplicate install on one
of Unidata Program Center’s (UPC) machines, enabling the availability of the
MapServer to Unidata’s large user community. The installation of the
University of Minnesota MapServer at Unidata has gone well and we will be testing
the service internally before we make it publicly available. To that end, the
UPC staff are currently investigating performance and scalability issues for
deployment to a larger community.
4. New methods for accessing and visualizing NEXRAD Level II and Level III
data
To enable remote access to NEXRAD Level II data, enhancements were made to
the NetCDF-Java library to read NEXRAD data into a “Common Data Model,” that
has been
developed by Unidata. As a result, the Common Data Model now provides uniform
access, using HTTP and OPeNDAP, to NetCDF, HDF5, GRIB 1 and 2, and NEXRAD datasets.
On the analysis and visualization end, advances were made to Unidata’s
Integrated Data Viewer (IDV) to use the netCDF Java library to read in NEXRAD
Level II and Level III. Using the netCDF Java library, both Level II and Level
III datasets can now be read from local disk as well as from remote THREDDS
servers and visualized by the IDV. With these advancements, archived data from
the National Climatic Data Center can now be downloaded and visualized using
the IDV. In addition, the GEMPAK analysis and display application can also
visualize Level II radar data.
Unidata has continued to collaborate with NCDC regarding their OPeNDAP and
THREDDS catalogs to provide access to the metadata archives at NCDC as well
as the Level II radar data. Additionally, the Common Data Model (CDM), created
for ease of data access in myriad formats and as part of the THREDDS initiative,
has made it possible to access model grids via a web services approach. As
a result, hydrologists in CUAHSI (David Maidment, personal communication) are
now able to access precipitation forecasts from the North American Mesoscale
model (which now uses the WRF-NMM model) over specific hydrologic basins and
watersheds, as well as incorporating the data into GIS clients.
5. Current state
Unidata and University of Iowa are currently transforming the programs that derive the metadata
from the level II radar data (level 4 metadata) and recoding these programs to allow implementing the LDM to generate
and propogate the data to the community and to populate Mapservers at both Unidata and University of Iowa. This
will likely be in place by time of March meeting.