NEXRAD ITR Collaborative Research Project (with University of Iowa, Princeton
University, and National Climatic Data Center)
Progress Report: September 2006-April 2007
Mohan Ramamurthy and Jeff Weber, Unidata
Unidata’s activities related to the NEXRAD ITR project during the past
year involved work in the following areas: 1) Provision of NEXRAD Level II
radar to universities participating in the project; 2) Exploration of methodologies
for distribution of metadata associated with Level II data as a new feed in
the Unidata Internet Data Distribution system; 3) Exploration and deployment
of NEXRAD-ITR MapServer; 4) New methods for accessing and visualizing NEXRAD
Level II and Level III data. Although all of these activities are relevant
to the NEXRAD-ITR project and are of benefit to the hydrology community, some
of them were leveraged using other resources within Unidata.
1. Provision of NEXRAD Level II radar data
Unidata continues to supply the University of Iowa with a top-level feed for
Level II radar data using the LDM/IDD. Originally the data feed included only10
radars of interest but during the last year that number was expanded to 30.
The eventual goal is to provide and process Level II data from all 158 WSR-88D
sites. The topology for data distribution to University of Iowa was changed
to establish a more reliable connection. To insure reliable and timely ingest
of the Level II radar data at the University of Iowa, we moved their site higher
up on the IDD topology, feeding it from the University of Wisconsin (a top-level
relay node). Previously, Iowa had been ingesting from Iowa State, which was
in a lower position in the IDD fan out. This relocation to a higher level position
allows the University of Iowa to request any or all of the 158 sites included
in the Level II feed with greater reliability. With the new topology, the LDM/IDD
has performed well in delivering the data in a timely manner. Since Unidata
also hosts a top level feed for Level II radar data, we plan to implement the
algorithms for metadata generation to accompany the installation and operational
use of the University of Minnesota MapServer (described below).
2. Insertion of Level II metadata
Researchers at the University of Iowa are developing metadata descriptions
as well as the algorithms for metadata generation. During the past year, Jeff
Weber, an associate scientist working on this project, explored the methodologies
for the insertion and distribution of metadata associated with Level II radar
data through a new LDM/IDD feed. The metadata generated from the Level II radar
feed is temporarily being distributed via the experimental (EXP) channel of
the IDD data stream. We are currently working to create a more permanent HYDRO
feed of the data stream for the benefit of the hydrologic community. To that
end, we are recommending file naming conventions to allow for uniqueness, and
therefore ease of access via regular expressions, which are utilized by the
LDM for matching data requests.
Jeff is exploring two options for accessing the NEXRAD Level II metadata, ingesting
the metadata via the LDM/IDD from the University of Iowa, and generating it
locally at Unidata from the Level II feed. In addition to distributing the
rich metadata to the community via the IDD, Jeff is exploring ways to integrate
the resulting metadata into THREDDS catalogs and provide via the THREDDS servers and Mapservers.
3. NEXRAD-ITR Mapserver
Past experience with the hydrologic community indicated they generally use
PC-based operating systems, and these are most often Windows machines. The
hydrologic community is also strongly tied with GIS applications. With this
in mind, it was deemed that a map server would enhance the ease of access and
usability for the hydrologic community. To implement a map server, we explored
commercial as well as open-source web-based (http protocol) mapservers for
displaying some of the data associated with this project, and settled on the
open-source University of Minnesota MapServer (http://mapserver.gis.umn.edu).
It is flexible,, easy to use, free, and runs on many platforms. MapServer’s
output is controlled via a template.
Here is a brief description of the process of map generation using the MapServer
and its potential use in this project: A configuration file controls the look-and-feel
(line widths, colors, fonts, layers, etc) of maps generated by the server.
When an http client accesses a webserver with a MapServer installation, the
webserver’s
CGI engine spawns MapSever, which reads the configuration file, generates the
map, and the webserver serves up the map to the client. We use MapSever in
conjunction with the Apache webserver and PHP preprocessing as follows. Clients
access our mapserver through a URL such as
www.mapserver.uiowa.edu/nexraditr/map?extent=us&huc=8&radar=yes...
The PHP preprocessor parses the URL arguments (extent=us&huc=8&radar=yes)
which in this case means: generate a map for the U.S., show 8-digit USGS hydrologic
unit codes (HUC), and also show the NEXRAD radar locations on the map. The
PHP preprocessor uses this information and generates an appropriate MapServer
configuration file and invokes MapServer and finally serves up the map to the
client. In other words, PHP generates MapServer configuration files dynamically
before it invokes MapServer.
During the past year, we developed and tested the above setup at The University
of Iowa and are now in the process of deploying a duplicate install on one
of Unidata Program Center’s (UPC) machines, enabling the availability of the
MapServer to Unidata’s large user community. The installation of the
University of Minnesota MapServer at Unidata has gone well and we will be testing
the service internally before we make it publicly available. To that end, the
UPC staff are currently investigating performance and scalability issues for
deployment to a larger community.
4. New methods for accessing and visualizing NEXRAD Level II and Level III
data
To enable remote access to NEXRAD Level II data, enhancements were made to
the NetCDF-Java library to read NEXRAD data into a “Common Data Model,” that
has been
developed by Unidata. As a result, the Common Data Model now provides uniform
access, using HTTP and OPeNDAP, to NetCDF, HDF5, GRIB 1 and 2, and NEXRAD datasets.
On the analysis and visualization end, advances were made to Unidata’s
Integrated Data Viewer (IDV) to use the netCDF Java library to read in NEXRAD
Level II and Level III. Using the netCDF Java library, both Level II and Level
III datasets can now be read from local disk as well as from remote THREDDS
servers and visualized by the IDV. With these advancements, archived data from
the National Climatic Data Center can now be downloaded and visualized using
the IDV. In addition, the GEMPAK analysis and display application can also
visualize Level II radar data.
Unidata has continued to collaborate with NCDC regarding their OPeNDAP and
THREDDS catalogs to provide access to the metadata archives at NCDC as well
as the Level II radar data. Additionally, the Common Data Model (CDM), created
for ease of data access in myriad formats and as part of the THREDDS initiative,
has made it possible to access model grids via a web services approach. As
a result, hydrologists in CUAHSI (David Maidment, personal communication) are
now able to access precipitation forecasts from the North American Mesoscale
model (which now uses the WRF-NMM model) over specific hydrologic basins and
watersheds, as well as incorporating the data into GIS clients.
5. Current state
Unidata and University of Iowa have transformed the programs that derive the metadata
from the level II radar data (level 4 metadata) and recoded these programs to allow implementing the LDM to generate
and propogate the data to the community and to populate a Mapserver at the University of Iowa. The University of Iowa is now
ingesting the level 4 metadata via the IDD/LDM as is Drexel University. Iowa is testing the feed currently to insure the
products being derived via pqact and the LDM are matching the metadata being derived locally.
Below is a schematic of the process:
The NEXRAD4 feed contains metadata about each radar sites scan as well as compressed level2 data. The volume of the feed is therefore intrinsically tied to the volume of the level2 feed. The volume we are currently seeing for the NEXRAD4 metadata and compressed level2 radar data is ranging around 300-> 400 MB/Hour. This volume is based on generating the data for all radar sites in the level2 feed. The toolset for viewing level IV NEXRAD meta-data is Hydro_Nexrad, developed at University of Iowa, and it relies on a MapServer for mapping utilities. Unidata is exploring how the IDV may be able to play a role as a client for displaying the NEXRAD4 metadata.