NEXRAD ITR Collaborative Research Project (with University of Iowa, Princeton University, and National Climatic Data Center)
Progress Report: August 2006-March 2007
Mohan Ramamurthy and Jeff Weber, Unidata

Unidata’s activities related to the NEXRAD ITR project during the past year involved work in the following areas: 1) Provision of NEXRAD Level II radar to universities participating in the project; 2) Exploration of methodologies for distribution of metadata associated with Level II data as a new feed in the Unidata Internet Data Distribution system; 3) Exploration and deployment of NEXRAD-ITR MapServer; 4) New methods for accessing and visualizing NEXRAD Level II and Level III data. Although all of these activities are relevant to the NEXRAD-ITR project and are of benefit to the hydrology community, some of them were leveraged using other resources within Unidata.

1. Provision of NEXRAD Level II radar data
Unidata continues to supply the University of Iowa with a top-level feed for Level II radar data using the LDM/IDD. Originally the data feed included only10 radars of interest but during the last year that number was expanded to 30. The eventual goal is to provide and process Level II data from all 158 WSR-88D sites. The topology for data distribution to University of Iowa was changed to establish a more reliable connection. To insure reliable and timely ingest of the Level II radar data at the University of Iowa, we moved their site higher up on the IDD topology, feeding it from the University of Wisconsin (a top-level relay node). Previously, Iowa had been ingesting from Iowa State, which was in a lower position in the IDD fan out. This relocation to a higher level position allows the University of Iowa to request any or all of the 158 sites included in the Level II feed with greater reliability. With the new topology, the LDM/IDD has performed well in delivering the data in a timely manner. Since Unidata also hosts a top level feed for Level II radar data, we plan to implement the algorithms for metadata generation to accompany the installation and operational use of the University of Minnesota MapServer (described below).

2. Insertion of Level II metadata
Researchers at the University of Iowa are developing metadata descriptions as well as the algorithms for metadata generation. During the past year, Jeff Weber, an associate scientist working on this project, explored the methodologies for the insertion and distribution of metadata associated with Level II radar data through a new LDM/IDD feed. The metadata generated from the Level II radar feed is temporarily being distributed via the experimental (EXP) channel of the IDD data stream. We are currently working to create a more permanent HYDRO feed of the data stream for the benefit of the hydrologic community. To that end, we are recommending file naming conventions to allow for uniqueness, and therefore ease of access via regular expressions, which are utilized by the LDM for matching data requests.
Jeff is exploring two options for accessing the NEXRAD Level II metadata, ingesting the metadata via the LDM/IDD from the University of Iowa, and generating it locally at Unidata from the Level II feed. In addition to distributing the rich metadata to the community via the IDD, Jeff is exploring ways to integrate the resulting metadata into THREDDS catalogs and provide via the THREDDS servers and Mapservers.

3. NEXRAD-ITR Mapserver
Past experience with the hydrologic community indicated they generally use PC-based operating systems, and these are most often Windows machines. The hydrologic community is also strongly tied with GIS applications. With this in mind, it was deemed that a map server would enhance the ease of access and usability for the hydrologic community. To implement a map server, we explored commercial as well as open-source web-based (http protocol) mapservers for displaying some of the data associated with this project, and settled on the open-source University of Minnesota MapServer (http://mapserver.gis.umn.edu). It is flexible,, easy to use, free, and runs on many platforms. MapServer’s output is controlled via a template.

Here is a brief description of the process of map generation using the MapServer and its potential use in this project: A configuration file controls the look-and-feel (line widths, colors, fonts, layers, etc) of maps generated by the server. When an http client accesses a webserver with a MapServer installation, the webserver’s CGI engine spawns MapSever, which reads the configuration file, generates the map, and the webserver serves up the map to the client. We use MapSever in conjunction with the Apache webserver and PHP preprocessing as follows. Clients access our mapserver through a URL such as
www.mapserver.uiowa.edu/nexraditr/map?extent=us&huc=8&radar=yes...

The PHP preprocessor parses the URL arguments (extent=us&huc=8&radar=yes) which in this case means: generate a map for the U.S., show 8-digit USGS hydrologic unit codes (HUC), and also show the NEXRAD radar locations on the map. The PHP preprocessor uses this information and generates an appropriate MapServer configuration file and invokes MapServer and finally serves up the map to the client. In other words, PHP generates MapServer configuration files dynamically before it invokes MapServer.
During the past year, we developed and tested the above setup at The University of Iowa and are now in the process of deploying a duplicate install on one of Unidata Program Center’s (UPC) machines, enabling the availability of the MapServer to Unidata’s large user community. The installation of the University of Minnesota MapServer at Unidata has gone well and we will be testing the service internally before we make it publicly available. To that end, the UPC staff are currently investigating performance and scalability issues for deployment to a larger community.

4. New methods for accessing and visualizing NEXRAD Level II and Level III data
To enable remote access to NEXRAD Level II data, enhancements were made to the NetCDF-Java library to read NEXRAD data into a “Common Data Model,” that has been
developed by Unidata. As a result, the Common Data Model now provides uniform access, using HTTP and OPeNDAP, to NetCDF, HDF5, GRIB 1 and 2, and NEXRAD datasets.
On the analysis and visualization end, advances were made to Unidata’s Integrated Data Viewer (IDV) to use the netCDF Java library to read in NEXRAD Level II and Level III. Using the netCDF Java library, both Level II and Level III datasets can now be read from local disk as well as from remote THREDDS servers and visualized by the IDV. With these advancements, archived data from the National Climatic Data Center can now be downloaded and visualized using the IDV. In addition, the GEMPAK analysis and display application can also visualize Level II radar data.
Unidata has continued to collaborate with NCDC regarding their OPeNDAP and THREDDS catalogs to provide access to the metadata archives at NCDC as well as the Level II radar data. Additionally, the Common Data Model (CDM), created for ease of data access in myriad formats and as part of the THREDDS initiative, has made it possible to access model grids via a web services approach. As a result, hydrologists in CUAHSI (David Maidment, personal communication) are now able to access precipitation forecasts from the North American Mesoscale model (which now uses the WRF-NMM model) over specific hydrologic basins and watersheds, as well as incorporating the data into GIS clients.

5. Current state
Unidata and University of Iowa are currently transforming the programs that derive the metadata from the level II radar data (level 4 metadata) and recoding these programs to allow implementing the LDM to generate and propogate the data to the community and to populate Mapservers at both Unidata and University of Iowa. This will likely be in place by time of March meeting.