AWIPS II and GEMPAK

Status Report: October 2014 - March 2015

Michael James

Strategic Focus Areas

AWIPS II activities support the following Unidata funding proposal focus areas:

  1. Enable widespread, efficient access to geoscience data

  2. Develop and provide open-source tools for effective use of geoscience data

  3. Provide cyberinfrastructure leadership in data discovery, access, and use

  4. Build, support, and advocate for the diverse geoscience community

Cloud activities

Unidata made available a free and open EDEX data server in the Microsoft Azure cloud for most of 2014 and early 2015. While active, this instance served data to over a dozen universities, including some universities who ran CAVE in classroom computer labs.

We're still unsure how many CAVE clients can simultaneously connect to a single EDEX instance, but with the Azure cloud instance we noticed that bandwidth was of critical importance for usability. Loading the 0.5 degree GFS in the NCP, for example, and the delay in rendering within the client is significant (10-20 sec for 10 frames). This was the reason I turned off the 0.5 and ingested 1.0 degree GFS on Azure and default on 14.2.1 EDEX installs at Universities.

Azure download volume ranged between 500-1000MB daily.

Connecting a dozen clients to an internal EDEX server shows much better performance, which is clear. So far we have reach bandwidth limits but not Qpid message broker pool size limits. So if bandwidth is good it appears that AWIPS II can scale to classroom sizes.

Building RPMs

Branches of the "AWIPS2 Builds" project version 14.4.1 and later have been reorganized into multiple code repositories instead of a single one as before. With this reorganization came changes in the build structure. Unidata has built custom RPMs for AWIPS II 14.2.1 to install at university sites, and would like to continue to build RPM sets for our user base. After updating the Unidata development branch to 14.4.1 the build structure is no longer functional.

  • Raytheon has not to my knowledge made available 14.4.1 RPM build instructions and working build scripts to development organizations.

  • The 14.4.1 branches (master, omaha, ncep, etc.) contain build files and scripts which have not been updated since version 14.2.1, and contain build targets for projects which have been removed from the baseline (such as datadelivery and dataaccess plugins), so the build fails.

  • Manually editing the 14.2.1 build scripts to adapt to 14.4.1 is a trial and error process and with enough edits we can force a build but the RPMs fail on install since they are missing critical executable files. Requests for help from NCEP have still not solved this problem as they themselves also find the build process confusing.

  • 14.4.1 saw the introduction of Groovy feature build files in awips2-builds/build/build.core (Feature.groovy and FeatureParser.groovy added December 2014 by Bryan Kowal), which suggest that Raytheon has moved from an Ant build to using a Groovy plugin in Eclipse. There is no documentation as to how these scripts work.

AWIPS II updates since last status report

  • Moved from java 1.6 to 1.7

  • Removed autostart CAVE and alertviz functionality, so no longer will clients start on login as happens at WFOs.

  • Created a Boulder, Colorado "localization" RPM to investigate what needs to be done in order to change an EDEX server site localization (answer: a lot, more than should be necessary).

  • Updated profile.d scripts to only source AWIPS II python for 'root' and 'awips' users, allowing other software packages like IDV to use system python without conflict.

  • Increased default LDM product queue size from 500M to 1500M after noticing data holes on Azure instance.

  • New grids supported: NAM 5km, CMC, NAM 32km, NAM243, HRRR (very slow rendering), FNMOC (some still in development)

  • Reorganization of satellite postgres database tables to support UNIWISC imagery.

  • Separated the build of shapefile RPMs from other EDEX RPMs, no need to re-build very large shapefile executable file when minor changes are made to EDEX.

  • Enabled NCP-specific data decoders ncpafm, ncscd, nctext, ntrans on EDEX. Noticed some large ingest latency due to dual decoding (by Raytheon 'text' and NCEP 'nctext' plugins).

  • Restarted UPC development branch as tracking remote ncep_14.4.1, folding in previous updates to UPC 14.2.1 orphan branch.

AWIPS II Training Workshop

As of 2014, the AWIPS II training workshop consists entirely of new material, adapted from the Raytheon document and NWS training workshop course material. Topics include

  • Understanding Linux Tools in the AWIPS II Environment
  • AWIPS II System Architecture
  • Install the EDEX Data Server
  • Regular Expressions in AWIPS II
  • The Qpid Message Broker
  • Data and Process Logging
  • Data URIs
  • Data Purging
  • Manual Data Ingest
  • Grid Configuration in EDEX
  • Adding a New Grid to EDEX
  • Site Localization Guide
  • CAVE Localization Perspective
  • CAVE National Centers Perspective
  • Adding a new Shapefile to CAVE

Updated NCP "Overlay" menu to indicate what overlays are currently loaded on screen and allow toggle of overlays on-click with new checkboxes.

Continued support of UNIWISC McIDAS imagery support (1km/4km GOES, GOES E/W composites, Arctic POES composites, Global Rectilinear and Mollweide composites)

Further grid support: CONDUIT, FNMOC and CMC feed partially supported (though CMC GEM model is known to seg fault 14.2.1 grib decoder, uh oh)

Working with a HS student on a college credit project to deploy LDM/EDEX in the Softlayer.com cloud.

Source Code

Moved large files outside of source code repo to allow for upload to Unidata github account (max file/commit pack size of 50M/100M firm)

Size

  • awips2-builds: 6 GB
  • awips2-core 55M
  • awips2-core-foss 360M
  • awips2-foss 212M
  • awips2-build-server 475M
  • external from repos: awipscm 6.2 G
  • Total: 13.3 GB of source code and files to build and deliver AWIPS II to universities

Files and lines of code

  • num of files: 49,262
  • num lines of code: 16,214,405
  • num commits to baseline: 7,842

Ongoing and Planned Activities

  • Deliver Unidata AWIPS II 14.4.1 to universities in spring/summer 2015. Git reorganization of tracking branch and merging will allow Unidata to more easily update our development branch and deliver updates along with remote NCEP/WFO branches.

GEMPAK/NAWIPS

  • GEMPAK now includes the capability to request and display data from an EDEX data server, both on internal networks and remotely (assuming remote tcp access is granted by the EDEX server). Data is returned from the EDEX request JVM via Thrift serialization as a Python object, then converted to a Fortran object for processing and display by GEMPAK programs. The functionality is limited (no NMAP2 data loading, for example), and only certain data types are supported (sfc and uair obs and gridded data), but the EDEX Data Access Framework Python interface allows for all data types to be returned.
  • Created a full GEMPAK grid inventory to display projection, available times and grid parameters.
  • NOAAport grib tables updates December 2014
  • NCEP NAWIPS maps/bounds updates
  • Internal memory increased to handle processing and display of 5km NAM
  • CONDUIT GFS pattern action updates