Cloud Activities

Status Report: Oct 2013 - May 2014

Tom Yoksas, Mike Schmidt, Mohan Ramamurthy, Michael James, Ward Fisher

Strategic Focus Areas

Unidata has a number of projects aimed at evaluating the capabilities and costs of providing data services from Cloud-based systems. These Cloud activities support the following Unidata funding proposal focus areas:

  1. Enable widespread, efficient access to geoscience data

    Unidata's cloud activities are exploring ways in which the community can take advantage of Cloud capabilities for improved access to geoscience data.


  2. Develop and provide open-source tools for effective use of geoscience data

    Efforts are focused on how Unidata's existing open source tools will perform on cloud systems and how to evolve these systems to better take advantage of the features of cloud systems.


  3. Provide cyberinfrastructure leadership in data discovery, access, and use

    Working with international groups


  4. Build, support, and advocate for the diverse geoscience community

    Partnering with agencies and other groups to explore how cloud capabilities can be used to improve the capabilities of the entire geoscience community.


Status of Current Projects

Demonstration System Generating Internet Data Distribution (IDD) Content

Unidata migrated the generation of NEXRAD Level 3 composite products from our motherlode data server to our Amazon EC-2 virtual machine (VM). We also migrated generation of Unidata-Wisconsin datastream image products from a very old Solaris machine housed in the University of Wisconsin (UW) Space Science and Engineering Center (SSEC) Data Center to the same Amazon EC-2 VM. The transition to use of the products created in EC-2 was made on the evening of March 22, 2014.

  • NEXRAD Level 3 Composites

    New 1km resolution national composite GINI images include Digital Hybrid Reflectivity (DHR), Digital Vertically Integrated Liquid (DVL), Enhanced Echo Tops (EET), and Hybrid Hydrometeor Classification (HHC). The generation of legacy products are also maintained: 1km Base Reflectivity (N0R), 2km One Hour Precipitation (N1P), and 4km Storm Total Precipitation (NTP). These products are created every five minutes and are available on the Unidata IDD FNEXRAD feed. More detailed product descriptions, example images, and LDM pattern actions can be found on the Unidata IDD Radar Products page.

  • Unidata-Wisconsin Satellite Image Sectors

    As part of the migration to use of Amazon EC-2 resources, we took the opportunity to enhance the product offerings in the Unidata-Wisconsin (IDD feedtype UNIWISC (aka MCIDAS)) datastream. The following represents the set of UNIWISC products as of March 22, 2014: 

    Product                                         Res   Form   Status          
----------------------------------------------+------+-----+-----------------
GOES-East Visible                               4 km  PNGA   existing product
GOES-East Shortwave Infrared                    4 km  PNGA   existing product
GOES-East Water Vapor                           4 km  PNGA   existing product
GOES-East Thermal Infrared                      4 km  PNGA   existing product
GOES-East Longwave Infrared (CO2)               4 km  PNGA   existing product 
GOES-West Visible                               4 km  PNGA   existing product
GOES-West Shortware Infrared                    4 km  PNGA   existing product
GOES-West Water Vapor                           4 km  PNGA   existing product
GOES-West Thermal Infrared                      4 km  PNGA   existing product
GOES-West Longwave Infrared (CO2)               4 km  PNGA   existing product
Antarctic Thermal Infrared composite            5 km  PNGA   existing product
Manually digitized radar                       10 km  PNGA   existing product

Global Mollweide Water Vapor composite         30 km  PNGA   higher resolution
Global Mollweide Thermal Infrared composite    30 km  PNGA   higher resolution
GOES-East HiRes Visible                         1 km  PNGA   new product
GOES-East/West NH Visible composite            10 km  PNGA   new product
GOES-East/West NH Shortwave Infrared composite 10 km  PNGA   new product
GOES-East/West NH Water Vapor composite        10 km  PNGA   new product
GOES-East/West NH Thermal Infrared composite   10 km  PNGA   new product
GOES-East/West NH Longwave Infrared composite  10 km  PNGA   new product
Arctic Visible composite                        4 km  PNGA   new product
Arctic Shortwave Infrared composite             4 km  PNGA   new product
Arctic Water Vapor composite                    4 km  PNGA   new product
Arctic Thermal Infrared composite               4 km  PNGA   new product
Arctic Longwave Infrared composite              4 km  PNGA   new product
Antarctic Visible composite                     5 km  PNGA   new product
Antarctic Water Vapor composite                 5 km  PNGA   new product
Global Rectilinear Water Vapor composite       20 km  PNGA   new product
Global Rectilinear Thermal Infrared composite  20 km  PNGA   new product
    Notes:
    • update cycles vary by product
    • all images are distribured in a PNG-compressed McIDAS AREA (PNGA) format
    • Arctic and Antarctic composites are provided by the UW/SSEC Antarctic Meteorological Research Center (AMRC)
    • Global WV and IR composites are provided by the UW/SSEC Data Center
    • Detailed information including sample displays of these satellite image sectors can be found on the Unidata-Wisconsin Datastream Satellite Imagery page.

Comment:

The most recent Amazon billing for the EC-2 VM suggest that operating these product generation services will cost on the order of $350/month. This estimate includes the cost for the VM itself, and the movement of the products out of EC-2 by LDM transfers to accumulators for the top level IDD relay cluster, idd.unidata.ucar.edu, operated by Unidata.

 

Application Streaming of IDV: Microsoft Research Award of Azure Resources

Overview

This project is evaluating application streaming as a strategy for making the IDV available to a new generation of users and computing platforms. It is using the Microsoft Azure cloud platform to look at delivering cloud-based IDV-as-a-service instances to our user community on an as-needed basis. The result will be a better understanding of how the IDV works in cloud environments and any changes that might improve that performance.

This project also serves as a pilot program; with it we will further develop expertise related to cloud computing and application streaming. This will allow us to extend cloud-based software offerings beyond the IDV to other Unidata projects.

Issues

  • How best to adapt mouse-driven interfaces to a touch-based interface, while minimizing the need to re-engineer any part of the software package.
  • Evaluation of bandwidth requirements for acceptable IDV use.
  • How to make this transition seamless and painless to our user community.
  • Evaluate the extent to which we can use "off-the-shelf" technology and under what circumstances do we need to create our own protocols and packages.

Current Status

Currently, we are able to instantiate cloud-based IDV instances, which are then streamed via existing remote-desktop protocols to iOS devices. Nothing in the existing technology limits this to iOS devices, however; those are simply the devices on hand for testing.

The next step will be to automate the creation and provisioning of these IDV instances, so that they might be generated dynamically and provided to the end user.


Open Weather and Climate Service: “Servers & NCEP” Experiment

The goal of the “Servers @ NCEP” project is to develop a time-limited (approximately 2 year), open, community-operated experimental prototype capability to receive unique, high-volume Numerical Weather Prediction (NWP) data from the National Centers for Environmental Prediction (NCEP) supercomputers in Reston, VA sufficient to enable new value-added processing by the community that was otherwise not possible.

Unidata is participating in the experiment, and will be installing computing equipment in the community data center co-located with NCEP. Our goal is to use the experimental setup to investigate the feasibility of data subsetting and server-side processing techniques that could enable us to deliver a wider range of data to university researchers and educators.

Partners in the experiment are in the process of establishing the necessary business, legal, and technical procedures and instruments. (Unidata is currently the only non-commercial organization that has expressed its intent to participate in the experiment.) When access to the NCEP data becomes available, the experiment will establish a high-volume data feed; the project partners will work with NCEP to determine which high-volume datasets are of interest in order to enable experiments and use by the project partners.

A slightly more detailed description of the project's history and progress is available on Unidata's OWCS Project Page.


Cloning the IDD Data Server (Motherlode) Capabilities

This project has two main goals. First, it will focus on the core datastreams available on the IDD and gather the minimal set of configuration needed by the LDM, TDS, and ADDE to distribute and provide data services for that data. Second, it will package the configuration files needed in a way that can easily be implemented at University sites and on systems running on cloud platforms.