Action Items
Committee Members
Acronym List
Members Attending
Kevin Tyle, Univ of Albany, Chair
Michael Baldwin, Purdue University
Martin Baxter, Central Michigan University
Steve Lazarus, Florida Institute of Technology
Sam Ng, Metropolitan State University of Denver
Russ Schumacher, Colorado State University
Student Rep:
Stefan Cecelski, Univ of Maryland
NCEP Reps:
Becky Cosgrove (CONDUIT) (Remote)
Michelle Mainelli (GEMPAK-NAWIPS/AWIPS II) (Remote)
USGS Rep:
Richard Signell (Remote)
Absent:
Anne Case Hanks, University of Louisiana Monroe
Jennifer Collins, University of South Florida
Bart Geerts, University of Wyoming
UPC Staff Attending
| Sean Arms
Julien Chastang Ethan Davis Doug Dirks Ben Domenico Steve Emmerson Ward Fisher Dennis Heimbigner Marcos Hermida Yuan Ho |
Michael James
Lansing Madry Linda Miller Terry Mitchell Mohan Ramamurthy Russ Rew Mike Schmidt Jeff Weber Tom Yoksas |
Meeting agenda
Mohan's slides
Highlights:
Mohan announced the new UCAR Community Programs (UCP) director, Emily CoBabe-Ammann. She took the helm April 15, 2013.
The UPC has had some staffing changes: Departures-Robb Kambic and Tina Campbell. Linda Miller is working at a reduced schedule. Unidata is hiring a student intern this summer.
The DeSouza Award will be presented to Larry Oolman, University of Wyoming, at the Fall Users Committee meeting. Several inches of snow hampered travel to Boulder for Larry to accept the award at this meeting.
The NSF core funding proposal will be submitted to NSF in late May. The Unidata 2018: Transforming GeoScience through Innovative Data Services proposal is a five year, approximately 25 page proposal that provides future directions, and in part, provides a roadmap developed through the strategic planning process with the governing committee's (Policy Committee and Users Committee), surveys, and staff's assistance. The five areas emphasized are: Data Services; Software and Tools; User Support and Training; Community Services; Cyberinfrastructure Leadership.
Rosetta-conversion of ASCII to netCDF for easing use in Unidata software. The project is mostly focused on ACADIS data. Some data folks just want better formatted data and don't necessarily care about netCDF, per se.
Software and support metrics are dominated by netCDF users, however IDV usage continues to rise. Since 2010, usage has doubled.
AWIPS II - Michael and Michelle will provide a presentation later. Broad release to Unidata community by the end of 2013. There was a training course last week for Unidata and COMET folks (coordinated. by Jim Poole, NWS/Kansas City Training Center and Linda Miller, Unidata). See the Photo of participants.
EarthCube - Unidata is actively involved in various EarthCube projects. Unidata organized a workshop on
Forecasting with CSU's Russ Schumacher. The workshop was well attended.
The latest regional workshop was conducted at San Jose State University. Regional workshops serve as a good venue to use for training and assisting several sites and to bring people together to get acquainted and to reach out to help each other.
An IT survey was conducted by UPC where 33 universities responded. The summary of the survey is here.
UPC conducted a science survey where they solicited 21 universities. Summary of that survey is here.
Equipment awards solicitation was sent and - deference to those who offered to share their THREDDS AND RAMADDA servers.
Software training workshops will be conducted this summer. The announcement for registration is posted and a few registrations are trickling in already.
No real recent changes. Current infrastructure cannot handle additional data due to limited LDM queue size. Updates are on hold pending a re-examination of their relationship with NOAA's Web Operations Center which hosts CONDUIT.
RAP 13 km data is now on NOAAPort for all cycles. Does the Unidata community still want this on CONDUIT?
ACTION 1: Tom Yoksas to check with RAP requesters to see whether what's on NOAAPORT is sufficient.
Unidata Configuration - A single yum script installs entire EDEX server. A 'single keystroke installation' should be very helpful to users. Currently there is a huge users manual. Kevin asked about the bottlenecks associated with multi-radar feed. Michael indicated that Raytheon fixed or added an additional thread and that he's done some work with improving this. This is especially important for the national mosaic.
UPC's release to community will be based on Oct/Nov release from NWS.
Michael to create reduced set of AWIPS II docs (1400->800 pages)
Kevin Tyle mentioned errors when CAVE loads. Michael responded that all current efforts have been on resolving data flow issues, not CAVE yet.
National deployment Q4FY15 - Unfortunately another
delay...
8 WFOs have AWIPS II operational. This
generated a bunch of Discrepancy Reports (DRs); they stopped
deployment to address those and will restart deployment in
Q3
Performance issues in NCP were solved, but new ones
appearing with additional data ingest. NCP code will be
baselined by late August, and included in the OB13.6.1
release in November
Functionality to come:
Gempak programs to read from both AWIPS II database (through AWIPS II services) and .gem files. Dave Plummer thinks this should be done in August baseline code delivered to Raytheon. NAWIPS/GEMPAK updates 6.9 (April 2013) and 6.10 (July 2013). These updates will include specific requirements for fire weather, SPC, and NHC.
Ethan asked about OpenDap access. AWIPS II will receive an OpenDap-related data delivery in May. No plans for OpenDap capability for N-AWIPS/GEMPAK.
Radar Mosaic, satellite, solar, ASCAT, Deterministic and Ensemble model data, conventional obs
Kevin asked about the loading of gridded model data. They have a test case where it took ~10 secs to load 20 panels.
The Thin Client is AWIPS-II CAVE running in remote mode connected to an instance of EDEX via an Apache proxy server to access data. Several questions have popped up - Tom Y asked whether the EDEX server can sustain multiple clients at once. Michelle couldn't answer it. Ethan asked why an apache proxy server was needed. Apparently this is a firewall issue. Thin client runs on Linux or windows with appropriate graphics card. Unclear what the max # of QPID connections an EDEX server can support.
ACTION 2: Michelle will ask Steve Schotz the following question:
Mohan asked about using it in the field on an IPad .... Michelle indicated that this was off the radar at the moment.... But might be pursued later.
Data Delivery - NOMADS came up. Becky Cosgrove is the point of contact for NCEP NOMADS issues. Michelle talked about "data discovery" requests.
NCEP baseline funcationality should be in 13.6.1 release in early November 2013.
2014 development? Space weather, high seas, N-Flow, Hazard Mapping, time series
Unidata is a critical partner - weekly telecons take place.
Mohan - is anyone working on putting AWIPS on the cloud? Answer - no.
Ethan asked about OpenDap again. What software is EDEX using to access OpenDap and grab data? Apparently this is not clear and Michael asked for more documentation.
Kevin Tyle:
Keeping pace with NAWIPS releases. Recent Gempak fix allowed gdpvs to work again. Some students starting to use IDL
again, MATLAB in heavy use, as is NCL. A bit of a problem to
support all the different packages, keep groups talking to
each other in terms of data they are using. 5th year of
computer applications class for Juniors; spent time pushing
IDV, creating /posting bundles. Dig down to ATM350 class folder on
Albany RAMADDA server. Students will be making a final class presentation using IDV and N-AWIPS to generate their graphics.
Sam Ng:
Spend first weeks of weather lab showing GEMPAK/IDV.
Students frustrated by slow load when getting data from
motherlode. Students generate their own case studies. We
have local repository of past month. Investigate putting
best of student case studies on motherlode, or get help
setting up a RAMADDA at metro state.
Marty Baxter:
Finally gave up on GARP. All synoptic case studies from GARP
now in IDV; lots of work but students were pleased. Similar
setup to Albany. Not using the LDM feed at all.
(Organization of RAMADDA?) Took intermediate python course
(with Julian). Getting an electronic map wall, will put up
GEMPAK/IDV products.
Stefan Cecelski:
Still using NCL and IDV to do realtime imagery. Looking at
google earth output/kml. UMD is quite disconnected from
Unidata. Students like realtime data from Unidata but
faculty not as enthusiastic.
Steven Lazarus:
Has become an IDV convert, use it exclusively in 1 credit
weather discussion class. Create exam directly out of IDV
graphics. Using LDM feed to run flat panel display (running
NMAP2).
Mike Baldwin:
Our situation is similar to Steve's; gone from having our
own physical server in the building, with staffing, to
having a vm on a college server farm. No one in campus IT
understands how to run LDM, decoders etc. Had students
using IDV, nmap2; left it up to students what to use, and
they mostly chose nmap2. A lot of frustration with
maintaining the technology and data streams. Started project
with state DOT to help with winter weather decisions; needed
to be able to share info. Ended up going with google docs --
easy to use but can't do science with it. Need good way to
share info with decision makers. (MR: look at google open
layers) Moving towards python to visualize WRF output.
Russ Schumacher:
Idea from December workshop to develop distributed national
ensemble. One of the issues we'll run into is sharing the
data that we're going to generate if ensemble takes off. Got
an electronic map wall up and running, still in progress.
Scaling up use of LDM for common data sets, creating an
archive.
Committee members had read the staff status reports before the meeting. There were a few points of discussion:
Equip Awards - went well yesterday - winners will be announced in the Unidata web site news
AWIPS II / GEMPAK - Marty asked about Gempak 7.0...was worried it wouldn't work without EDEX server. Not clear.
Unidata citation site (software, products, etc.) - Should draw peoples attention to the citation.
LDM - Idea for LDM 7.0 came from a prof. at Virginia. Used an EGER grant funded a neat project to reduce the bandwidth used at places like Unidata.
McIDAS - McIdas V parallels IDV - but is behind latest IDV.
Rosetta - Create and advocate for standard specs on taking pt samples. There's no set structure to ASCII files. Excel compatible. Driven by ACADIS user community. NetCDF to Excel and vice versa. Need volunteers/beta testers.
ACTION 3: Committee members with obs data sources/sensor data amenable to importing into Rosetta should contact Sean to serve as usability testers. (Mike Baldwin, Steven Lazarus offered)
Rich Signell on the line. Validation services for Discrete Sampling Geometries? Check for certain file attributes? Suggested that Rosetta do something like this...otherwise not sure you have the conversion right.
Ethan's slides
Systems researched:
Ethan - RAMADDA Alternatives? ScienceBase (USGS) interested in connecting with TDS. Rich Signell notes that it's worth making a trip to Fort Collins (USGS) to talk with them because they started with the RAMADDA functionality as their baseline. Maybe EarthCube proposal?
All researched systems support data uploading, adding metadata, discovery, http download, restful web API. Some of them support more advanced options. None support server side processing. So no pulling up a time series, for example, as can be done with RAMADDA. NcETL and Science Gateway Framework have interest in collaborating. Recommendation is to stay in contact with them; no resources at present to do anything further.
Rich pointed to http://geo.gov.ckan.org/dataset as an example.
Jeff Weber pointed out that RAMADDA is being used by NASA and NOAA (Peter Griffith/ABoVE project)
Last fall's joint meeting recommendation for Yuan Ho to devote partial time to work with RAMADDA remains in place; due to demands of getting IDV 4 released, this has not been accomplished yet.
Mohan gave a summary of the NSF proposal for the next 5 years. Shared an Outline and then followed with details.
Uncertain Future! Funding for IT has decreased in the University environment. Cloud computing has matured! It is being embraced by academia. Even some universities have their own cloud services. Not just email and course ware- this also includes science! Matlab, etc. Gave an example - Cornell. They are providing tools! George Washington U. as well.
BYOD - Bring Your Own Device. University computer labs are being shut down. Tablets are outselling laptops, laptops are outselling PCs. Students have grown up in a world without dumb terminals, telnet, FTP. They want simple tools - browsers and apps! Should we be looking to phase out old stuff - bring in the new? Need to be 'bold but not crazy'. Adaptation to change is important.
Comments:
There was a short discussion of the two surveys conducted by the UPC since the last meeting: the IT survey and the university science survey. Did not go over the results, but there were a couple of comments:
Russ Schumacher: As we work to make tools more transparent and easy to use, but you may greatly reduce the pool of people who actually know how this stuff works.
Linda Miller: People are suffering from survey fatigue -- why did so few people respond?
Mike Baldwin: I probably get at least one request like that per week -- my first reaction is to delete.
Break for dinner
Jeff Weber reports:
HRRR data has been coming in since March. We'll be getting full 3D fields as of today. We are now getting the Sigma levels; Should we have pressure levels instead? Need feedback from usercom.
Data are fairly big: 8-10 Gb/run
Currently testing: not bringing in to the IDD cluster yet. Long term objective is not to push -- want to provide pull access via TDS. The longer term goal to create a script that lets you get required fields for WRF init.
Mike Baldwin: Don't know if the preprocessing is set up to use the sigma levels
Russ Schumacher: Don't know if WRF can deal with sigma levels -- want pressure
ACTION 4: Jeff Weber will change to ingest pressure levels rather than sigma
Jeff Weber: GSD is providing updated GRIB tables. We are willing to push the data from gale (one of UPC's machines) hourly.
Kevin Tyle: We've been pulling via LDM from GSD (Pressure levels). 2D fields only; a cool thing is that time step is every 15 minutes.
Marty Baxter: Would it be possible to have just the surface fields on Motherlode?
Kevin Tyle: We're getting just 2D, output every 15 minutes
Russ Schumacher: 2D surface fields would be good; Mike Baldwin agreed.
Jeff Weber: FIM is not coming in yet (Global model). Navgen is currently available on TDS
Marty's slides.
Marty gave a nice talk on creating six case studies and the use of IDV and RAMADDA. This is a junior level class, using case study events from COMET archive. Most files are in GEMPAK format. Mary puts together bundles to help student answer questions.
Support requests - list of problems/keeps track of them 6 case studies!
CMU has its own RAMADDA. Marty has developed bundles and some nice exercises to go along. Group projects. Currently saving exercises in the cloud (e.g., Google Docs) - could save in the RAMADDA. But Marty said he was unable to get the RAMADDA to hide the posted work from the student teams.
Students are happy with his assignments. Issue with color table (building one's own). Cross-sections > Map and Transect view. How do we use the IDV most effectively for education? Marty said he liked working with IDV tabs - multiple displays - open at once. Easy to bounce around. Having the data local a huge difference - quick load. Memory upgrade from Isentropic surfaces...built first in Gempak. Don't have to do this - can do the interpolation within IDV. Subtracted out storm motion in IDV - for storm relative flow.
Data Challenges
Radar products - composites? Can do hydrometeor type. Tom Y said... Group liked this idea. With dual-pol there are lot's of possibilities.
RAMADDA is good since easy for students and faculty. Marty said that learning ADDE has been tough and needs help from staff. Tom Y is willing to remotely log into any site that needs hep setting up ADDE. Plans to show this to Marty after close of meeting. Will also assist Kevin at U Albany.
IDV issues:
Can CSU begin to archive radar composites (N0Q mosaic with regular structure (15 min?) for archive) Maybe use an ADDE server?
ACTION 5: Tom Yoksas will check on having both N0R and N0Q at once.
Can IDV have a regular release schedule July and December? Fits with academic schedules...
Should we have a formal meeting to share "what you're doing with IDV"?
Site visits might be less intimidating than regional workshop
Idea: Create a way to share desktop with support -- make a support appointment? WebEx / GoToMeeting?
Sam Ng
RE BYOD: make an app or lite version with model data, don't
need all the models. Augment radar and satellite apps that
exist from elsewhere (U. Wisc.). In IDV- still can't get upper air data
to work in IDV.
Mike Baldwin
Russ Schumacher -
NWS is heading towards AWIPS II. COMET is
losing its NWS support however. Who will do Training? How is all of
this going to work? How does our focus on NWS related
activities like the AWIPS2 migration and the EDEX server
issues factor in to the kinds of effort from Unidata and
the community at large?
University ensemble issue: will there be a proposal to do
this as a prototype? How do we make the data available, who
will store, etc. Issue of client server calculations. Post-processing
issues. Data needs to be in one place.
Steven Lazarus
Stefan Cecelski
How feasible would it be to make the IDV use a google map as a default background?
What is the status of ISL scripting; can it be enhanced?
Rich Signell
Enthusiasm for R & Python is building -- can a common data
model library fit into these environments? MET office has
package called iris (google scitools iris) all on github.
Richard Hattersley is tech lead. Look for video of
Hattersley describing how they chose this direction. Rich
will circulate links.
Kevin Tyle
Visited Dave Dempsey in San Francisco. Is there a way we can
facilitate sharing the products we create?
Can we reach out to students with a monthly IDV bundle contest?
How about a Usercomm blog?
ACTION 6: Kevin will follow up with UserComm members to carry out site contacts over the summer before the fall meeting.
Does community want access to IRIS surface pressure data? Already available via web. Should it be distributed via the IDD? Potentially interesting. Mostly eastern US. The IRIS folks are seismologists. See http://www.iris.edu/dms/products. Mohan tasked committee to look at the data and provide feedback.
LAPS has been around for 25 years, they are looking at what to do going forward. Unidata doesn't have any experience with LAPS, but would be very interested in integrating output from LAPS into AWIPS II. Perhaps we should wait until an AWIPS II perspective is available, rather than distributing/supporting just the model. Don't want to get into the business of supporting/distributing a model.
Brent Gordon is eager to distribute Space
Weather data. The volume is not large.
Mike Baldwin: We absolutely want this.
ACTION 7: Unidata (Linda Miller) should pursue gaining access to space weather data.
A special thanks to Steven Lazarus for providing his notes from the meeting!
Linda Miller and Doug Dirks