Welcome back to AWIPS Tips! With this entry, we’re going to touch on the topic of adding a new grib product into AWIPS. We’ll be discussing updates that need to be made for both EDEX and CAVE in order to display your new data properly. In order to do this you need to be running your own EDEX server. Most of the recommendations and tips in this blog can be found in our documentation about ingesting new grid data.
Near the beginning of this year, the European Centre for Medium-Range Weather Forecasts (ECMWF) made a significant amount of their data openly available, and we’ve had a few users express interest about this data. At this time, Unidata does not distribute ECMWF data, but we figured we’d take the opportunity to help explain how you can modify your AWIPS setup to handle it. We are specifically going to be addressing ECMWF model data which is formatted as grib2 files, for this walkthrough.
NOTE: The ECMWF data is not a product currently available via Unidata's Internet Data Distribution system but if you do use ECMWF data please provide proper attribution.
The very first step to working with any new data is to try ingesting it with manual ingest. This provides initial troubleshooting information, and possibly allows you to view the data right away. For Unidata’s EDEX the manual ingest point is at /awips2/data_store/ingest/. Simply copy the data file into this directory and EDEX will attempt to ingest the data. Since we are working with grid data, there’s a good chance EDEX will ingest the data without alteration, but it may not be labeled or properly configured yet.
Here is an example ECMWF grib2 file named 20220406000000-15h-oper-fc.grib2. When this is copied into EDEX’s manual endpoint, with no configuration changes, it shows up in CAVE in the product browser under GribModel:98:0:153.
The numbers are the Center (98), Subcenter (0), and Process ID (153) for the file. You can confirm these values by using wgrib2 on the command line as well.
NOTE: wgrib2 is a tool that is installed by default with EDEX
wgrib2 -center 20220406000000-15h-oper-fc.grib2
1:0:center=European Center for Medium-Range Weather Forecasts (RSMC)
wgrib2 -subcenter 20220406000000-15h-oper-fc.grib2
1:0:subcenter=0
wgrib2 -processid 20220406000000-15h-oper-fc.grib2
1:0:background generating process=255 forecast generating process=153
For the center, you will need to look up the output name, in this case “European Center for Medium-Range Weather Forecasts (RSMC)” in this table.
Now we must check to see if a location file exists for our specific grib file. In order to do that, we’ll need to gather some more information directly from our file using wgrib2 and create a new projection file, similar to what’s described in our documentation.
wgrib2 -grid -nxny 20220406000000-15h-oper-fc.grib2
Looking at the output from one of the parameters we have:
82:51159324:grid_template=0:winds(N/S):
lat-lon grid:(900 x 451) units 1e-06 input WE:NS output WE:SN res 48
lat 90.000000 to -90.000000 by 0.400000
lon 180.000000 to 179.600000 by 0.400000 #points=405900:(900 x 451)
This tells us the following:
- Projection type: lat-lon
- la1: 90.000
- lo1: 180.000
- nx: 900
- ny: 451
- dx: 0.400
- dy: 0.400
- First grid corner: west->east, north->south = upper-left
We can search through all the existing grib projection files to see if any of them already contain the matching information we need. These files are located in /awips2/edex/data/utility/common_static/base/grib/grids/. For example:
cd /awips2/edex/data/utility/common_static/base/grib/grids/
grep -r ‘
We find this returns no results, so we have to make a new file. We can start by copying an existing file, let's use ECMWF-Grid13.xml.
cp ECMWF-Grid13.xml ECMWF-Grid14.xml
And now we’ll use the information we gathered previously and modify the content to look like this:
<latLonGridCoverage>
<name>255114</name>
<description>ECMWF</description>
<la1>90.000</la1>
<lo1>180.000</lo1>
<firstGridPointCorner>UpperLeft</firstGridPointCorner>
<nx>900</nx>
<ny>451</ny>
<dx>0.400</dx>
<dy>0.400</dy>
<spacingUnit>degree</spacingUnit>
</latLonGridCoverage>
NOTE: The <name> flag can be whatever we want, but it must be unique, and we’ll need it for the following step, so take note of it.
To get the data to show up with the proper labels in CAVE, we have to modify the appropriate grib model file, in this case, /awips2/edex/data/utility/common_static/base/grib/models/gribModels_ECMWF-98.xml. A new model entry is needed, which looks like this:
<model>
<name>ECMWFO</name>
<center>98</center>
<subcenter>0</subcenter>
<grid>255114</grid>
<process>
<id>153</id>
</process>
</model>
Again, here the name can be whatever we’d like and this is what will show up in the Product Browser. The center, subcenter, and processID are the same from earlier, and the grid tag is the name we gave our new projection file earlier.
At this point, we’re ready to restart EDEX and re-ingest a file. For simplicity sake, we’ll ingest a new file (because EDEX won’t ingest duplicate records, if we wanted to ingest the same file we need to remove database and HDF5 records).
Let’s ingest this 30 hour forecast now, 20220406000000-30h-oper-fc.grib2. Now we see our record under the new category we created ECMWFO:
Now we can view some data in CAVE. We’ll load a temperature map from the following: Temperature (T) > Fixed Height Above Ground (FHAG) > 0.0 FHAG > Load as Image
Grid parameters are defined in the tables found in /awips2/edex/data/utility/common_static/base/grib/tables/. Parameters will match to the default products found in /awips2/edex/data/utility/common_static/base/grib/tables/-1/-1/ and in their specific tables, in this case, found in: /awips2/edex/data/utility/common_static/base/grib/tables/98/0/4.2.0.1.table.
Unfortunately, there are two known parameters that do not map correctly by default. ECMWF uses two local parameters:
- var0_1_193: Total precipitation in metres of water equivalent - paramId 228; and
- var2_0_201: Runoff in metres - paramId 205
These are mapped to 193 and 201 which are incorrect for this particular model. You can change the specific 4.2.0.1.table mentioned above (.../grib/tables/98/0/4.2.0.1.table), but be aware this will change those parameters for other models using that table as well.
One final note, now that you have your EDEX completely set up to decode and ingest ECMWF data correctly you will want to have some way to ingest it other than using the manual ingest node. If you can receive this data through an LDM, that is the ideal method, and you would simply modify the pqact.conf to manage this new data. If that is not possible, you may be able to write a script of your own that gets the data from some source and then uses the notify-AWIPS.py script found in /awips2/edex/ldm/dev/ to easily insert your data into EDEX as well.
With a consistent data stream set up and the EDEX decoder defined and working correctly, you may also want to add new menu items into CAVE to make the data easier to access. All the data would still be available via the Product Browser, but you could make common/high-interest data products available from the Model menu.
Thanks for joining us this week, please check back in two weeks for the next blog post, how to view warnings, watches and advisories in python-awips.
To view archived blogs, visit the AWIPS Tips blog tag, and get notified of the latest updates from the AWIPS team by signing up for the AWIPS mailing list. Questions or suggestions for the team on future topics? Let us know at support-awips@unidata.ucar.edu
This blog was posted in reference to v18.2.1-5 of NSF Unidata AWIPS
