Welcome back to AWIPS Tips! In today’s blog, we’re going to describe how GOES satellite data is ingested by EDEX. If you ever need to troubleshoot GOES satellite products on your EDEX, you can use this blog to guide you through verifying the correct setup of the configuration files and learn other relevant information. We will start by detailing the initial components, then demonstrate an example.
GOES Product Source
GOES satellite products are in netCDF file format and distributed over NOAAPort as tiles that when combined make a full sector, such as CONUS or Full Disk.
At Unidata, before re-distributing the data to our users, we stitch the tiles together to produce one netCDF file.
GOES NetCDF Files
You can get quite a bit of information from just the name of the netCDF file.
In order for the data to be ingested by EDEX, configuration files must match the metadata and relate to what EDEX is expecting, most important are the Physical Element, Sector ID, and Creating Entity. We will reference these in a later step.
If you are noticing an error in your edex-ingest-satellite logfile (like below), you can investigate which of the configuration files are missing or need updating.
WARN 2022-07-08 16:25:03,345 9508 [Ingest.GOESR-2] GoesrNetcdfDecoder: No valid records were found in file: OR_ABI-L2-CMIPF-M6C13_G16_s20221882200216_e20221882209536_c20221882210036.nc
You can view the metadata of the raw file by running the ncdump command, which is installed with EDEX (and when the netcdf rpm is installed). Note: Because the netCDF output is large, you can pipe the output to more (see below) to show the output in a readable format.
ncdump OR_GLM-L2-GLMF-M6_G16_s20221891558000_e20221891559000_c20221891558300.nc | more
GOESR EDEX Plugin
There are two types of configurations used by the GOESR edex plugin depending on the products you are working with:
Single Channel (Channel 1-16)
/awips2/edex/data/utility/common_static/base/satellite/goesr/descriptions/Sectorized_CMI
Derived Products (ex. GLM, Cloud Mask, Sea Surface Temp, etc.)
/awips2/edex/data/utility/common_static/base/satellite/goesr/descriptions/Level2
NOTE: While GLM Flash Point data and Derived Motion Winds are GOES products, they are decoded by the dmw plugin, not the GOESR edex plugin.
Example
Let’s now take a closer look at an example of GLM data.
OR_GLM-L2-GLMF-M6_G16_s20221891558000_e20221891559000_c20221891558300.nc
We know from the filename that we are working with GOES-East data, so we’ll start by looking at the GOES-East-Sectors.xml configuration file.
We need to find the sectorid for the data, which is a property that EDEX needs for proper configuration. To find the sectorid, we match the orbital_slot and scene_id values in the netCDF file to the values in GOES-East-Sectors.xml.
NOTE: When working with GOES-West data, use GOES-West-Sectors.xml
Using ncdump, locate the dataset_name, orbital_slot, and scene_id values in the metadata. For example:
ncdump OR_GLM-L2-GLMF-M6_G16_s20221891558000_e20221891559000_c20221891558300.nc | grep 'dataset_name\|orbital_slot\|scene_id'
These sector configurations are used for all derived level2 products.
Next, we will look at a product description file, specific to GLM
/awips2/edex/data/utility/common_static/base/satellite/goesr/descriptions/Level2/GLMG_FD_tile.xml
to decode the different variables in the GLM file. We can look at the metadata to see which variables are included by using ncdump again:
ncdump OR_GLM-L2-GLMF-M6_G16_s20221881848000_e20221881849000_c20221881848300.nc | more
Click to enlarge
After the X and Y coordinates, the specific GLM products follow; each variable needs an entry in the configuration file. For example, below is the associated configuration for the Total_Optical_energy variable. The variable name and units are found in the metadata for the netCDF file. (Note that not every variable will have units.) There is also a discreteMask variable in this configuration. If the value is set to 0, then data will be displayed when loaded for each gridpoint, otherwise it will be masked and not displayed. The physicalElement is then set to a value, this is the value EDEX stores the processed data as.
After Configuration
When referencing how to add new products to a menu, where to find them in the product browser, or access them via python-awips, you’ll need to know the:
- creatingentity
- sectorid
- physicalelement
In the event you have additional questions about ingesting GOES products into EDEX, you can reach out to us at support-awips@unidata.ucar.edu. And check back in two weeks for the next blog post, where we will share options for visualizing Watches, Warnings, and Advisories in CAVE.
To view archived blogs, visit the AWIPS Tips blog tag, and get notified of the latest updates from the AWIPS team by signing up for the AWIPS mailing list. Questions or suggestions for the team on future topics? Let us know at support-awips@unidata.ucar.edu
