Greetings fellow LDMers, This is an important message to anyone ingesting GOES-R FullDisk tiles from NOAAPort or the IDD and using goes-restitch.py to stitch them together, including Unidata. TL;DR: When ingesting these ABI tiles (pattern of TIRSii where ii is 01-16), if FullDisk GLM tiles are also ingested (pattern TIRS00) then there is a random chance of that corrupting the stitching process and the resulting data/imagery. The fix is to change the pattern to prevent ingest of the GLM tiles into goes-restitch.py; see below for such a pattern change. I have been working with COD and some within NOAA/NWS to understand an apparent bad data issue as seen on the COD page. Thankfully there's nothing wrong with the data or metadata itself. However this continues to result in bad data on Unidata's AWIPS/EDEX (see attached) and TDS, and likely anyone else using a similar process. The problem: Conventional guidance was to ingest GOES-R tiles with a pattern such as TIRSii, the simplest form of which (TIRS..) would allow anything in for those final two chars. Originally the only products that would match that pattern were the ABI bands so there was no issue. But a while back gridded GLM data was added ( https://www.weather.gov/media/notification/pdf_2023_24/scn22-112_goes-r_glm_gridded_data_products_aaa.pdf) with TIRS00 and TIRT00 for all tiles. This matches the conventional pattern for ABI and the tiles go into the goes-restitch.py script. The process fails almost immediately because the GLM data does not contain the required ABI/SCMI metadata. A user opened a github issue last summer for this problem (https://github.com/Unidata/ldm-alchemy/issues/8) and at the time I thought this behavior was benign (though it could bloat log files). For reasons that are not fully understood, this inclusion of GLM tiles appears to be enough to hit some sort of critical threshold and corruption of the stitched ABI data can result. To our knowledge this has only been seen on GOES-East, likely because GOES-West would have significantly less lightning activity and GLM tiles only come in when strikes are sampled. We have been able to both resolve this issue at COD as well as recreate this issue & rule out other causes at other sites. The Solution: Change the pattern used in your REQUEST and/or PQACT statements to filter out TIR[ST]00 from being PIPEd to the goes-restitch.py script. Here are samples that worked for us: old: NOTHER ^TI(R[ST]).. KNES new: NOTHER ^TI(R[ST])(0[1-9]|1[0-6]) KNES To Unidata: Be advised that data going into your EDEX(s) and TDS are being impacted ongoingly (that's a word right?). I respectfully request those patterns be changed to resolve this issue on your end. I would appreciate if someone could let me know when that happens so I can confirm this fix worked for you as well. A few notes: If this change is made to PQACT statements, be aware that in this example a new parens group has been added so you may need to adjust positional arguments depending on your details. So far this has only been seen in GOES-East, but I'd filter both if it were me. There's a strong argument in my mind to adjust goes-restitch.py to further prevent GLM tiles from entering any meaningful code. I may submit that PR later, but honestly if the LDM patterns get adjusted that's enough. If you log goes-restitch.py output (on any level), check your RS.log file. COD's had bloated to over 1GB in size and they were set to "-q" or quiet logging. As a good handful of us have been scrutinizing satellite imagery, we've noticed that recently the odd CONUS tile or two has been missed. This is unrelated, the tile just doesn't arrive /shrug. Sorry for the hefty post, please let me know if there are any questions. Best, -Mike -- - Mike Zuranski, Weather Data Nerd ---------------------------------- Disclaimer: Thoughts, opinions and all general nonsenses are those of myself and in no way reflect the views or positions of any other person or entity.
Attachment:
062150_band14_unidataedex.png
Description: PNG image
ldm-users archives: