Passing the complete solution along . . . FYI, the solution to the pqact
entry to account for the new GLM data is specific to your use.
Stonie
>> forwarded response from AWIPS support >>
Hello,
I'd like to point out that the change MikeZ proposed doesn't fully resolve
the issue. His suggestion only includes the WFD and EFD sectors, but we are
also stitching the E/W CONUS and OCONUS FD (AK and PR) sectors as well.
We have updated our LDM (around 2024Z on 11/7/24) to use the following
pqact entry for stitching the GOES ABI data:
NOTHER ^TIR.(0[1-9]|1[0-6]) KNES
Note: the above pqact change is ONLY for users running their own
goes-restitch.py. No change needs to be made downstream of Unidata's IDD
user's (including any AWIPS or THREDDS users) that were previously
requesting the already stitched data as that's being re-inserted the same
as before:
# NIMAGE 000
/data/ldm/pub/native/satellite/GOES/GOES16/Products/CloudAndMoistureImagery/CONUS/Channel06/20241107/OR_ABI-L2-CMIPC-M6C06_G16_s20243122241170_e20243122241170_c20243122241170.nc
# GOES 16/17 Single Channel (ABI) via Unidata IDD
NIMAGE
^/data/ldm/pub/native/satellite/GOES/([^/]*)/Products/CloudAndMoistureImagery/([^/]*)/([^/]*)/([0-9]{8})/([^/]*)(c[0-9]{7})(..)(.....).nc
Thanks,
~Tiffany
<< forwarded response from AWIPS support <<
*Stonie Cooper, PhD*
Software Engineer III
NSF Unidata Program Center
University Corporation for Atmospheric Research
*I acknowledge that the land I live and work on is the traditional home of
The **Chahiksichahiks (**Pawnee), The **Umoⁿhoⁿ (**Omaha), and The **Jiwere
(**Otoe).*
On Wed, Nov 6, 2024 at 8:58 PM Mike Zuranski <zuranski.wx@xxxxxxxxx> wrote:
> Greetings fellow LDMers,
>
> This is an important message to anyone ingesting GOES-R FullDisk tiles
> from NOAAPort or the IDD and using goes-restitch.py to stitch them
> together, including Unidata.
>
> TL;DR: When ingesting these ABI tiles (pattern of TIRSii where ii is
> 01-16), if FullDisk GLM tiles are also ingested (pattern TIRS00) then there
> is a random chance of that corrupting the stitching process and the
> resulting data/imagery. The fix is to change the pattern to prevent ingest
> of the GLM tiles into goes-restitch.py; see below for such a pattern change.
>
>
> I have been working with COD and some within NOAA/NWS to understand an
> apparent bad data issue as seen on the COD page. Thankfully there's
> nothing wrong with the data or metadata itself. However this continues to
> result in bad data on Unidata's AWIPS/EDEX (see attached) and TDS, and
> likely anyone else using a similar process.
>
> The problem:
> Conventional guidance was to ingest GOES-R tiles with a pattern such as
> TIRSii, the simplest form of which (TIRS..) would allow anything in for
> those final two chars. Originally the only products that would match that
> pattern were the ABI bands so there was no issue. But a while back gridded
> GLM data was added (
> https://www.weather.gov/media/notification/pdf_2023_24/scn22-112_goes-r_glm_gridded_data_products_aaa.pdf)
> with TIRS00 and TIRT00 for all tiles. This matches the conventional
> pattern for ABI and the tiles go into the goes-restitch.py script. The
> process fails almost immediately because the GLM data does not contain the
> required ABI/SCMI metadata. A user opened a github issue last summer for
> this problem (https://github.com/Unidata/ldm-alchemy/issues/8) and at the
> time I thought this behavior was benign (though it could bloat log files) .
>
> For reasons that are not fully understood, this inclusion of GLM tiles
> appears to be enough to hit some sort of critical threshold and corruption
> of the stitched ABI data can result. To our knowledge this has only been
> seen on GOES-East, likely because GOES-West would have significantly less
> lightning activity and GLM tiles only come in when strikes are sampled. We
> have been able to both resolve this issue at COD as well as recreate this
> issue & rule out other causes at other sites.
>
> The Solution:
> Change the pattern used in your REQUEST and/or PQACT statements to filter
> out TIR[ST]00 from being PIPEd to the goes-restitch.py script. Here are
> samples that worked for us:
>
> old: NOTHER ^TI(R[ST]).. KNES
> new: NOTHER ^TI(R[ST])(0[1-9]|1[0-6]) KNES
>
>
> To Unidata:
> Be advised that data going into your EDEX(s) and TDS are being impacted
> ongoingly (that's a word right?). I respectfully request those patterns be
> changed to resolve this issue on your end. I would appreciate if someone
> could let me know when that happens so I can confirm this fix worked for
> you as well.
>
>
> A few notes:
> If this change is made to PQACT statements, be aware that in this example
> a new parens group has been added so you may need to adjust positional
> arguments depending on your details.
>
> So far this has only been seen in GOES-East, but I'd filter both if it
> were me.
>
> There's a strong argument in my mind to adjust goes-restitch.py to further
> prevent GLM tiles from entering any meaningful code. I may submit that PR
> later, but honestly if the LDM patterns get adjusted that's enough.
>
> If you log goes-restitch.py output (on any level), check your RS.log
> file. COD's had bloated to over 1GB in size and they were set to "-q" or
> quiet logging.
>
> As a good handful of us have been scrutinizing satellite imagery, we've
> noticed that recently the odd CONUS tile or two has been missed. This is
> unrelated, the tile just doesn't arrive /shrug.
>
>
> Sorry for the hefty post, please let me know if there are any questions.
>
> Best,
> -Mike
>
>
> --
> - Mike Zuranski, Weather Data Nerd
> ----------------------------------
> Disclaimer: Thoughts, opinions and all general nonsenses are those of
> myself and in no way reflect the views or positions of any other person
> or entity.
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
>
