Hi all,
Thanks for the forward, Stonie! And yes, this only affects people using
their own goes-restitch.py. I personally have only seen this on FD
domains, but no harm in filtering 00 out from everything afaik, good call.
Taking a quick glance at AWIPS, conditions have improved but I do still see
the occasionally missing tile. Comparing to my own environment, it appears
these missing tiles aren't arriving so it's not the same issue as what the
GLM data was causing, that seems to have cleared up with the PQACT change.
I think this is what we're seeing on CONUS too, tiles not arriving, my
personal suspicions are upstream but that's mostly a guess. Either way
thank you for making that change! It certainly has helped.
Best,
-Mike
On Thu, Nov 7, 2024 at 5:54 PM Stonie Cooper <cooper@xxxxxxxx> wrote:
> Passing the complete solution along . . . FYI, the solution to the pqact
> entry to account for the new GLM data is specific to your use.
>
> Stonie
> >> forwarded response from AWIPS support >>
>
> Hello,
>
> I'd like to point out that the change MikeZ proposed doesn't fully resolve
> the issue. His suggestion only includes the WFD and EFD sectors, but we are
> also stitching the E/W CONUS and OCONUS FD (AK and PR) sectors as well.
>
> We have updated our LDM (around 2024Z on 11/7/24) to use the following
> pqact entry for stitching the GOES ABI data:
>
> NOTHER ^TIR.(0[1-9]|1[0-6]) KNES
>
> Note: the above pqact change is ONLY for users running their own
> goes-restitch.py. No change needs to be made downstream of Unidata's IDD
> user's (including any AWIPS or THREDDS users) that were previously
> requesting the already stitched data as that's being re-inserted the same
> as before:
>
> # NIMAGE 000
>
> /data/ldm/pub/native/satellite/GOES/GOES16/Products/CloudAndMoistureImagery/CONUS/Channel06/20241107/OR_ABI-L2-CMIPC-M6C06_G16_s20243122241170_e20243122241170_c20243122241170.nc
> # GOES 16/17 Single Channel (ABI) via Unidata IDD
> NIMAGE
>
> ^/data/ldm/pub/native/satellite/GOES/([^/]*)/Products/CloudAndMoistureImagery/([^/]*)/([^/]*)/([0-9]{8})/([^/]*)(c[0-9]{7})(..)(.....).nc
>
> Thanks,
> ~Tiffany
> << forwarded response from AWIPS support <<
>
> *Stonie Cooper, PhD*
> Software Engineer III
> NSF Unidata Program Center
> University Corporation for Atmospheric Research
> *I acknowledge that the land I live and work on is the traditional home of
> The **Chahiksichahiks (**Pawnee), The **Umoⁿhoⁿ (**Omaha), and The **Jiwere
> (**Otoe).*
>
>
> On Wed, Nov 6, 2024 at 8:58 PM Mike Zuranski <zuranski.wx@xxxxxxxxx>
> wrote:
>
>> Greetings fellow LDMers,
>>
>> This is an important message to anyone ingesting GOES-R FullDisk tiles
>> from NOAAPort or the IDD and using goes-restitch.py to stitch them
>> together, including Unidata.
>>
>> TL;DR: When ingesting these ABI tiles (pattern of TIRSii where ii is
>> 01-16), if FullDisk GLM tiles are also ingested (pattern TIRS00) then there
>> is a random chance of that corrupting the stitching process and the
>> resulting data/imagery. The fix is to change the pattern to prevent ingest
>> of the GLM tiles into goes-restitch.py; see below for such a pattern change.
>>
>>
>> I have been working with COD and some within NOAA/NWS to understand an
>> apparent bad data issue as seen on the COD page. Thankfully there's
>> nothing wrong with the data or metadata itself. However this continues to
>> result in bad data on Unidata's AWIPS/EDEX (see attached) and TDS, and
>> likely anyone else using a similar process.
>>
>> The problem:
>> Conventional guidance was to ingest GOES-R tiles with a pattern such as
>> TIRSii, the simplest form of which (TIRS..) would allow anything in for
>> those final two chars. Originally the only products that would match that
>> pattern were the ABI bands so there was no issue. But a while back gridded
>> GLM data was added (
>> https://www.weather.gov/media/notification/pdf_2023_24/scn22-112_goes-r_glm_gridded_data_products_aaa.pdf)
>> with TIRS00 and TIRT00 for all tiles. This matches the conventional
>> pattern for ABI and the tiles go into the goes-restitch.py script. The
>> process fails almost immediately because the GLM data does not contain the
>> required ABI/SCMI metadata. A user opened a github issue last summer for
>> this problem (https://github.com/Unidata/ldm-alchemy/issues/8) and at
>> the time I thought this behavior was benign (though it could bloat log
>> files).
>>
>> For reasons that are not fully understood, this inclusion of GLM tiles
>> appears to be enough to hit some sort of critical threshold and corruption
>> of the stitched ABI data can result. To our knowledge this has only been
>> seen on GOES-East, likely because GOES-West would have significantly less
>> lightning activity and GLM tiles only come in when strikes are sampled. We
>> have been able to both resolve this issue at COD as well as recreate this
>> issue & rule out other causes at other sites.
>>
>> The Solution:
>> Change the pattern used in your REQUEST and/or PQACT statements to filter
>> out TIR[ST]00 from being PIPEd to the goes-restitch.py script. Here are
>> samples that worked for us:
>>
>> old: NOTHER ^TI(R[ST]).. KNES
>> new: NOTHER ^TI(R[ST])(0[1-9]|1[0-6]) KNES
>>
>>
>> To Unidata:
>> Be advised that data going into your EDEX(s) and TDS are being impacted
>> ongoingly (that's a word right?). I respectfully request those patterns be
>> changed to resolve this issue on your end. I would appreciate if someone
>> could let me know when that happens so I can confirm this fix worked for
>> you as well.
>>
>>
>> A few notes:
>> If this change is made to PQACT statements, be aware that in this example
>> a new parens group has been added so you may need to adjust positional
>> arguments depending on your details.
>>
>> So far this has only been seen in GOES-East, but I'd filter both if it
>> were me.
>>
>> There's a strong argument in my mind to adjust goes-restitch.py to
>> further prevent GLM tiles from entering any meaningful code. I may submit
>> that PR later, but honestly if the LDM patterns get adjusted that's
>> enough.
>>
>> If you log goes-restitch.py output (on any level), check your RS.log
>> file. COD's had bloated to over 1GB in size and they were set to "-q" or
>> quiet logging.
>>
>> As a good handful of us have been scrutinizing satellite imagery, we've
>> noticed that recently the odd CONUS tile or two has been missed. This is
>> unrelated, the tile just doesn't arrive /shrug.
>>
>>
>> Sorry for the hefty post, please let me know if there are any questions.
>>
>> Best,
>> -Mike
>>
>>
>> --
>> - Mike Zuranski, Weather Data Nerd
>> ----------------------------------
>> Disclaimer: Thoughts, opinions and all general nonsenses are those of
>> myself and in no way reflect the views or positions of any other person
>> or entity.
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web. Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>>
>>
>> ldm-users mailing list
>> ldm-users@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> https://www.unidata.ucar.edu/mailing_lists/
>>
>
--
- Mike Zuranski
