Using Pete's example as the template:
DDS|IDS ^S[AP].* .... ([0-3][0-9])([0-2][0-9])
FILE /usr/local/ldm/data/weather/(\1:yy)(\1:mm)\1\2.METAR
Which works for everything but the new messages, for the new 99 headed
data, *an* option:
DDS|IDS ^S[AP]US99 .... ([0-3][0-9])([0-2][0-9])([0-5][0-9])
PIPE /usr/local/ldm/util/saus99_binning.sh
/usr/local/ldm/data/weather/ (\1:yy)(\1:mm)\1\2.METAR \3
Where /usr/local/ldm/util/saus99_binning.sh *could* be something like:
--- cut here ---
#!/bin/bash
#assessing the next hour based upon 10 minute look forward as described by
NOAA
datahome="${1}"
datafile="${2}"
minutes="${3}"
mkdir -p ${datahome}
dateonly=`echo "${datafile}" | grep -E -o "[0-9]*"`
echo "${#dateonly}"
if [ "${#dateonly}" -eq "8" ]; then
outputdate=`date -d "20${datafile:0:6} ${datafile:6:2}:${minutes}:00 Z
+10minutes" -u "+%y%m%d%H"`
else
outputdate=`date -d "${datafile:0:8} ${datafile:8:2}:${minutes}:00 Z
+10minutes" -u "+%Y%m%d%H"`
fi
outputfile=`echo "${datafile}" | sed -e "s|${dateonly}|${outputdate}|g"`
cat >> ${datahome}/${outputfile}
--- cut here ---
If you do this, you will want to edit your original pattern to exclude the
S[AP]US99 . . . for everything else.
Or you can pipe everything through the script.
-
Stonie Cooper, PhD
Software Engineer III
NSF Unidata
cooper@xxxxxxxx
On Wed, Jan 29, 2025 at 11:07 AM Mike Zuranski <zuranski.wx@xxxxxxxxx>
wrote:
> Morning all,
>
> I received a response to my inquiry:
>
> First thank you for reaching out to us on this matter. *This was an
> unintentional change* but I can describe how it happened. The SAUS70 &
> SAUS80 METAR collectives that were previously disseminated over the
> SBN/NOAAPort prior to this change would get disseminated every 10 minutes,
> including the latest observations in that period. For example, many
> observations are time stamped at 55 after the hour, which are then included
> in the top of the hour SAUS70 METAR collective. However, SAUS99 METAR
> collectives are issued upwards of several times a minute containing the
> latest observation. We get a big benefit here in more timely observational
> data. In addition to the improved timeliness, SAUS70 & SAUS80 require
> manual intervention to add sites whereas sites automatically are added to
> SAUS99. Given these two benefits, *we hope the end-users applications
> are able to be updated to work with SAUS99*.
>
> Feel free to let us know if you have any additional questions or concerns!
>
> So there it is, this is the new norm and expected behavior moving forward .
>
> If this is negatively impactful to community members enough, my personal
> opinion is this might be addressed the easiest if Unidata could offer a
> decoder of sorts to adapt to this new norm. I can't say how widespread any
> impacts are, all I've heard is what's been said here. And of course I'm
> sure Unidata staff are hard at work as always and it's not my place to
> suggest how their energies should be spent. But that's where my head is
> heading.
>
> Best,
> -Mike
>
> On Tue, Jan 28, 2025 at 2:47 PM Sebenste, Gilbert <sebensteg@xxxxxxx>
> wrote:
>
>> This is what I see with the raw NADIN/WMSCR feed, which I’ve had access
>> to. These are indeed the RAW METARs sent without any processing from the
>> NWS, which slowed things down and had duplicate observations. The upsides
>> are that new METARs show up immediately, and assuming the feed to the NWS
>> is timely, you’ll get them 1-10 minutes faster than before.
>>
>>
>>
>> Gilbert Sebenste
>>
>> Meteorology Support Analyst
>>
>>
>>
>> *From:* ldm-users <ldm-users-bounces@xxxxxxxxxxxxxxxx> *On Behalf Of *Pete
>> Pokrandt
>> *Sent:* Monday, January 27, 2025 2:20 PM
>> *To:* Herzmann, Daryl E [AGRON] <akrherz@xxxxxxxxxxx>;
>> ldm-users@xxxxxxxxxxxxxxxx; Tyle, Kevin R <ktyle@xxxxxxxxxx>
>> *Subject:* [External] Re: [ldm-users] The S[AP]US99 update for METARs
>> just happened
>>
>>
>>
>> CAUTION: This email originated from outside of COD’s system. Do not
>> click links, open attachments, or respond with sensitive information unless
>> you recognize the sender and know the content is safe.
>>
>>
>>
>> All,
>>
>>
>>
>> I just noticed a curiosity about the new SAUS99 METAR format that I'm not
>> immediately sure how to deal with. I am getting the SAUS99 and SPUS99
>> reports, but the time on the WMO header is different, and that's messing up
>> my data filing in the LDM.
>>
>>
>>
>> To save data for the old Pete Neilley weather program (yes, I do still
>> use that..) I use the following
>>
>>
>>
>> DDS|IDS ^S[AP].* .... ([0-3][0-9])([0-2][0-9]) FILE
>> /usr/local/ldm/data/weather/(\1:yy)(\1:mm)\1\2.METAR
>>
>>
>>
>> Note that it is using the DDMM in the WMO product header (the
>> ([0-3][0-9])([0-2][0-9]) part ) to determine the day/hour of the file to
>> put it in.
>>
>>
>>
>> As we all know, the METAR times are a bit before the actual hour they are
>> valid for (e.g. the METAR with a time of 1253 is actually the 13 UTC ob,
>> and should be put in the 13 UTC file.
>>
>>
>>
>> With the old format (SAUS70/80/90) the time on the WMO header actually
>> was the correct time (1300) for the 1253 ob.
>>
>>
>>
>> SAUS80 KWBC 271300 RRB^M^M
>>
>> METAR^M^M
>>
>>
>>
>> KMSN 271253Z 23007KT 200V270 10SM FEW180 FEW250 M07/M14 A2991 RMK^M^M
>>
>> AO2 SLP141 T10721144 $=^M^M
>>
>>
>>
>>
>>
>> However, with the new format (SAUS99) the time on the WMO is generally
>> earlier, so in the example here, the 1753 Ob (which should be in the 18 UTC
>> file) ends up in the 17 UTC file.
>>
>>
>>
>> SAUS99 KWBC 271756 RRH^M^M
>>
>> METAR^M^M
>>
>>
>>
>> KMSN 271753Z 25011G23KT 10SM BKN160 BKN250 04/M12 A2973 RMK AO2^M^M
>>
>> SLP077 T00391122 10044 21072 58033=^M^M
>>
>>
>>
>> I think I found a workaround for my purposes, but wanted to put it out
>> there in case it's affecting anyone else..
>>
>>
>>
>> Pete
>>
>>
>>
>>
>>
>>
>>
>> Pete Pokrandt - System Engineer IV
>>
>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>
>> 608-262-3086 - poker@xxxxxxxxxxxx
>> ------------------------------
>>
>> *From:* ldm-users <ldm-users-bounces@xxxxxxxxxxxxxxxx> on behalf of
>> Herzmann, Daryl E [AGRON] <akrherz@xxxxxxxxxxx>
>> *Sent:* Monday, January 27, 2025 11:27 AM
>> *To:* ldm-users@xxxxxxxxxxxxxxxx <ldm-users@xxxxxxxxxxxxxxxx>
>> *Subject:* [ldm-users] The S[AP]US99 update for METARs just happened
>>
>>
>>
>> Howdy,
>>
>> A reminder that this change just happened.
>>
>> The NWS National Centers for Environmental Prediction (NCEP) Central
>> Operations (NCO) creates routine collectives of observational METAR data
>> in SAUS70 KWBC, SAUS80 KWBC, SAUS90 KWBC and their counterparts for
>> Special observations SPUS70, SPUS80 and SPUS90. These collectives
>> require manual intervention to add METAR stations to them. These are
>> then distributed over the SBN. Another product, SAUS99 KWBC, and the
>> Special observation counterpart SPUS99 KWBC contain any METAR observation
>> received every minute. Delivering SAUS99 and SPUS99 over the SBN rather
>> than the aforementioned collectives will provide more observational data
>> and more timely data to users of the SBN.
>>
>> The replacement with SAUS99 and SPUS99 is expected to occur on or about
>> 1200 Coordinated Universal Time (UTC), January 27, 2025. If a Critical
>> Weather Day has been issued, this change will be postponed until 1200 UTC
>> on the next non-Critical Weather Day.
>>
>>
>> https://urldefense.com/v3/__https://www.weather.gov/media/notification/pdf_2023_24/scn24-99_add_saus99_to_sbn_aaa.pdf__;!!Mak6IKo!K44CJVSMFQBBR4h9RLswmLpN1JShZZJLM-zvWlrCQT26pq7B-6JV6h9ZIAqtSvktkgbOL-pBQRZTgA9LCC8p$
>>
>> So far, I have observed 265 SAUS99 products and 114 SPUS99 products cross
>> the IDD since about 15 Z
>>
>> daryl
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web. Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>>
>>
>> ldm-users mailing list
>> ldm-users@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> https://urldefense.com/v3/__https://www.unidata.ucar.edu/mailing_lists/__;!!Mak6IKo!K44CJVSMFQBBR4h9RLswmLpN1JShZZJLM-zvWlrCQT26pq7B-6JV6h9ZIAqtSvktkgbOL-pBQRZTgL9fV5rk$
>>
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web. Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>>
>>
>> ldm-users mailing list
>> ldm-users@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> https://www.unidata.ucar.edu/mailing_lists/
>>
>
>
> --
> - Mike Zuranski
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
>
