Mike (and all),
This has been an ongoing issue for several months now. The root problem is that
the queue sizes on the ingest machines at NCEP are not nearly large enough to
handle the amount of data flowing through them. When the lags get large,
products age off of their queues before the downstream sites have a chance to
get them.
We are a top level CONDUIT relay, so we are pulling data direct from NCEP and
we have been plagued with these missing products for several months now.
Unidata's servers are also generally missing the same products that we are .
The Unidata folks are also aware of the issue, but since NCEP runs the servers,
there's not much the Unidata folks can do other than let NCEP know that they
need larger queues. It sounds like NCEP is planning to address this, but not
until at least sometime in 2022.
The short term solution (and Tom from Unidata can correct me if I'm way off
base) is probably either
1. get rid of the 0.25 degree data from the CONDUIT stream until they can
increase the queue sizes on the source machines
or
1. Set up a data feed separate from CONDUIT where one site (Unidata?)
downloads the data from the NCEP's server and injects it into a data feed
separate from CONDUIT - sort of a parallel feed of the data that is flowing
through CONDUIT.
This has been frustrating for a while now.. The GFS data coming through CONDUIT
is so unreliable that it is pretty useless as it is.
Pete
<http://www.weather.com/tv/shows/wx-geeks/video/the-incredible-shrinking-cold-pool>-----
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086 - poker@xxxxxxxxxxxx
________________________________
From: conduit <conduit-bounces@xxxxxxxxxxxxxxxx> on behalf of Michael Leuthold
<leuthold@xxxxxxxxxxxxxxxxx>
Sent: Wednesday, November 17, 2021 9:28 AM
To: conduit@xxxxxxxxxxxxxxxx <conduit@xxxxxxxxxxxxxxxx>
Subject: [conduit] Partial GFS files
Hello,
Has anyone else had recent issues with missing GFS files? Over the
past few weeks, I've had many that are short. The problem usually
starts around forecast hour 100 and resolves around hour 200. Both .25
and .5 degree have this issue. It's worse at 0 and 12Z.
thanks.
Mike
-rw-rw-r-- 1 ldm ldm 523M Nov 17 04:16 2132100F081.grib2
-rw-rw-r-- 1 ldm ldm 523M Nov 17 03:57 2132100F084.grib2
-rw-rw-r-- 1 ldm ldm 522M Nov 17 04:17 2132100F087.grib2
-rw-rw-r-- 1 ldm ldm 524M Nov 17 03:59 2132100F090.grib2
-rw-rw-r-- 1 ldm ldm 521M Nov 17 04:00 2132100F093.grib2
-rw-rw-r-- 1 ldm ldm 526M Nov 17 04:01 2132100F096.grib2
-rw-rw-r-- 1 ldm ldm 127M Nov 17 04:18 2132100F099.grib2
-rw-rw-r-- 1 ldm ldm 379M Nov 17 04:19 2132100F102.grib2
-rw-rw-r-- 1 ldm ldm 521M Nov 17 04:03 2132100F105.grib2
-rw-rw-r-- 1 ldm ldm 314M Nov 17 04:20 2132100F108.grib2
-rw-rw-r-- 1 ldm ldm 519M Nov 17 04:04 2132100F111.grib2
-rw-rw-r-- 1 ldm ldm 209M Nov 17 04:22 2132100F114.grib2
-rw-rw-r-- 1 ldm ldm 524M Nov 17 04:06 2132100F117.grib2
-rw-rw-r-- 1 ldm ldm 528M Nov 17 04:07 2132100F120.grib2
-rw-rw-r-- 1 ldm ldm 181M Nov 17 04:24 2132100F123.grib2
-rw-rw-r-- 1 ldm ldm 160M Nov 17 04:24 2132100F126.grib2
-rw-rw-r-- 1 ldm ldm 524M Nov 17 04:08 2132100F129.grib2
-rw-rw-r-- 1 ldm ldm 525M Nov 17 04:09 2132100F132.grib2
-rw-rw-r-- 1 ldm ldm 317M Nov 17 04:25 2132100F135.grib2
-rw-rw-r-- 1 ldm ldm 524M Nov 17 04:10 2132100F138.grib2
-rw-rw-r-- 1 ldm ldm 474M Nov 17 04:27 2132100F141.grib2
-rw-rw-r-- 1 ldm ldm 522M Nov 17 04:28 2132100F144.grib2
-rw-rw-r-- 1 ldm ldm 220M Nov 17 04:29 2132100F147.grib2
-rw-rw-r-- 1 ldm ldm 309M Nov 17 04:31 2132100F150.grib2
-rw-rw-r-- 1 ldm ldm 519M Nov 17 04:14 2132100F153.grib2
-rw-rw-r-- 1 ldm ldm 149M Nov 17 04:32 2132100F156.grib2
-rw-rw-r-- 1 ldm ldm 519M Nov 17 04:15 2132100F159.grib2
-rw-rw-r-- 1 ldm ldm 252M Nov 17 04:35 2132100F162.grib2
-rw-rw-r-- 1 ldm ldm 515M Nov 17 04:16 2132100F165.grib2
-rw-rw-r-- 1 ldm ldm 520M Nov 17 04:17 2132100F168.grib2
-rw-rw-r-- 1 ldm ldm 498M Nov 17 04:38 2132100F171.grib2
-rw-rw-r-- 1 ldm ldm 520M Nov 17 04:19 2132100F174.grib2
-rw-rw-r-- 1 ldm ldm 517M Nov 17 04:19 2132100F177.grib2
-rw-rw-r-- 1 ldm ldm 355M Nov 17 04:39 2132100F180.grib2
-rw-rw-r-- 1 ldm ldm 427M Nov 17 04:40 2132100F183.grib2
-rw-rw-r-- 1 ldm ldm 519M Nov 17 04:22 2132100F186.grib2
-rw-rw-r-- 1 ldm ldm 515M Nov 17 04:23 2132100F189.grib2
--
Mike Leuthold
Manager, Regional Weather Modeling Program
University of Arizona, Hydrology and Atmospheric Sciences
520-282-1478
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
conduit mailing list
conduit@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
https://www.unidata.ucar.edu/mailing_lists/
