Pete,
I like the idea of getting rid of the 0.25 degree so we could at
least get all files of the 0.5 data.
Mike
On 11/17/2021 10:06 AM, Pete Pokrandt wrote:
*External Email*
Mike (and all),
This has been an ongoing issue for several months now. The root
problem is that the queue sizes on the ingest machines at NCEP are not
nearly large enough to handle the amount of data flowing through them.
When the lags get large, products age off of their queues before the
downstream sites have a chance to get them.
We are a top level CONDUIT relay, so we are pulling data direct from
NCEP and we have been plagued with these missing products for several
months now. Unidata's servers are also generally missing the same
products that we are.
The Unidata folks are also aware of the issue, but since NCEP runs the
servers, there's not much the Unidata folks can do other than let NCEP
know that they need larger queues. It sounds like NCEP is planning to
address this, but not until at least sometime in 2022.
The short term solution (and Tom from Unidata can correct me if I'm
way off base) is probably either
1. get rid of the 0.25 degree data from the CONDUIT stream until they
can increase the queue sizes on the source machines
or
2. Set up a data feed separate from CONDUIT where one site (Unidata?)
downloads the data from the NCEP's server and injects it into a
data feed separate from CONDUIT - sort of a parallel feed of the
data that is flowing through CONDUIT.
This has been frustrating for a while now.. The GFS data coming
through CONDUIT is so unreliable that it is pretty useless as it is.
Pete
-----
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086 - poker@xxxxxxxxxxxx
------------------------------------------------------------------------
*From:* conduit <conduit-bounces@xxxxxxxxxxxxxxxx> on behalf of
Michael Leuthold <leuthold@xxxxxxxxxxxxxxxxx>
*Sent:* Wednesday, November 17, 2021 9:28 AM
*To:* conduit@xxxxxxxxxxxxxxxx <conduit@xxxxxxxxxxxxxxxx>
*Subject:* [conduit] Partial GFS files
Hello,
Has anyone else had recent issues with missing GFS files? Over the
past few weeks, I've had many that are short. The problem usually
starts around forecast hour 100 and resolves around hour 200. Both .25
and .5 degree have this issue. It's worse at 0 and 12Z.
thanks.
Mike
-rw-rw-r-- 1 ldm ldm 523M Nov 17 04:16 2132100F081.grib2
-rw-rw-r-- 1 ldm ldm 523M Nov 17 03:57 2132100F084.grib2
-rw-rw-r-- 1 ldm ldm 522M Nov 17 04:17 2132100F087.grib2
-rw-rw-r-- 1 ldm ldm 524M Nov 17 03:59 2132100F090.grib2
-rw-rw-r-- 1 ldm ldm 521M Nov 17 04:00 2132100F093.grib2
-rw-rw-r-- 1 ldm ldm 526M Nov 17 04:01 2132100F096.grib2
-rw-rw-r-- 1 ldm ldm 127M Nov 17 04:18 2132100F099.grib2
-rw-rw-r-- 1 ldm ldm 379M Nov 17 04:19 2132100F102.grib2
-rw-rw-r-- 1 ldm ldm 521M Nov 17 04:03 2132100F105.grib2
-rw-rw-r-- 1 ldm ldm 314M Nov 17 04:20 2132100F108.grib2
-rw-rw-r-- 1 ldm ldm 519M Nov 17 04:04 2132100F111.grib2
-rw-rw-r-- 1 ldm ldm 209M Nov 17 04:22 2132100F114.grib2
-rw-rw-r-- 1 ldm ldm 524M Nov 17 04:06 2132100F117.grib2
-rw-rw-r-- 1 ldm ldm 528M Nov 17 04:07 2132100F120.grib2
-rw-rw-r-- 1 ldm ldm 181M Nov 17 04:24 2132100F123.grib2
-rw-rw-r-- 1 ldm ldm 160M Nov 17 04:24 2132100F126.grib2
-rw-rw-r-- 1 ldm ldm 524M Nov 17 04:08 2132100F129.grib2
-rw-rw-r-- 1 ldm ldm 525M Nov 17 04:09 2132100F132.grib2
-rw-rw-r-- 1 ldm ldm 317M Nov 17 04:25 2132100F135.grib2
-rw-rw-r-- 1 ldm ldm 524M Nov 17 04:10 2132100F138.grib2
-rw-rw-r-- 1 ldm ldm 474M Nov 17 04:27 2132100F141.grib2
-rw-rw-r-- 1 ldm ldm 522M Nov 17 04:28 2132100F144.grib2
-rw-rw-r-- 1 ldm ldm 220M Nov 17 04:29 2132100F147.grib2
-rw-rw-r-- 1 ldm ldm 309M Nov 17 04:31 2132100F150.grib2
-rw-rw-r-- 1 ldm ldm 519M Nov 17 04:14 2132100F153.grib2
-rw-rw-r-- 1 ldm ldm 149M Nov 17 04:32 2132100F156.grib2
-rw-rw-r-- 1 ldm ldm 519M Nov 17 04:15 2132100F159.grib2
-rw-rw-r-- 1 ldm ldm 252M Nov 17 04:35 2132100F162.grib2
-rw-rw-r-- 1 ldm ldm 515M Nov 17 04:16 2132100F165.grib2
-rw-rw-r-- 1 ldm ldm 520M Nov 17 04:17 2132100F168.grib2
-rw-rw-r-- 1 ldm ldm 498M Nov 17 04:38 2132100F171.grib2
-rw-rw-r-- 1 ldm ldm 520M Nov 17 04:19 2132100F174.grib2
-rw-rw-r-- 1 ldm ldm 517M Nov 17 04:19 2132100F177.grib2
-rw-rw-r-- 1 ldm ldm 355M Nov 17 04:39 2132100F180.grib2
-rw-rw-r-- 1 ldm ldm 427M Nov 17 04:40 2132100F183.grib2
-rw-rw-r-- 1 ldm ldm 519M Nov 17 04:22 2132100F186.grib2
-rw-rw-r-- 1 ldm ldm 515M Nov 17 04:23 2132100F189.grib2
--
Mike Leuthold
Manager, Regional Weather Modeling Program
University of Arizona, Hydrology and Atmospheric Sciences
520-282-1478
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
conduit mailing list
conduit@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
https://www.unidata.ucar.edu/mailing_lists/
--
Mike Leuthold
Manager, Regional Weather Modeling Program
University of Arizona, Hydrology and Atmospheric Sciences
520-282-1478
