[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [conduit] Partial GFS files

Hello Pete and Mike (and all),

Thank you Pete for your excellent assessment of the problem.  Since I
run twice-daily WRF runs initialized and forced by GFS 0.25-degree
model forecasts for hours 0 to 180, I have had to develop a program
that uses wgrib2 to find out which GFS files are missing or incomplete
(which is actually much more likely than being completely missing).  I
then run a wget command to retrieve these bad grids from either or both
of these servers:
 server1 = "ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod";
 server2 = "https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod";

I also need to do this for 1-degree GFS files to produce GFS forecast
graphics for our web page.  It took a fair amount of effort to set
this up, but it has been necessary and now, after many iterations, it
is quite reliable.  On a couple of rare occasions, however, certain
fields at certain pressure levels never made it into a specific GFS
forecast hour's file on either of those servers and I had to pivot.

On average, over the past 4 0Z/12Z cycles, my scripts needed to use
wget to retrieve about 12 files per run (of the 61 files for hour
0-180) due to CONDUIT files being incomplete.

I still recommend that CONDUIT carry the GFS 0.25-degree files, since
about 75% of the data are useful and CONDUIT does save us from
retrieving it all on our own. 

If people are interested and have already built wgrib2, I suppose I
could make my PERL and CSH scripts available to the community.  They
would need some editing to remove our local idiosyncracies.

Cheers,

David
-- 
David Ovens              e-mail: address@hidden
Research Meteorologist    phone: (206) 685-8108
Dept of Atm. Sciences      plan: Real-time WRF forecasting for the
Box 351640                        Pacific Northwest
University of Washington          https://atmos.uw.edu/wrfrt
Seattle, WA  98195               Weather Graphics and Loops
                                  https://atmos.uw.edu/~ovens/loops


On Wed, Nov 17, 2021 at 05:06:13PM +0000, Pete Pokrandt via conduit wrote:
> Mike (and all),
> 
> This has been an ongoing issue for several months now. The root problem is 
> that the queue sizes on the ingest machines at NCEP are not nearly large 
> enough to handle the amount of data flowing through them. When the lags get 
> large, products age off of their queues before the downstream sites have a 
> chance to get them.
> 
> We are a top level CONDUIT relay, so we are pulling data direct from NCEP and 
> we have been plagued with these missing products for several months now. 
> Unidata's servers are also generally missing the same products that we are .
> 
> The Unidata folks are also aware of the issue, but since NCEP runs the 
> servers, there's not much the Unidata folks can do other than let NCEP know 
> that they need larger queues. It sounds like NCEP is planning to address 
> this, but not until at least sometime in 2022.
> 
> The short term solution (and Tom from Unidata can correct me if I'm way off 
> base) is probably either
> 
> 
>   1.  get rid of the 0.25 degree data from the CONDUIT stream until they can 
> increase the queue sizes on the source machines
> 
> or
> 
>   1.  Set up a data feed separate from CONDUIT where one site (Unidata?) 
> downloads the data from the NCEP's server and injects it into a data feed 
> separate from CONDUIT - sort of a parallel feed of the data that is flowing 
> through CONDUIT.
> 
> This has been frustrating for a while now.. The GFS data coming through 
> CONDUIT is so unreliable that it is pretty useless as it is.
> 
> Pete
> 
> 
> <http://www.weather.com/tv/shows/wx-geeks/video/the-incredible-shrinking-cold-pool>-----
> Pete Pokrandt - Systems Programmer
> UW-Madison Dept of Atmospheric and Oceanic Sciences
> 608-262-3086  - address@hidden
> 
> ________________________________
> From: conduit <address@hidden> on behalf of Michael Leuthold <address@hidden>
> Sent: Wednesday, November 17, 2021 9:28 AM
> To: address@hidden <address@hidden>
> Subject: [conduit] Partial GFS files
> 
> Hello,
> 
>      Has anyone else had recent issues with missing GFS files? Over the
> past few weeks, I've had many that are short.  The problem  usually
> starts around forecast hour 100 and resolves around hour 200.  Both .25
> and .5 degree have this issue.  It's worse at 0 and 12Z.
> 
> thanks.
> 
> Mike
> 
> -rw-rw-r-- 1 ldm       ldm       523M Nov 17 04:16 2132100F081.grib2
> -rw-rw-r-- 1 ldm       ldm       523M Nov 17 03:57 2132100F084.grib2
> -rw-rw-r-- 1 ldm       ldm       522M Nov 17 04:17 2132100F087.grib2
> -rw-rw-r-- 1 ldm       ldm       524M Nov 17 03:59 2132100F090.grib2
> -rw-rw-r-- 1 ldm       ldm       521M Nov 17 04:00 2132100F093.grib2
> -rw-rw-r-- 1 ldm       ldm       526M Nov 17 04:01 2132100F096.grib2
> -rw-rw-r-- 1 ldm       ldm       127M Nov 17 04:18 2132100F099.grib2
> -rw-rw-r-- 1 ldm       ldm       379M Nov 17 04:19 2132100F102.grib2
> -rw-rw-r-- 1 ldm       ldm       521M Nov 17 04:03 2132100F105.grib2
> -rw-rw-r-- 1 ldm       ldm       314M Nov 17 04:20 2132100F108.grib2
> -rw-rw-r-- 1 ldm       ldm       519M Nov 17 04:04 2132100F111.grib2
> -rw-rw-r-- 1 ldm       ldm       209M Nov 17 04:22 2132100F114.grib2
> -rw-rw-r-- 1 ldm       ldm       524M Nov 17 04:06 2132100F117.grib2
> -rw-rw-r-- 1 ldm       ldm       528M Nov 17 04:07 2132100F120.grib2
> -rw-rw-r-- 1 ldm       ldm       181M Nov 17 04:24 2132100F123.grib2
> -rw-rw-r-- 1 ldm       ldm       160M Nov 17 04:24 2132100F126.grib2
> -rw-rw-r-- 1 ldm       ldm       524M Nov 17 04:08 2132100F129.grib2
> -rw-rw-r-- 1 ldm       ldm       525M Nov 17 04:09 2132100F132.grib2
> -rw-rw-r-- 1 ldm       ldm       317M Nov 17 04:25 2132100F135.grib2
> -rw-rw-r-- 1 ldm       ldm       524M Nov 17 04:10 2132100F138.grib2
> -rw-rw-r-- 1 ldm       ldm       474M Nov 17 04:27 2132100F141.grib2
> -rw-rw-r-- 1 ldm       ldm       522M Nov 17 04:28 2132100F144.grib2
> -rw-rw-r-- 1 ldm       ldm       220M Nov 17 04:29 2132100F147.grib2
> -rw-rw-r-- 1 ldm       ldm       309M Nov 17 04:31 2132100F150.grib2
> -rw-rw-r-- 1 ldm       ldm       519M Nov 17 04:14 2132100F153.grib2
> -rw-rw-r-- 1 ldm       ldm       149M Nov 17 04:32 2132100F156.grib2
> -rw-rw-r-- 1 ldm       ldm       519M Nov 17 04:15 2132100F159.grib2
> -rw-rw-r-- 1 ldm       ldm       252M Nov 17 04:35 2132100F162.grib2
> -rw-rw-r-- 1 ldm       ldm       515M Nov 17 04:16 2132100F165.grib2
> -rw-rw-r-- 1 ldm       ldm       520M Nov 17 04:17 2132100F168.grib2
> -rw-rw-r-- 1 ldm       ldm       498M Nov 17 04:38 2132100F171.grib2
> -rw-rw-r-- 1 ldm       ldm       520M Nov 17 04:19 2132100F174.grib2
> -rw-rw-r-- 1 ldm       ldm       517M Nov 17 04:19 2132100F177.grib2
> -rw-rw-r-- 1 ldm       ldm       355M Nov 17 04:39 2132100F180.grib2
> -rw-rw-r-- 1 ldm       ldm       427M Nov 17 04:40 2132100F183.grib2
> -rw-rw-r-- 1 ldm       ldm       519M Nov 17 04:22 2132100F186.grib2
> -rw-rw-r-- 1 ldm       ldm       515M Nov 17 04:23 2132100F189.grib2
> 
> --
> Mike Leuthold
> Manager, Regional Weather Modeling Program
> University of Arizona, Hydrology and Atmospheric Sciences
> 520-282-1478
> 
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
> 
> 
> conduit mailing list
> address@hidden
> For list information or to unsubscribe, visit: 
> https://www.unidata.ucar.edu/mailing_lists/

> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
> 
> 
> conduit mailing list
> address@hidden
> For list information or to unsubscribe, visit: 
> https://www.unidata.ucar.edu/mailing_lists/ 


_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


conduit mailing list
address@hidden
For list information or to unsubscribe, visit: 
https://www.unidata.ucar.edu/mailing_lists/