Hi Pete,
Since Friday we've been having a problem with our connections that feed
data from the supercomputer to our College Park data center. This is
causing various random files to be missing or hours late, and we are
working to get this fixed today.
I apologize that I did not send out a notice to the CONDUIT list. We were
fighting this starting Tuesday until late Friday at which time it was
decided we would put it down for the weekend and come back fresh today, and
I forgot to send an email to your list about the expected impacts with all
the other notification I was doing. I will modify our critical notification
procedure to include the CONDUIT email moving forward.
As I said we are working on the problem today but it is turning out to be a
thorny problem that has been hard to pin down so far. We do have a
mitigation that we could do but it would prevent us from further
troubleshooting so I can't put it in place at this time.
I will let you know when we find anything or if it will remain this way
overnight. I apologize for the inconvenience.
Thanks,
Anne
On Mon, Nov 25, 2019 at 12:08 PM 'Pete Pokrandt' via _NCEP
list.pmb-dataflow <ncep.list.pmb-dataflow@xxxxxxxx> wrote:
> Carissa and all at NCEP/NOAA,
>
> Not sure if you've seen any of the discussion happening on the
> conduit@xxxxxxxxxxxxxxxx list over the weekend, but beginning with the 18
> UTC model runs on Friday 11/22/2019, we have been seeing missing/very late
> delivery on CONDUIT of various random forecast hours.
>
> What I mean is, the forecast hours that come through are timely and
> complete, but there are certain forecast hours that just don't come through
> at all when the grib2 files are initially sent through CONDUIT. The
> forecast hour times that are missing/late are not consistent from run to
> run, but every NAM/GFS model data set since 18 UTC 11/22 has had this
> behavior. I have not checked other models.
>
> What is strange is 3, 8, 11, etc hours later, the files that were missing
> in the original broadcast will come through, with the data complete in
> them.
>
> Here's an example listing of the 06 UTC NAM 212 grid grib2 files [yes, I
> still prefix the file names as eta, even though it hasn't been the ETA
> model in a long time.. eta212.YYMMDDHH_HHH], one per forecast hour, showing
> in CST when the files arrived, followed by a similar lisitng of the 1 deg
> GFS files [gblav.YYMMDDHH_FHHH]. Notice how some of the files came in,
> complete, but much later than the rest of the run.
>
> Any ideas what might be causing this?
>
> The 12 UTC cycle this morning seems to be still having this problem, but
> so far with the lag on the missing files being more like 2-3 hours, maybe
> 30 minutes with the GFS
>
> Thanks,
> Pete
>
>
> NAM files from 06 UTC 25 run
>
> ls -l eta212.19112506*
> -rw-r--r--. 1 ldm ldm 10706782 Nov 25 01:34 eta212.19112506_F000
> -rw-r--r--. 1 ldm ldm 10850967 Nov 25 01:35 eta212.19112506_F001
> -rw-r--r--. 1 ldm ldm 10928390 Nov 25 01:36 eta212.19112506_F002
> -rw-r--r--. 1 ldm ldm 11074716 Nov 25 01:37 eta212.19112506_F003
> -rw-r--r--. 1 ldm ldm 11103363 Nov 25 01:37 eta212.19112506_F004
> -rw-r--r--. 1 ldm ldm 11199350 Nov 25 09:12 eta212.19112506_F005
> -rw-r--r--. 1 ldm ldm 11256869 Nov 25 01:40 eta212.19112506_F006
> -rw-r--r--. 1 ldm ldm 11341962 Nov 25 01:40 eta212.19112506_F007
> -rw-r--r--. 1 ldm ldm 11432550 Nov 25 01:41 eta212.19112506_F008
> -rw-r--r--. 1 ldm ldm 11405541 Nov 25 09:12 eta212.19112506_F009
> -rw-r--r--. 1 ldm ldm 11475304 Nov 25 01:43 eta212.19112506_F010
> -rw-r--r--. 1 ldm ldm 11485186 Nov 25 01:44 eta212.19112506_F011
> -rw-r--r--. 1 ldm ldm 11508093 Nov 25 01:45 eta212.19112506_F012
> -rw-r--r--. 1 ldm ldm 11568221 Nov 25 01:45 eta212.19112506_F013
> -rw-r--r--. 1 ldm ldm 11594775 Nov 25 01:46 eta212.19112506_F014
> -rw-r--r--. 1 ldm ldm 11633416 Nov 25 01:47 eta212.19112506_F015
> -rw-r--r--. 1 ldm ldm 11575325 Nov 25 01:48 eta212.19112506_F016
> -rw-r--r--. 1 ldm ldm 11602750 Nov 25 01:49 eta212.19112506_F017
> -rw-r--r--. 1 ldm ldm 11636219 Nov 25 01:50 eta212.19112506_F018
> -rw-r--r--. 1 ldm ldm 11588922 Nov 25 01:51 eta212.19112506_F019
> -rw-r--r--. 1 ldm ldm 11561516 Nov 25 01:52 eta212.19112506_F020
> -rw-r--r--. 1 ldm ldm 11582476 Nov 25 01:53 eta212.19112506_F021
> -rw-r--r--. 1 ldm ldm 11559156 Nov 25 01:54 eta212.19112506_F022
> -rw-r--r--. 1 ldm ldm 11552704 Nov 25 01:54 eta212.19112506_F023
> -rw-r--r--. 1 ldm ldm 11568099 Nov 25 01:56 eta212.19112506_F024
> -rw-r--r--. 1 ldm ldm 11555984 Nov 25 01:56 eta212.19112506_F025
> -rw-r--r--. 1 ldm ldm 11553377 Nov 25 01:57 eta212.19112506_F026
> -rw-r--r--. 1 ldm ldm 11543457 Nov 25 01:58 eta212.19112506_F027
> -rw-r--r--. 1 ldm ldm 11508141 Nov 25 01:58 eta212.19112506_F028
> -rw-r--r--. 1 ldm ldm 11568508 Nov 25 02:00 eta212.19112506_F029
> -rw-r--r--. 1 ldm ldm 11574458 Nov 25 02:01 eta212.19112506_F030
> -rw-r--r--. 1 ldm ldm 11596841 Nov 25 02:01 eta212.19112506_F031
> -rw-r--r--. 1 ldm ldm 11619896 Nov 25 02:02 eta212.19112506_F032
> -rw-r--r--. 1 ldm ldm 11609740 Nov 25 02:03 eta212.19112506_F033
> -rw-r--r--. 1 ldm ldm 11650583 Nov 25 02:04 eta212.19112506_F034
> -rw-r--r--. 1 ldm ldm 11652815 Nov 25 02:05 eta212.19112506_F035
> -rw-r--r--. 1 ldm ldm 11614441 Nov 25 02:07 eta212.19112506_F036
> -rw-r--r--. 1 ldm ldm 11619411 Nov 25 02:09 eta212.19112506_F039
> -rw-r--r--. 1 ldm ldm 11600874 Nov 25 02:11 eta212.19112506_F042
> -rw-r--r--. 1 ldm ldm 11567631 Nov 25 02:14 eta212.19112506_F045
> -rw-r--r--. 1 ldm ldm 11629227 Nov 25 02:17 eta212.19112506_F048
> -rw-r--r--. 1 ldm ldm 11635452 Nov 25 02:20 eta212.19112506_F051
> -rw-r--r--. 1 ldm ldm 11709421 Nov 25 09:13 eta212.19112506_F054
> -rw-r--r--. 1 ldm ldm 11730117 Nov 25 02:25 eta212.19112506_F057
> -rw-r--r--. 1 ldm ldm 11743457 Nov 25 02:28 eta212.19112506_F060
> -rw-r--r--. 1 ldm ldm 11812365 Nov 25 02:30 eta212.19112506_F063
> -rw-r--r--. 1 ldm ldm 11673062 Nov 25 02:31 eta212.19112506_F066
> -rw-r--r--. 1 ldm ldm 11613053 Nov 25 02:32 eta212.19112506_F069
> -rw-r--r--. 1 ldm ldm 11572871 Nov 25 02:34 eta212.19112506_F072
> -rw-r--r--. 1 ldm ldm 11662848 Nov 25 02:34 eta212.19112506_F075
> -rw-r--r--. 1 ldm ldm 11674272 Nov 25 02:36 eta212.19112506_F078
> -rw-r--r--. 1 ldm ldm 11737922 Nov 25 02:37 eta212.19112506_F081
> -rw-r--r--. 1 ldm ldm 11825375 Nov 25 02:38 eta212.19112506_F084
>
> 1 deg GFS files from 06 UTC 25 run
>
> ls -l gblav.19112506*
> -rw-r--r--. 1 ldm ldm 25948578 Nov 25 03:28 gblav.19112506_F000
> -rw-r--r--. 1 ldm ldm 28208595 Nov 25 09:14 gblav.19112506_F003
> -rw-r--r--. 1 ldm ldm 28487686 Nov 25 03:30 gblav.19112506_F006
> -rw-r--r--. 1 ldm ldm 28317150 Nov 25 03:31 gblav.19112506_F009
> -rw-r--r--. 1 ldm ldm 28503727 Nov 25 03:32 gblav.19112506_F012
> -rw-r--r--. 1 ldm ldm 28253875 Nov 25 09:14 gblav.19112506_F015
> -rw-r--r--. 1 ldm ldm 28537198 Nov 25 03:34 gblav.19112506_F018
> -rw-r--r--. 1 ldm ldm 28436967 Nov 25 09:14 gblav.19112506_F021
> -rw-r--r--. 1 ldm ldm 28307680 Nov 25 03:35 gblav.19112506_F024
> -rw-r--r--. 1 ldm ldm 28365362 Nov 25 03:36 gblav.19112506_F027
> -rw-r--r--. 1 ldm ldm 28425865 Nov 25 03:37 gblav.19112506_F030
> -rw-r--r--. 1 ldm ldm 28352127 Nov 25 03:38 gblav.19112506_F033
> -rw-r--r--. 1 ldm ldm 28226798 Nov 25 03:39 gblav.19112506_F036
> -rw-r--r--. 1 ldm ldm 28103177 Nov 25 03:40 gblav.19112506_F039
> -rw-r--r--. 1 ldm ldm 28374173 Nov 25 09:14 gblav.19112506_F042
> -rw-r--r--. 1 ldm ldm 28279555 Nov 25 03:41 gblav.19112506_F045
> -rw-r--r--. 1 ldm ldm 28469244 Nov 25 03:42 gblav.19112506_F048
> -rw-r--r--. 1 ldm ldm 28411426 Nov 25 03:43 gblav.19112506_F051
> -rw-r--r--. 1 ldm ldm 28582565 Nov 25 03:44 gblav.19112506_F054
> -rw-r--r--. 1 ldm ldm 28416114 Nov 25 03:44 gblav.19112506_F057
> -rw-r--r--. 1 ldm ldm 28468644 Nov 25 03:46 gblav.19112506_F060
> -rw-r--r--. 1 ldm ldm 28319256 Nov 25 03:46 gblav.19112506_F063
> -rw-r--r--. 1 ldm ldm 28390734 Nov 25 03:47 gblav.19112506_F066
> -rw-r--r--. 1 ldm ldm 28295272 Nov 25 03:48 gblav.19112506_F069
> -rw-r--r--. 1 ldm ldm 28482154 Nov 25 03:50 gblav.19112506_F072
> -rw-r--r--. 1 ldm ldm 28334833 Nov 25 03:51 gblav.19112506_F075
> -rw-r--r--. 1 ldm ldm 28454461 Nov 25 03:51 gblav.19112506_F078
> -rw-r--r--. 1 ldm ldm 28352212 Nov 25 03:52 gblav.19112506_F081
> -rw-r--r--. 1 ldm ldm 28484037 Nov 25 03:53 gblav.19112506_F084
> -rw-r--r--. 1 ldm ldm 28395249 Nov 25 03:54 gblav.19112506_F087
> -rw-r--r--. 1 ldm ldm 28559990 Nov 25 03:55 gblav.19112506_F090
> -rw-r--r--. 1 ldm ldm 28359489 Nov 25 03:56 gblav.19112506_F093
> -rw-r--r--. 1 ldm ldm 28402032 Nov 25 03:56 gblav.19112506_F096
> -rw-r--r--. 1 ldm ldm 28317115 Nov 25 03:57 gblav.19112506_F099
> -rw-r--r--. 1 ldm ldm 28446237 Nov 25 03:58 gblav.19112506_F102
> -rw-r--r--. 1 ldm ldm 28452633 Nov 25 03:58 gblav.19112506_F105
> -rw-r--r--. 1 ldm ldm 28523375 Nov 25 04:00 gblav.19112506_F108
> -rw-r--r--. 1 ldm ldm 28435342 Nov 25 04:00 gblav.19112506_F111
> -rw-r--r--. 1 ldm ldm 28460528 Nov 25 04:02 gblav.19112506_F114
> -rw-r--r--. 1 ldm ldm 28375935 Nov 25 04:03 gblav.19112506_F117
> -rw-r--r--. 1 ldm ldm 28660943 Nov 25 04:03 gblav.19112506_F120
> -rw-r--r--. 1 ldm ldm 28506890 Nov 25 04:04 gblav.19112506_F123
> -rw-r--r--. 1 ldm ldm 28571581 Nov 25 04:05 gblav.19112506_F126
> -rw-r--r--. 1 ldm ldm 28518364 Nov 25 04:05 gblav.19112506_F129
> -rw-r--r--. 1 ldm ldm 28501567 Nov 25 04:06 gblav.19112506_F132
> -rw-r--r--. 1 ldm ldm 28283418 Nov 25 04:07 gblav.19112506_F135
> -rw-r--r--. 1 ldm ldm 27803420 Nov 25 04:07 gblav.19112506_F138
> -rw-r--r--. 1 ldm ldm 27631946 Nov 25 04:08 gblav.19112506_F141
> -rw-r--r--. 1 ldm ldm 27558595 Nov 25 04:09 gblav.19112506_F144
> -rw-r--r--. 1 ldm ldm 27550949 Nov 25 04:10 gblav.19112506_F147
> -rw-r--r--. 1 ldm ldm 27113543 Nov 25 04:10 gblav.19112506_F150
> -rw-r--r--. 1 ldm ldm 27031939 Nov 25 04:11 gblav.19112506_F153
> -rw-r--r--. 1 ldm ldm 27005491 Nov 25 04:12 gblav.19112506_F156
> -rw-r--r--. 1 ldm ldm 26845869 Nov 25 04:13 gblav.19112506_F159
> -rw-r--r--. 1 ldm ldm 27016989 Nov 25 04:13 gblav.19112506_F162
> -rw-r--r--. 1 ldm ldm 26846814 Nov 25 09:16 gblav.19112506_F165
> -rw-r--r--. 1 ldm ldm 26969147 Nov 25 04:15 gblav.19112506_F168
> -rw-r--r--. 1 ldm ldm 26864444 Nov 25 04:15 gblav.19112506_F171
> -rw-r--r--. 1 ldm ldm 27035966 Nov 25 04:16 gblav.19112506_F174
> -rw-r--r--. 1 ldm ldm 26837093 Nov 25 04:17 gblav.19112506_F177
> -rw-r--r--. 1 ldm ldm 26907129 Nov 25 04:17 gblav.19112506_F180
> -rw-r--r--. 1 ldm ldm 26726151 Nov 25 04:18 gblav.19112506_F183
> -rw-r--r--. 1 ldm ldm 26912072 Nov 25 04:19 gblav.19112506_F186
> -rw-r--r--. 1 ldm ldm 26906480 Nov 25 04:20 gblav.19112506_F189
> -rw-r--r--. 1 ldm ldm 27001604 Nov 25 04:21 gblav.19112506_F192
> -rw-r--r--. 1 ldm ldm 26960131 Nov 25 04:22 gblav.19112506_F195
> -rw-r--r--. 1 ldm ldm 26966294 Nov 25 04:23 gblav.19112506_F198
> -rw-r--r--. 1 ldm ldm 27292235 Nov 25 04:23 gblav.19112506_F201
> -rw-r--r--. 1 ldm ldm 27783954 Nov 25 04:24 gblav.19112506_F204
> -rw-r--r--. 1 ldm ldm 28032221 Nov 25 04:25 gblav.19112506_F207
> -rw-r--r--. 1 ldm ldm 28233796 Nov 25 04:26 gblav.19112506_F210
> -rw-r--r--. 1 ldm ldm 28137999 Nov 25 04:26 gblav.19112506_F213
> -rw-r--r--. 1 ldm ldm 28270830 Nov 25 04:27 gblav.19112506_F216
> -rw-r--r--. 1 ldm ldm 28272067 Nov 25 04:28 gblav.19112506_F219
> -rw-r--r--. 1 ldm ldm 28456850 Nov 25 04:28 gblav.19112506_F222
> -rw-r--r--. 1 ldm ldm 28387918 Nov 25 04:29 gblav.19112506_F225
> -rw-r--r--. 1 ldm ldm 28504091 Nov 25 04:30 gblav.19112506_F228
> -rw-r--r--. 1 ldm ldm 28439894 Nov 25 04:31 gblav.19112506_F231
> -rw-r--r--. 1 ldm ldm 28491075 Nov 25 04:31 gblav.19112506_F234
> -rw-r--r--. 1 ldm ldm 28540531 Nov 25 04:32 gblav.19112506_F237
> -rw-r--r--. 1 ldm ldm 28736168 Nov 25 04:33 gblav.19112506_F240
> -rw-r--r--. 1 ldm ldm 28518449 Nov 25 04:34 gblav.19112506_F243
> -rw-r--r--. 1 ldm ldm 28440478 Nov 25 04:34 gblav.19112506_F246
> -rw-r--r--. 1 ldm ldm 28287326 Nov 25 04:35 gblav.19112506_F249
> -rw-r--r--. 1 ldm ldm 28398892 Nov 25 04:36 gblav.19112506_F252
> -rw-r--r--. 1 ldm ldm 28357124 Nov 25 04:36 gblav.19112506_F255
> -rw-r--r--. 1 ldm ldm 28431066 Nov 25 04:38 gblav.19112506_F258
> -rw-r--r--. 1 ldm ldm 28337230 Nov 25 04:38 gblav.19112506_F261
> -rw-r--r--. 1 ldm ldm 27982204 Nov 25 04:39 gblav.19112506_F264
> -rw-r--r--. 1 ldm ldm 27900819 Nov 25 04:40 gblav.19112506_F267
> -rw-r--r--. 1 ldm ldm 27904866 Nov 25 09:18 gblav.19112506_F270
> -rw-r--r--. 1 ldm ldm 27723533 Nov 25 04:41 gblav.19112506_F273
> -rw-r--r--. 1 ldm ldm 27762817 Nov 25 04:41 gblav.19112506_F276
> -rw-r--r--. 1 ldm ldm 27862896 Nov 25 04:43 gblav.19112506_F279
> -rw-r--r--. 1 ldm ldm 27785393 Nov 25 04:43 gblav.19112506_F282
> -rw-r--r--. 1 ldm ldm 27766632 Nov 25 04:44 gblav.19112506_F285
> -rw-r--r--. 1 ldm ldm 27902707 Nov 25 04:45 gblav.19112506_F288
> -rw-r--r--. 1 ldm ldm 27824193 Nov 25 04:45 gblav.19112506_F291
> -rw-r--r--. 1 ldm ldm 28370031 Nov 25 04:46 gblav.19112506_F294
> -rw-r--r--. 1 ldm ldm 28137794 Nov 25 04:47 gblav.19112506_F297
> -rw-r--r--. 1 ldm ldm 28142570 Nov 25 04:48 gblav.19112506_F300
> -rw-r--r--. 1 ldm ldm 28396951 Nov 25 04:49 gblav.19112506_F303
> -rw-r--r--. 1 ldm ldm 28616476 Nov 25 09:20 gblav.19112506_F306
> -rw-r--r--. 1 ldm ldm 28507791 Nov 25 04:50 gblav.19112506_F309
> -rw-r--r--. 1 ldm ldm 28645158 Nov 25 04:50 gblav.19112506_F312
> -rw-r--r--. 1 ldm ldm 28786653 Nov 25 04:52 gblav.19112506_F315
> -rw-r--r--. 1 ldm ldm 28813386 Nov 25 04:53 gblav.19112506_F318
> -rw-r--r--. 1 ldm ldm 28781423 Nov 25 04:53 gblav.19112506_F321
> -rw-r--r--. 1 ldm ldm 28929342 Nov 25 04:55 gblav.19112506_F324
> -rw-r--r--. 1 ldm ldm 28804112 Nov 25 04:55 gblav.19112506_F327
> -rw-r--r--. 1 ldm ldm 28867361 Nov 25 04:55 gblav.19112506_F330
> -rw-r--r--. 1 ldm ldm 28671022 Nov 25 04:56 gblav.19112506_F333
> -rw-r--r--. 1 ldm ldm 28726393 Nov 25 04:57 gblav.19112506_F336
> -rw-r--r--. 1 ldm ldm 28551409 Nov 25 04:58 gblav.19112506_F339
> -rw-r--r--. 1 ldm ldm 28583592 Nov 25 04:58 gblav.19112506_F342
> -rw-r--r--. 1 ldm ldm 28426808 Nov 25 04:59 gblav.19112506_F345
> -rw-r--r--. 1 ldm ldm 28490162 Nov 25 04:59 gblav.19112506_F348
> -rw-r--r--. 1 ldm ldm 28401839 Nov 25 05:00 gblav.19112506_F351
> -rw-r--r--. 1 ldm ldm 28653720 Nov 25 05:01 gblav.19112506_F354
> -rw-r--r--. 1 ldm ldm 28492451 Nov 25 05:01 gblav.19112506_F357
> -rw-r--r--. 1 ldm ldm 28577838 Nov 25 05:03 gblav.19112506_F360
> -rw-r--r--. 1 ldm ldm 28622702 Nov 25 05:03 gblav.19112506_F363
> -rw-r--r--. 1 ldm ldm 28635485 Nov 25 05:04 gblav.19112506_F366
> -rw-r--r--. 1 ldm ldm 28453420 Nov 25 05:05 gblav.19112506_F369
> -rw-r--r--. 1 ldm ldm 28466246 Nov 25 05:06 gblav.19112506_F372
> -rw-r--r--. 1 ldm ldm 28321178 Nov 25 05:06 gblav.19112506_F375
> -rw-r--r--. 1 ldm ldm 28385549 Nov 25 05:07 gblav.19112506_F378
> -rw-r--r--. 1 ldm ldm 28396144 Nov 25 05:07 gblav.19112506_F381
>
>
>
> <http://www.weather.com/tv/shows/wx-geeks/video/the-incredible-shrinking-cold-pool>
> -----
> Pete Pokrandt - Systems Programmer
> UW-Madison Dept of Atmospheric and Oceanic Sciences
> 608-262-3086 - poker@xxxxxxxxxxxx
> _______________________________________________
> Ncep.list.pmb-dataflow mailing list
> Ncep.list.pmb-dataflow@xxxxxxxxxxxxxxxxxxxx
> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.pmb-dataflow
>
--
Anne Myckow
WCCIS Data & Sciences Team Lead
NOAA Affiliate
301-683-3825
