Pete,
Apologies for the issues. We have just determined that this is a disk
read/write issue which is why you see it as random. We have engaged our IT
folks to resolve this issue. We will let you know as soon as the problem is
fixed.
Carissa Klemmer
NCEP Central Operations
Production Management Branch Dataflow Team
301-683-3835
On Thu, Apr 23, 2015 at 3:29 PM, Pete Pokrandt <poker@xxxxxxxxxxxx> wrote:
> All,
>
> We have noticed a strange problem with the grids coming in for the 0.5
> degree GFS files. The problem started with the 12 UTC run of the 2015/04/22
> run, and several files (though not always the same ones) have been missing
> grids in each run since. The complete files contain 366 grids, the
> incomplete files always contain 327 and are always missing exactly the
> following grids:
>
> 5WAVH 5-Wave Geopotential Height [gpm]:500 mb
> ABSV Absolute Vorticity [1/s]:100 mb
> ABSV Absolute Vorticity [1/s]:975 mb
> CLWMR Cloud Mixing Ratio [kg/kg]:150 mb
> CLWMR Cloud Mixing Ratio [kg/kg]:550 mb
> CLWMR Cloud Mixing Ratio [kg/kg]:800 mb
> CSNOW Categorical Snow [-]:surface:
> DLWRF Downward Long-Wave Rad. Flux [W/m^2]:surface
> HGT Geopotential Height [gpm]:1000 mb
> HGT Geopotential Height [gpm]:300 mb
> HGT Geopotential Height [gpm]:50 mb
> HGT Geopotential Height [gpm]:highest tropospheric freezing level
> HGT Geopotential Height [gpm]:PV=2e-06 (Km^2/kg/s) surface
> HGT Geopotential Height [gpm]:tropopause
> HPBL Planetary Boundary Layer Height [m]:surface
> O3MR Ozone Mixing Ratio [kg/kg]:200 mb
> PRES Pressure [Pa]:low cloud top level:
> RH Relative Humidity [%]:400 mb
> TMAX Maximum Temperature [K]:2 m above ground:114-120 hour max fcst:
> TMP Temperature [K]:10 mb
> TMP Temperature [K]:350 mb
> TMP Temperature [K]:650 mb
> TMP Temperature [K]:900 mb
> TMP Temperature [K]:max wind
> TSOIL Soil Temperature Validation to deprecate [K]:1-2 m below ground
> UGRD U-Component of Wind [m/s]:1829 m above mean sea level
> UGRD U-Component of Wind [m/s]:70 mb
> VGRD V-Component of Wind [m/s]:20 mb
> VGRD V-Component of Wind [m/s]:500 mb
> VGRD V-Component of Wind [m/s]:750 mb
> VGRD V-Component of Wind [m/s]:950 mb
> V-GWD Meridional Flux of Gravity Wave Stress [N/m^2]:surface:
> VVEL Vertical Velocity (Pressure) [Pa/s]:0.995 sigma level
> VVEL Vertical Velocity (Pressure) [Pa/s]:450 mb
> VVEL Vertical Velocity (Pressure) [Pa/s]:700 mb
> VVEL Vertical Velocity (Pressure) [Pa/s]:925 mb
>
>
> I don't think it's a latency issue for us, because our latencies for
> conduit have been < 30 seconds or so for at least the past 3 days.
>
> One curious thing, the files are complete on motherlode.ucar.edu, but the
> files that we have as short, have a later modification time than the rest .
> For example, for the 12 UTC 23 run, we were missing grids for the F021,
> F102, F123 and F252 times. Here's a subset of the times on motherlode that
> shows these files:
>
> gfs.t12z.pgrb2.0p50.f018
> <http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f018>
> 23-Apr-2015 09:35 67M gfs.t12z.pgrb2.0p50.f021
> <http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f021>
> 23-Apr-2015 09:49 67M gfs.t12z.pgrb2.0p50.f024
> <http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f024>
> 23-Apr-2015 09:37 66M
> gfs.t12z.pgrb2.0p50.f099
> <http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f099>
> 23-Apr-2015 10:00 66M gfs.t12z.pgrb2.0p50.f102
> <http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f102>
> 23-Apr-2015 10:42 67M gfs.t12z.pgrb2.0p50.f105
> <http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f105>
> 23-Apr-2015 10:03 66M gfs.t12z.pgrb2.0p50.f108
> <http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f108>
> 23-Apr-2015 10:03 66M gfs.t12z.pgrb2.0p50.f111
> <http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f111>
> 23-Apr-2015 10:04 66M gfs.t12z.pgrb2.0p50.f114
> <http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f114>
> 23-Apr-2015 10:05 66M gfs.t12z.pgrb2.0p50.f117
> <http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f117>
> 23-Apr-2015 10:06 66M gfs.t12z.pgrb2.0p50.f120
> <http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f120>
> 23-Apr-2015 10:07 66M gfs.t12z.pgrb2.0p50.f123
> <http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f123>
> 23-Apr-2015 10:51 66M gfs.t12z.pgrb2.0p50.f126
> <http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f126>
> 23-Apr-2015 10:09 67M gfs.t12z.pgrb2.0p50.f129
> <http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f129>
> 23-Apr-2015 10:10 66M
> gfs.t12z.pgrb2.0p50.f237
> <http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f237>
> 23-Apr-2015 10:43 66M gfs.t12z.pgrb2.0p50.f240
> <http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f240>
> 23-Apr-2015 10:44 67M gfs.t12z.pgrb2.0p50.f252
> <http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f252>
> 23-Apr-2015 11:32 62M gfs.t12z.pgrb2.0p50.f264
> <http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f264>
> 23-Apr-2015 10:49 62M gfs.t12z.pgrb2.0p50.f276
> <http://motherlode.ucar.edu/native/conduit/data/nccf/com/gfs/prod/gfs.2015042312/gfs.t12z.pgrb2.0p50.f276>
> 23-Apr-2015 10:50 63M
>
>
> We are feeding redundantly from conduit.ncep.noaa.gov and
> ncepldm4.woc.noaa.gov
>
> Ideas?
>
> Here is a (I think) complete list of the times that had short files since
> 12UTC 4/22. I'm guessing all of these files are complete on motherlode, but
> have a later modification time than the rest of the run.
>
> 2015042212_F081
> 2015042212_F112
> 2015042212_F141
> 2015042212_F156
> 2015042212_F159
> 2015042212_F195
> 2015042212_F228
>
> 2015042218_F063
> 2015042218_F081
> 2015042218_F198
>
> 2015042300_F006
> 2015042300_F018
> 2015042300_F021
> 2015042300_F087
> 2015042300_F114
> 2015042300_F120
> 2015042300_F138
>
> 2015042306_F006
> 2015042306_F069
>
> 2015042312_F021
> 2015042312_F102
> 2015042312_F123
> 2015042312_F252
>
>
>
> Pete
>
>
>
>
>
>
> --
> Pete Pokrandt - Systems Programmer
> UW-Madison Dept of Atmospheric and Oceanic Sciences608-262-3086 -
> poker@xxxxxxxxxxxx
>
>
> _______________________________________________
> conduit mailing list
> conduit@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
