Since the last update, which involved testing only the ingest and decoding of CONDUIT 0.5/2.5 degree GFS, I've opened up the NGRID and NEXRAD3 feeds, as well as text and satellite products from the WMO and NIMAGE feeds, respectively.
The goal is to compare the speed of the grid decoder on high-resolution CONDUIT GFS runs alone versus running in parallel with the full nationwide NEXRAD3 feed and other products.
So far so good. I was using 8 grid decoder threads up until roughly April 26 0000 UTC, after which I increased the count to 12. While there is a noticeable decrease in the decoding latency (maxing out in the 1000-2000 second range rather than 2000-3000 seconds), this could just as well be caused by a reduction in file size of the GFS given the absence of activity in the atmosphere since April 26 compared to the previous week.
Regardless of the reason, we have shown that the payoff from increasing the thread count to 12 from 8 is diminishing: we're being limited more by the Raw Data Store write times than we are by the time it takes the grid decoder to process files.
Even so, the total decoding time is good: if 1500 seconds (25 minutes) is the longest latency time for the high-resolution GFS, we're in pretty good shape. For comparison, using GEMPAK grib2 decoders with the LDM, the typical setup for GEMPAK users these days, the entire 0.5 degree GFS run takes about 70 minutes to be fully decoded and available on disk. For AWIPS II EDEX, this time is roughly 90 minutes. That is encouraging.
