Last month we received the first version of AWIPS II which included the new unified grib decoder (13.1.2). The install procedure for 13.1.2 was more complicated than usual - we needed the full 13.1.1 installation plus a 13.1.2 "update" - so around 8 GB of RPMs to manage.
This first version of the unified grib decoder had a number of problems, likely from the complicated installation procedure (we do it a little differently here compared to a operational forecast office).
Now with the full AWIPS II 13.2.1 release from the NWS, I can finally test our method for increasing the number of decoder threads on CONDUIT data. This is the same method used to handle ingest of the entire NEXRAD3 feed (~190 radar sites), and is described in detail on the page linked at the bottom of this post.
As delivered, AWIPS II will take roughly 4 hours to decode the high-resolution CONDUIT 0.5 degree GFS with 4 threads, and the message broker bottleneck would get worse with products simultaneously being decoded by other plugins (obs, satellite, radar, etc.)
Unidata's solution to this is to increase the allocated number of threads for the grid (formerly called grib) decoder, a task which involves either re-building the EDEX core RPMs, or editing the already-installed plugin-specific jar archives. I managed to incorporate the new decoder thread settings into an add-on RPM which I have added to the Unidata AWIPS II release (not available to the public at this time, sorry).
Initial results are promising: the total time to ingest and decode the 0.5 degree GFS on CONDUIT (25k files, ~3.6 GB) was just over 1 hour.
I let this run for a few days to make sure the Qpid message queue remained active as in the past I've noticed high-volume grib message decoding would sometimes bottle-neck the system even though the dataflow through-put was low. Here, it seems, the system can more than handle the 0.5/2.5 degree CONDUIT GFS.
Of course ingesting CONDUIT grids alone is one thing, ingesting them alongside point, satellite and radar data is another. That's the next step.
For more info on modifying the number of decoder threads, see: https://www.unidata.ucar.edu/staff/mjames/awips2/docs/threads.html
-Michael
