[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20031125: ldm/craft data latencies at NWS Central Region HQ

Steve,
After our earlier phone call, I made the same changes on a couple of the other machines and the latencies on them are also way down. I've talked with Carl Sinclair at OU and we are going to make the changes on all boxes we have control of, and also notify the other CRAFT LDM sysadmins of the changes to make so that we can get the latencies down for all of the radars.
I think we didn't use this before because we had problems getting the pqing_bdds to work from within the ldmd.conf. I can't remember exactly what the problems were, but it may have simply been impatience on our part. I notice it often takes 20 seconds or more for the pqing_bdds process to start from within the ldmd.conf as an "exec" command.
Thank you VERY much for your assistance in solving this problem. I'm not sure we would have ever found this on our own, and I hope it will make all of our LDM feeds for the CRAFT data much more efficient and bring all of the latencies down. 

Steve Emmerson wrote:


Karen,

The cause of the large latencies from the LDM at the KBIS WFO was due
to the SIGCONT signals from the radar ingester not being delivered to
the LDM because the radar ingester was not in the process-group of the
LDM (SIGCONT is used to signal that a data-product has just been added
to the product-queue).  This caused the LDM to fallback to polling the
product-queue every 30 seconds -- resulting in the large latencies.
Based on the latency records in the file

   
/home/ldm/data/rtstats/CRAFT/20031126/ldmradar2.crh.noaa.gov/kbis.bis.noaa.gov_v_204.227.116.90

the latency statistics of this setup are the following:

   number of latency records:  1009
   minimum latency:               0 s
   maximum latency:              30 s
   mean latency:                 18.3657 s
   latency standard deviation:    7.1399 s

The solution was to move the execution of the radar ingester from
outside the LDM process-group to inside the LDM process-group by having
it executed via an EXEC entry in the KBIS LDM configuration-file
(ldmd.conf).

Here's what I did:

   * Changed /etc/init.d/ldm (the original file is "ldm.orig"):

       * Removed the execution of the radar ingester (pqing_bdds).

       * Followed the example at

           
http://my.unidata.ucar.edu/content/software/ldm/ldm-6.0.14/basics/configuring.html#boot

         for the "start" action.

   * Changed the LDM configuration-file (/home/ldm/etc/ldmd.conf) by
      adding an EXEC entry for the radar ingester (pqing_bdds).

   * Stopped the radar ingester by sending it a SIGTERM.

   * Restarted the LDM via an "ldmadmin stop" and "ldmadmin stat".

As of 2003-11-26 17:10:54 UTC, the KBIS LDM is using this new
configuration.

Based on the latency records in the previously-mentioned file,
the above chages have resulted in the following latency statistics:

   number of latency records:  48
   minimum latency:             0 s
   maximum latency:             1 s
   mean latency:                0.275862 s
   latency standard deviation:  0.450851 s

We strongly recommend that the configuration of all WFO LDM-s be changed
to match this new configuration of the LDM at KBIS.

Regards,
Steve Emmerson
Tom Yoksas


--
address@hidden
Phone:405-366-0434      
Cell:405-834-8559
CIMMS/University of Oklahoma
Affiliate of National Severe Storms Laboratory