I see that I'm getting a message in gnome-system-monitor regarding futex wait.
I then did a search and found there is an issue with Haswell Processors (which
I have)and CentOS 6.6, and that futex will hang.
Brian
From: Magnuson.Erik [mailto:Magnuson.Erik@xxxxxxxxx]
Sent: Tuesday, March 29, 2016 12:15 PM
To: Brian Bernard <bbernard@xxxxxxxxxxxx>; awips2-users@xxxxxxxxxxxxxxxx
Subject: RE: High Latency in Ingest Logs
I'm suggesting that you run some of the normal system performance analysis
tools (top, systat, etc) see if there is something obvious like:
* Excessive CPU time
* Excessive "sys" time in top
* I/O saturation of one of the disk devices
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide
Certainly for older builds and older hardware, AWIPS2 on a single system seemed
to be mostly I/O limited. (AWIPS1 as well.) While you have better hardware,
the NOAAport feed and pqact.conf have also been increasing. The NWS runs
AWIPS2 on distributed HW which may help balance some of the I/O load. On older
HW, I ended up moving postgres and pypies to a different machine than EDEX/LDM
which helped.
--
Erik
From: Brian Bernard [mailto:bbernard@xxxxxxxxxxxx]
Sent: Tuesday, March 29, 2016 11:25 AM
To: Magnuson.Erik <Magnuson.Erik@xxxxxxxxx<mailto:Magnuson.Erik@xxxxxxxxx>>;
awips2-users@xxxxxxxxxxxxxxxx<mailto:awips2-users@xxxxxxxxxxxxxxxx>
Subject: RE: High Latency in Ingest Logs
Hi Erik,
The system I'm running EDEX Server has 64GB of RAM and four 256GB SSD. CentOS
6.6 is the operating system. Would doing a system update help this issue. (That
is, could it be an issue with the kernel?)
Brian
From: Magnuson.Erik [mailto:Magnuson.Erik@xxxxxxxxx]
Sent: Tuesday, March 29, 2016 11:09 AM
To: Brian Bernard <bbernard@xxxxxxxxxxxx<mailto:bbernard@xxxxxxxxxxxx>>;
awips2-users@xxxxxxxxxxxxxxxx<mailto:awips2-users@xxxxxxxxxxxxxxxx>
Subject: RE: High Latency in Ingest Logs
Another possible reason for this would be some resource starvation of that
ingest process:
? Excessive memory, swapping
? I/O bottlenecks (slow filesystem, large directory traversal w/some
filesystems)
? Insufficient threads in the pool for number of items to be processed
--
Erik
From:
awips2-users-bounces@xxxxxxxxxxxxxxxx<mailto:awips2-users-bounces@xxxxxxxxxxxxxxxx>
[mailto:awips2-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Brian Bernard
Sent: Tuesday, March 29, 2016 9:19 AM
To: awips2-users@xxxxxxxxxxxxxxxx<mailto:awips2-users@xxxxxxxxxxxxxxxx>
Subject: [awips2-users] High Latency in Ingest Logs
Hi,
I've been checking the ingest logs, to see what has been causing a frustrating
problem where my ingestGrib stops after a couple of days, and noticed in many
of the observation products such as raob and synop that I have a very high
latency. Here is an example:
INFO 2016-03-29 12:57:10,095 [Ingest.sfcobs-1] Ingest:
EDEX: Ingest - sfcobs::
/awips2/data_store/maritime/20160329/12/SXUS23_KWNB_291200_469498449.2016032912
processed in: 2.2350 (sec) Latency: 50.6560 (sec)
INFO 2016-03-29 13:08:05,361 [Ingest.goessounding-1]
Ingest: EDEX: Ingest - goessounding::
/awips2/data_store/goessounding/JUTX06_KNES_291306_469516938.bufr.2016032913
processed in: 0.0030 (sec) Latency: 67.1950 (sec)
INFO 2016-03-29 13:08:06,210 [Ingest.goessounding-1] Ingest: EDEX: Ingest -
goessounding::
/awips2/data_store/goessounding/JUTX06_KNES_291306_469516939.bufr.2016032913
processed in: 0.0030 (sec) Latency: 68.0450 (sec)
INFO 2016-03-29 13:08:10,327 [Ingest.goessounding-1] Ingest: EDEX: Ingest -
goessounding::
/awips2/data_store/goessounding/JUTX06_KNES_291306_469516940.bufr.2016032913
processed in: 0.0030 (sec) Latency: 72.1610 (sec)
INFO 2016-03-29 13:08:14,203 [Ingest.goessounding-1] Ingest: EDEX: Ingest -
goessounding::
/awips2/data_store/goessounding/JUTX06_KNES_291306_469516941.bufr.2016032913
processed in: 0.0040 (sec) Latency: 76.0380 (sec)
>From presusing an older document that Michael James put together, the issue
>with high latency is due QPID. Should I re-install QPID?
Thanks,
Brian Bernard
________________________________
The information contained in this email message is intended only for the use of
the individual(s) to whom it is addressed and may contain information that is
privileged and sensitive. If you are not the intended recipient, or otherwise
have received this communication in error, please notify the sender immediately
by email at the above referenced address and note that any further
dissemination, distribution or copying of this communication is strictly
prohibited.
The U.S. Export Control Laws regulate the export and re-export of technology
originating in the United States. This includes the electronic transmission of
information and software to foreign countries and to certain foreign nationals.
Recipient agrees to abide by these laws and their regulations -- including the
U.S. Department of Commerce Export Administration Regulations and the U.S.
Department of State International Traffic in Arms Regulations -- and not to
transfer, by electronic transmission or otherwise, any content derived from
this email to either a foreign national or a foreign destination in violation
of such laws.
