Hi jay:
Debugging tomcat server performance is complicated.
The first question is, how many simultaneous requests are there? Each
will consume one thread, until request completes . So very slow
connections can shutdown a server; the default number of threads depends
on version i think maybe 200 for tomcat 7. This can be changed in
tomcat/conf/server.xml.
1. Start up the tomcat manager application and see what it has to say
about the connections.
2. Better, run jvisualvm on the same machine as tomcat (can also be done
remotely) and connect to the running tomcat server.
there are more complicated things to do; there are commercial versions
of tomcat that add visibility and performance metrics. You might also
google "tomcat performance monitoring tools".
you want to upgrade to Java 7; also make sure you are on latest version
of tomcat 7.
send me the threddsServlet.log file for one of the hours where the
server bogged down and ill see if theres anything obvious.
John
On 12/12/2013 10:47 AM, Gerry Creager - NOAA Affiliate wrote:
Jay,
What OS, release and Tomcat version are you running? I've seen a similar
issue on another piece of software (Ramadda). Since I've seen this
behavior with the standalone server and the tomcat-based server, I'm
beginning to suspect my Java installation, but have not had sufficient
time to investigate yet.
There may be an OS correlation here, so I'm interested. I'm running RHEL
6 and the various updated flavors of OpenJDK and Tomcat6.
gerry
On Wed, Dec 11, 2013 at 4:33 PM, Jay Alder <alderj@xxxxxxxxxxxxxxxxxxxx
<mailto:alderj@xxxxxxxxxxxxxxxxxxxx>> wrote:
Hi, we’ve recently released a web application that uses TDS for
mapping, which is getting a lot of traffic. At one point the server
stopped responding altogether, which is a major problem. A quick
restart of tomcat got it going again, so I’m starting to dig into
the logs. We normally get the GET / request complete behavior, but
occasionally we’ll have:
GET …url…
GET …url…
GET …url…
GET …url…
GET …url…
GET …url…
GET …url…
GET …url…
meanwhile having a 100% CPU spike (with 12 CPUs) for a minute or more
request compete
request compete
request compete
request cancelled by client
request cancelled by client
request compete
request compete
While watching the logs the few times I’ve seen this occur it seems
to pull out of it ok. However the time the server failed, requests
were never returned. From the logs, requests came in for roughly 40
minutes without being completed. Unfortunately do to the high
visibility we started to get emails from users and the press about
the application no longer working.
Has anyone experienced this before and/or can you give guidance on
how to diagnose or prevent this?
Here are some config settings:
CentOS 5.7
Java 1.6
TDS 4.3.17
only WMS is enabled
Java -Xmx set to 8Gb (currently taking 5.3, the dataset is 600 Gb of
30-arcsecond grids for the continental US, 3.4 Gb per file)
For better or worse we are configured to use 2 instances of TDS to
keep the catalogs and configuration isolated. I’m not sure if this
matters, but I didn’t want to omit it. Since it is a live server I
can’t easily change to the preferred proxy configuration.
I am trying not to panic yet. However, if the server goes
unresponsive again, staying calm may no longer be an option.
Jay Alder
US Geological Survey
Oregon State University
104 COAS Admin Building
Office Burt Hall 166
http://ceoas.oregonstate.edu/profile/alder/
_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
--
Gerry Creager
NSSL/CIMMS
405.325.6371
++++++++++++++++++++++
“Big whorls have little whorls,
That feed on their velocity;
And little whorls have lesser whorls,
And so on to viscosity.”
Lewis Fry Richardson (1881-1953)
_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
