Re: [thredds] ncWMS GetCapabilities return sometimes empty document

Hi Jon, Heiko,

Jon, forgot to mention, we have code in the TDS WMS controller for
catching SocketException-s and ClientAbortException/IOException-s
similar to the change you reference.

Have either of you seen this with ncWMS? Or is it so far just showing up
in TDS?

Most of the references to this exception that I've found deal with
exceptions interrupting JSP processing and causing a switch to a JSP
error page that then can't create a session object. Since the problem
happens in one of our main JSP pages (capabilities_xml.jsp) and the
ClientAbortException-s are happening in a different thread, I'm leaning
towards the "thread in an invalid state" idea as well.

One thing we could try is to set our JSP pages to not create session
objects: "<%@ page session='false' ...%>". But since we don't know
if/how the threads might be invalid, I'm afraid that would just make
this problem even harder to see.

I'm going to try for a bit to get things setup so I can debug into
Tomcat code and see if I can catch this in action.

Ethan

On 3/17/2010 1:24 AM, Ethan Davis wrote:
> Hi Jon, Heiko,
> 
> I'm having a hard time reproducing the problem reliably. I've only seen
> it twice so far. And haven't been able to catch it in action. I did get
> the IllegalStateException from the Jasper JspFactoryImpl both times:
> 
>   "java.lang.IllegalStateException: Cannot create a session
>   after the response has been committed"
> 
> Googling on the above exception message brings up a fair number of hits
> but none of them seem totally relevant. Or when they sound somewhat
> relevant, there isn't a solution.
> 
> More tomorrow.
> 
> Ethan
> 
> On 3/16/2010 9:05 AM, Jonathan Blower wrote:
>> Hi Heiko, Ethan,
>>
>> I've been trying to figure out what might be going on here - thanks for
>> your great diagnosis Heiko.  I'm not sure what can be done in the
>> THREDDS/WMS code to prevent this, but I've modified some code in the
>> ncWMS WmsController class to attempt to catch exceptions that are thrown
>> when clients disconnect:
>>
>> http://www.resc.rdg.ac.uk/trac/ncWMS/browser/branches/tds-refactor-2010-
>> 01-09/src/java/uk/ac/rdg/resc/ncwms/controller/WmsController.java#L250
>>
>> Heiko - you mentioned that it's possible that poor exception handling
>> could cause a tomcat thread to be left in an invalid state.  I can't
>> reproduce your problem reliably so I can't test this myself, but I
>> wonder if you could find out whether this is happening in your setup?
>> The VisualVM tool
>> (http://java.sun.com/javase/6/docs/technotes/guides/visualvm/) can give
>> a thread dump from your running Tomcat system - perhaps you could run
>> this when an error occurs and see if anything is suspicious?
>>
>> Ethan - are you able to reproduce Heiko's problem?
>>
>> Cheers, Jon
>>
>>
>> -----Original Message-----
>> From: Ethan Davis [mailto:edavis@xxxxxxxxxxxxxxxx] 
>> Sent: 05 March 2010 23:32
>> To: Heiko Klein
>> Cc: Jonathan Blower; thredds@xxxxxxxxxxxxxxxx
>> Subject: Re: [thredds] ncWMS GetCapabilities return sometimes empty
>> document
>>
>> Hi Heiko,
>>
>> That is great. Thanks.
>>
>> I'm a bit swamped through next week but I will jump on this the
>> following week before I get to some other WMS work.
>>
>> Thanks,
>>
>> Ethan
>>
>> On 3/5/2010 4:44 AM, Heiko Klein wrote:
>>> Hi,
>>>
>>> finally, I managed to generate a simple case how to reproduce this
>> error:
>>>
>>> a) start tomcat
>>> b) abort fetching some maps (3-4 times on a tomcat with
>>> 10thredds/HTTP1.1 connector), i.e. call
>>> GET
>>>
>> 'http://localhost:8081/thredds/wms/data/met.no/hirlam12/wam_nsea.fc.2009
>> 0604.nc?REQUEST=GetMap&LAYERS=significant_wave_height&PALETTE=redblue&SE
>> RVICE=WMS&FORMAT=image/png&VERSION=1.3.0&CRS=EPSG:4326&BBOX=-30,-60,30,9
>> 0&WIDTH=400&HEIGHT=400&STYLES=BOXFILL/ncview'
>>>> /dev/null
>>> and abort this call before it is finished. (Ctrl-C)
>>> c) get some 'getCapabilities' documents:
>>> while [ 1 ]; do GET
>>>
>> 'http://localhost:8081/thredds/wms/data/met.no/hirlam12/wam_nsea.fc.2009
>> 0604.nc?service=WMS&version=1.3.0&request=GetCapabilities'
>>>> testCap.xml; ls -l testCap.xml; done
>>> ...
>>> -rw-r--r-- 1 heikok heikok 172126 2010-03-05 12:11 testCap.xml
>>> -rw-r--r-- 1 heikok heikok 172126 2010-03-05 12:11 testCap.xml
>>> -rw-r--r-- 1 heikok heikok 172126 2010-03-05 12:11 testCap.xml
>>> -rw-r--r-- 1 heikok heikok 0 2010-03-05 12:11 testCap.xml
>>> -rw-r--r-- 1 heikok heikok 172126 2010-03-05 12:11 testCap.xml
>>> -rw-r--r-- 1 heikok heikok 172126 2010-03-05 12:11 testCap.xml
>>>
>>>
>>> When the capabilites document with 0 byte size is fetched, the earlier
>>> mentioned exception is thrown.
>>>
>>>
>>> Since step b) is the crucial step to reproduce the 0 byte document, I
>>> guess there is somewhere a badly caught
>> ClientAbortException/IOException
>>> when writing to the responses OutputStream, leaving a tomcat-thread in
>>> an invalid state.
>>>
>>>
>>> We can solve this problem currently by adding a proxy in front of
>>> thredds (apache mod_proxy) which seems to swallow all
>>> client-connection-aborts before they reach tomcat.
>>>
>>> Best regards,
>>>
>>> Heiko
>>>
>>> On 2010-02-19 16:54, Heiko Klein wrote:
>>>> Hi,
>>>>
>>>> I'm still trying to track down this problem, so here a short update:
>>>>
>>>> The ncWMS/TDS crashes seems to be connected to concurrency: The
>> problem
>>>> occurs when reloading the wms-client (openlayers in firefox), that
>> means
>>>> GetCapabilites and GetMap are run in parallel. We're downloading
>> tiles,
>>>> so reloading will fetch one capability doc + approx 12 maps.
>>>>
>>>> * If I configure tomcat to only use one thread, and use the default
>>>> 'HTTP/1.1' connector, the problem disappears (but response-times are
>>>> very slow).
>>>> * Using 4 threads + 'HTTP/1.1', the problem appears very fast. (one
>> or
>>>> two reloads)
>>>> * Using the 'Nio' connector and 4 threads, the problem appears, but
>> not
>>>> so fast (3-4 reloads)
>>>> * Using the 'Nio' connector with only one thread, is about the same
>> as
>>>> Nio with several threads.
>>>>
>>>>
>>>> Best regards,
>>>>
>>>> Heiko
>>>>
>>>> _______________________________________________
>>>> thredds mailing list
>>>> thredds@xxxxxxxxxxxxxxxx
>>>> For list information or to unsubscribe,  visit:
>> http://www.unidata.ucar.edu/mailing_lists/ 
>>>
> 
> _______________________________________________
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit: 
> http://www.unidata.ucar.edu/mailing_lists/ 



  • 2010 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: