[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Fwd: Re: problems reading GFS data from NOMADS-HD; DConnect]

Hi John,
The problem with SocketException still persists, looking like a network
problem. Sometimes it occurs only sporadically, sometimes we cannot read
GFS data for several hours (with exactly the same code that worked without
problems during the past months). Today for instance between 4:50 and
10:30 UTC in a number of tries from different computers (IP addresses
131.130.157.14, 131.130.157.15, 131,130,157.49), all of them got stuck at
some point reading the GFS data. The data were searched at:
"dods://nomads.ncep.noaa.gov:9090/dods/gfs_hd/gfs_hd20100429/gfs_hd_00z"
According to my system administrator our connections to the rest of the
world (like ECMWF, etc.) work fine, so there is a reason to assume that
the error could be on your side (especially in times of network
congestion). By the way, the problems ocur since the first day of the
volcano eruption in Island, and seem to be more frequent in times when a
new GFS run is available (and many users try to get the data at the same
time).
If you need more exact information to check your log files, please let me
know.
Hope some of you can help. Best Regards, Gerald

On Fr, 23.04.2010, 17:02, John Caron wrote:
> Hi Dan, Glenn:
>
> java.net.SocketException: Connection reset
>
> probably means that the socket was closed on the server.
>
> To go further with this we need to look at the threddsServlet.log files
> and see if we are getting any error messages in there that correspond to
> what Gerald is seeing. this may be hard without knowing the exact URL
> and a small time window during which the errors happened, and maybe the
> client IP address. Gerald do you have any of that?
>
> this can also happen when the connection gets very slow, eg if one of
> the IP routes is flaky. Gerald can you ask your network guys (if you
> have them) to test to see if your network routing to nomads has any
> problems?
>
> On 4/23/2010 6:59 AM, Dan Swank wrote:
>> John,
>> This addy is for NCDC-NOMADS servers nomads.ncdc.* only.
>> For NCEP-NOMADS ~ you should contact their HelpDesk: @
>> http://nomads.ncep.noaa.gov/txt_descriptions/Help_Desk_doc.shtml
>>
>> -Dan
>>
>>
>> address@hidden wrote the following on 4/22/2010 8:41 PM:
>>
>>> hmm-
>>> John
>>> can you assist here?
>>>
>>> thx, glenn
>>>
>>> ----- Original Message -----
>>> From: Gerald Spreitzhofer<address@hidden>
>>> Date: Thursday, April 22, 2010 4:28 pm
>>> Subject: Re: [Fwd: Re: problems reading GFS data from NOMADS-HD;
>>> DConnect]
>>>
>>>
>>>> Thanks, John.
>>>> Glenn, could you please answer my mail or explain me the error
>>>> messages I
>>>> am receiving from time to time (see below)? I think there were similar
>>>> messages some days in fall, some people from the mailing list
>>>> complainedand then you were able to fix them.
>>>> Best regards, Gerald
>>>>
>>>> On Do, 22.04.2010, 22:18, John Caron wrote:
>>>>
>>>>> gerald, send reports to
>>>>>
>>>>> address@hidden
>>>>>
>>>>>
>>>>>
>>>>> On 4/22/2010 12:54 PM, Gerald Spreitzhofer wrote:
>>>>>
>>>>>> John,
>>>>>> I am having the same problems as before and the same error
>>>>>>
>>>> message (see
>>>>
>>>>>> below, java.net.SocketException) most of the day, using
>>>>>>
>>>> DODS/OPENDAP.>>  Also
>>>>
>>>>>> just clicked again on the motherlode-link, there I can download GFS
>>>>>> 0.5deg
>>>>>> data (well, some gridpoints, I did not check for big amounts of
>>>>>>
>>>> data).>>  The
>>>>
>>>>>> amount of data I am downloading is max. 20 MB (like always), I
>>>>>>
>>>> don't>>  think
>>>>
>>>>>> this is a problem. The error usually ocurs some time during the
>>>>>> downloading, not just at the beginning.
>>>>>> Can you forward this mail to persons who might help?
>>>>>> Thanks, Gerald
>>>>>>
>>>>>> On Mo, 19.04.2010, 16:44, john caron wrote:
>>>>>>
>>>>>>
>>>>>>> gerald, can you try hitting datasets on
>>>>>>> http://motherlode.ucar.edu/thredds/ and see if you have any
>>>>>>>
>>>> problems?>>>
>>>>
>>>>>>> Gerald Spreitzhofer wrote:
>>>>>>>
>>>>>>>
>>>>>>>> Hi John,
>>>>>>>> Thanks for answering my mail last Friday that quickly.
>>>>>>>>
>>>> Problems still
>>>>
>>>>>>>> persist, but it looks like they are not permanent (sometimes in
>>>>>>>> between
>>>>>>>> GFS data can be read without problems). Maybe the error
>>>>>>>>
>>>> messages below
>>>>
>>>>>>>> help (they were produced this morning around 8 UTC).
>>>>>>>> Best Regards, Gerald
>>>>>>>>
>>>>>>>> ===== Reading 3d-variable 58: pressfc=====
>>>>>>>> java.net.SocketException: Connection reset
>>>>>>>>        at java.net.SocketInputStream.read(SocketInputStream.java:168)
>>>>>>>>        at
>>>>>>>>
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:183)>>>>
>>>>    at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
>>>>
>>>>>>>>        at
>>>>>>>>
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:277)>>>>
>>>>    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:741)
>>>>
>>>>>>>>        at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:702)
>>>>>>>>        at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:712)
>>>>>>>>        at
>>>>>>>>
>>>>>>>>
>>> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:583)>>>>
>>>     at dods.dap.DConnect.openConnection(DConnect.java:174)
>>>
>>>>>>>>        at dods.dap.DConnect.getDataFromUrl(DConnect.java:380)
>>>>>>>>        at dods.dap.DConnect.getData(DConnect.java:339)
>>>>>>>>        at dods.dap.DConnect.getData(DConnect.java:522)
>>>>>>>>        at
>>>>>>>>
>>> ucar.nc2.dods.DODSNetcdfFile.readDODSData(DODSNetcdfFile.java:536)>>>>
>>> at ucar.nc2.dods.DODSVariable.readDODSData(DODSVariable.java:497)
>>>
>>>>>>>>        at ucar.nc2.dods.DODSVariable.read(DODSVariable.java:393)
>>>>>>>>        at SEVIS.NetcdfUtil.readNcArrayLayer(NetcdfUtil.java:1846)
>>>>>>>>        at SEVIS.NetcdfUtil.readNcArray(NetcdfUtil.java:1708)
>>>>>>>>        at
>>>>>>>>
>>>> SEVIS.NetcdfUtil.readNcArray_0MedAdapted(NetcdfUtil.java:1659)>>>>
>>>>    at SEVIS.DODSReader.writeNcVariables(DODSReader.java:825)
>>>>
>>>>>>>>        at SEVIS.DODSReader.createNetcdfFile(DODSReader.java:676)
>>>>>>>>        at SEVIS.DODSReader.<init>(DODSReader.java:159)
>>>>>>>>        at SEVIS.SuperDataFile.getDodsFile(SuperDataFile.java:2526)
>>>>>>>>        at
>>>>>>>>
>>>>>>>>
>>> SEVIS.SuperDataFile.getSpecifiedNetcdfFile(SuperDataFile.java:2344)>>>>
>>>     at
>>> SEVIS.MainFrame.forecastUpdate(MainFrame.java:2131)
>>>
>>>>>>>>        at
>>>>>>>>
>>>>>>>>
>>>> SEVIS.MainFrame.jButtonDataUpdate_actionPerformed(MainFrame.java:2163)>>>>
>>>>
>>>>>>>> ------------------------ Ursprüngliche Nachricht
>>>>>>>> ------------------------
>>>>>>>> Betreff: Re: problems reading GFS data from NOMADS-HD; DConnect
>>>>>>>> Von:     "Glenn Rutledge"<address@hidden>
>>>>>>>> Datum:   Fr, 16.04.2010, 19:41
>>>>>>>> An:      "John Caron"<address@hidden>
>>>>>>>> Cc:      "Gerald Spreitzhofer"<address@hidden>
>>>>>>>>            address@hidden
>>>>>>>> ---------------------------------------------------------------
>>>>>>>>
>>>> -----------
>>>>
>>>>>>>> Thanks John,
>>>>>>>> We don't seem to be experiencing any issues.  I've cc:ed Dan
>>>>>>>>
>>>> Swank>>>>  here
>>>>
>>>>>>>> and he might be able to track down this issue.
>>>>>>>> Regards, Glenn
>>>>>>>>
>>>>>>>> John Caron said the following on 4/16/2010 1:01 PM:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Hi gerald: im not having any trouble with that URL. is it still
>>>>>>>>> happening (11 am MST) ?
>>>>>>>>>
>>>>>>>>> On 4/16/2010 10:15 AM, Gerald Spreitzhofer wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Hi John,
>>>>>>>>>> I am operationally reading GFS 0.5deg data from the NOMADS-
>>>>>>>>>>
>>>> HD server
>>>>
>>>>>>>>>> via
>>>>>>>>>> DODS and the Java/NetCDF library. I did not change anything
>>>>>>>>>>
>>>> in my
>>>>
>>>>>>>>>> code
>>>>>>>>>> during the last couple of months, but recently (for instance
>>>>>>>>>>
>>>> the>>>>>>  last
>>>>
>>>>>>>>>> 20
>>>>>>>>>> hours) there have been some errors reading data from there
>>>>>>>>>>
>>>> (see the
>>>>
>>>>>>>>>> protocol below). I think I remember that there was a similar
>>>>>>>>>>
>>>> problem>>>>>>  last
>>>>
>>>>>>>>>> fall, and the reason was a change in some security or firewall
>>>>>>>>>> setting on
>>>>>>>>>> NOMADS, just concerning this protocol. Do you know what the
>>>>>>>>>>
>>>> reason>>>>>>  for the
>>>>
>>>>>>>>>> messages can be or who might be responsible? Thanks, Gerald
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ===== Reading 3d-variable 28: hgtsfc=====
>>>>>>>>>> DConnect getData failed; retry (1,100)
>>>>>>>>>>
>>>>>>>>>>
>>> http://nomads.ncep.noaa.gov:9090/dods/gfs_hd/gfs_hd20100415/gfs_hd_18z.dods?hgtsfc[5:5][250:299][704:719]>>>>>>
>>>
>>>>>>>>>> DConnect getData failed; retry (2,200)
>>>>>>>>>>
>>>>>>>>>>
>>> http://nomads.ncep.noaa.gov:9090/dods/gfs_hd/gfs_hd20100415/gfs_hd_18z.dods?hgtsfc[5:5][250:299][704:719]>>>>>>
>>>
>>>>>>>>>> DConnect getData failed; retry (3,400)
>>>>>>>>>>
>>>>>>>>>>
>>> http://nomads.ncep.noaa.gov:9090/dods/gfs_hd/gfs_hd20100415/gfs_hd_18z.dods?hgtsfc[5:5][250:299][704:719]>>>>>>
>>>
>>>>>>>>>> DConnect getData failed; retry (4,800)
>>>>>>>>>>
>>>>>>>>>>
>>> http://nomads.ncep.noaa.gov:9090/dods/gfs_hd/gfs_hd20100415/gfs_hd_18z.dods?hgtsfc[5:5][250:299][704:719]>>>>>>
>>>
>>>>>>>>>> DConnect getData failed; retry (5,1600)
>>>>>>>>>>
>>>>>>>>>>
>>> http://nomads.ncep.noaa.gov:9090/dods/gfs_hd/gfs_hd20100415/gfs_hd_18z.dods?hgtsfc[5:5][250:299][704:719]>>>>>>
>>>
>>>>>>>>>> DConnect getData failed; retry (1,100)
>>>>>>>>>>
>>>>>>>>>>
>>> http://nomads.ncep.noaa.gov:9090/dods/gfs_hd/gfs_hd20100415/gfs_hd_18z.dods?hgtsfc[0:0][250:299][0:66]>>>>>>
>>>
>>>>>>>>>> DConnect getData failed; retry (2,200)
>>>>>>>>>>
>>>>>>>>>>
>>> http://nomads.ncep.noaa.gov:9090/dods/gfs_hd/gfs_hd20100415/gfs_hd_18z.dods?hgtsfc[0:0][250:299][0:66]>>>>>>
>>>
>>>>>>>>>> DConnect getData failed; retry (3,400)
>>>>>>>>>>
>>>>>>>>>>
>>> http://nomads.ncep.noaa.gov:9090/dods/gfs_hd/gfs_hd20100415/gfs_hd_18z.dods?hgtsfc[0:0][250:299][0:66]>>>>>>
>>>
>>>>>>>>>> DConnect getData failed; retry (4,800)
>>>>>>>>>>
>>>>>>>>>>
>>> http://nomads.ncep.noaa.gov:9090/dods/gfs_hd/gfs_hd20100415/gfs_hd_18z.dods?hgtsfc[0:0][250:299][0:66]>>>>>>
>>>
>>>>>>>>>> DConnect getData failed; retry (5,1600)
>>>>>>>>>>
>>>>>>>>>>
>>> http://nomads.ncep.noaa.gov:9090/dods/gfs_hd/gfs_hd20100415/gfs_hd_18z.dods?hgtsfc[0:0][250:299][0:66]>>>>>>
>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>> -----------------------------------------
>>>>>>>>>>>> Dr. Gerald Spreitzhofer, Researcher
>>>>>>>>>>>> Institute of Meteorology and Geophysics
>>>>>>>>>>>> University of Vienna
>>>>>>>>>>>> A-1090 Vienna, Althanstrasse 14, UZA2
>>>>>>>>>>>> Tel.: +43-1-4277-537 33
>>>>>>>>>>>> email: address@hidden
>>>>>>>>>>>> -----------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>
>
>
>