Re: Paper on THREDDS Data Server (TDS)

• To: Leilei Wang <Leilei.Wang@xxxxxxxx>
• Subject: Re: Paper on THREDDS Data Server (TDS)
• From: Ethan Davis <edavis@xxxxxxxxxxxxxxxx>
• Date: Wed, 16 Nov 2005 12:27:16 -0700

Hi Leilei,

Ethan

Leilei Wang wrote:

Yes! I've deleted and restalled the latest 3.3.05, it works great now! All problems are gone. Seems simply expanding thredds.war to overwrite old version always generates weird responses ( I got the impression somewhere in TDS documentaion saying that newer one can automatically overwrite old stuff).

Thank you,
Leilei

John Caron wrote:

Hi Leilei :

I have made a new release that hopefully will work, find it at the same place. It will have version 3.3.05. Im not sure how you are installing, but its a good idea to delete the expanded webapp/thredds directory to be sure you get the new stuff.

Leilei Wang wrote:

I'd already tried the latest v3.3 downloaded from your tech status page:

http://www.unidata.ucar.edu/projects/THREDDS/tech/index.html#TDS*
*TDS server just can't up and run at all:*

**type* Exception report

*message*

*description* _The server encountered an internal error () that prevented it from fulfilling this request._

*exception*

javax.servlet.ServletException: Servlet.init() for servlet ThreddsDefault threw exception org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462) org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118) org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160) org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799) org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705) org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577) org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)

    java.lang.Thread.run(Thread.java:534)

*root cause*

java.lang.NoSuchMethodError: ucar.nc2.iosp.grib.GribServiceProvider.extendIndex(Z)V

    thredds.servlet.ThreddsDefault.init(ThreddsDefault.java:55)
    javax.servlet.GenericServlet.init(GenericServlet.java:211)

org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462) org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118) org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160) org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799) org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705) org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577) org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)

    java.lang.Thread.run(Thread.java:534)

Leilei

Ethan Davis wrote:

Hi Leilei,

I grabbed the file mentioned below from your ftp site. Robb tried it out and could read it fine (and a few others I got from there).

We are not using TDS 3.2 on motherlode. We are using the code from our development (currently 3.3) branch. We're guessing there is a bug fix on that branch that didn't get into 3.2. If you are interested in trying that version, I can send you the thredds.war file. Since it isn't a stable release yet, we don't advise relying on it but it should be fine for testing and such.

Let me know if you want the 3.3 thredds.war file.

Ethan

Leilei Wang wrote:

Seems it is an unfortunate day for me.

I've reloaded correct GRIB-2 NDFD data files onto our TDS. Metadata seems fine with Netcdf Tool UI. But on "DODS Dataset Access Form", when I try to get ascii for a subset of some variables, I've got an error. I'm using latest official thredds.war 3.2 and even reinstalled tomcat with the version you suggested, still got no luck. Part of related error msg is followed. Seems array index is out of bound. I remember we got similar problem back to Aug and you've fixed it for other data. Weirdest thing is same data file(downloaded one NDFD file from your site)works on your motherlode while not working on our TDS. Are you using v3.2 with your motherlode thredds?

http://eclipse.ncdc.noaa.gov:9090/thredds/dodsC/ndfd/20040901/YGUZ99_KWBN_200409012154.html

Error {

   code = 0;

message = "Connection cannot be read http://eclipse.ncdc.noaa.gov:9090/thredds/dodsC/ndfd/20040901/YGUZ99_KWBN_200409012154.dods?Maximum_temperature[0:1:0][0:1:10][0:1:10]";;

};

2005-11-14T16:46:41.302 -0500 [ 2395772][ 348] INFO - thredds.servlet.ServletUtil - Remote host: 205.167.25.154 - Request: "GET /thredds/dodsC/ndfd/20040901/YGUZ99_KWBN_200409012154.dods?Maximum_temperature[0:1:0][0:1:10][0:1:10] HTTP/1.1" 2005-11-14T16:46:41.384 -0500 [ 2395854][ 348] ERROR - dods.servers.netcdf.NcSDArray - ERROR NcSDArray Read Maximum_temperature

java.lang.ArrayIndexOutOfBoundsException: 121
       at ucar.ma2.ArrayFloat.setFloat(ArrayFloat.java:193)
       at ucar.ma2.IteratorFast.setFloatNext(IteratorFast.java:64)

at ucar.nc2.iosp.grib.GribServiceProvider.readXY(GribServiceProvider.java:218) at ucar.nc2.iosp.grib.GribServiceProvider.readData(GribServiceProvider.java:177)

       at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:967)
       at ucar.nc2.Variable._read(Variable.java:786)
       at ucar.nc2.Variable.read(Variable.java:500)
       at dods.servers.netcdf.NcSDArray.read(NcSDArray.java:110)
       at dods.servers.netcdf.NcSDGrid.read(NcSDGrid.java:62)
       at dods.dap.Server.SDGrid.serialize(SDGrid.java:490)
       at dods.dap.Server.CEEvaluator.send(CEEvaluator.java:254)
       at dods.servlet.DODSServlet.doGetDODS(DODSServlet.java:651)
       at dods.servlet.DODSServlet.doGet(DODSServlet.java:1454)

at dods.servers.netcdf.NcDODSServlet.doGet(NcDODSServlet.java:291)

Thanks,
Leilei


Ethan Davis wrote:

Interesting. I wonder what is different about them. Can you send us (or make available for us to grab) an example file?

Thanks,

Ethan

Leilei Wang wrote:

Thanks, Ethan! I'm attaching the error messages. And also you can check our server for these data: http://eclipse.ncdc.noaa.gov:9090/thredds/dodsC/ndfd/20050928/catalog.html

This is the error message I've got when the opendap link is clicked.

***************************************************************

Error { code = 5; message = "NcDODSServlet ERROR opening NcDataset dodsC/ndfd/20050928/ndfd-all_020238_20050929_0000_fff.grb access = /raid/ftp/pub/ndfd/20050928/ndfd-all_020238_20050929_0000_fff.grb null"; };

*****************************************************************
And this is message in log file:
******************************************************************

2005-11-14T11:06:21.588 -0500 [ 331405962][ 268] INFO - thredds.servlet.ServletUtil - Remote host: 192.153.129.249 - Request: "GET /thredds/dodsC/ndfd/20050928/ndfd-all_020238_20050929_0000_fff.grb.html HTTP/1.1" 2005-11-14T11:06:21.653 -0500 [ 331406027][ 268] ERROR - dods.servers.netcdf.NcDODSServlet - NcDODSServlet ERROR opening NcDataset dodsC/ndfd/20050928/ndfd-all_020238_20050929_0000_fff.grb access = /raid/ftp/pub/ndfd/20050928/ndfd-all_020238_20050929_0000_fff.grb

null
java.lang.NullPointerException
       at ucar.grib.grib1.Grib1Lookup.<init>(Grib1Lookup.java:36)

at ucar.nc2.iosp.grib.Grib1ServiceProvider.open(Grib1ServiceProvider.java:72)

       at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:754)
       at ucar.nc2.NetcdfFile.open(NetcdfFile.java:334)
       at ucar.nc2.NetcdfFile.open(NetcdfFile.java:219)
       at ucar.nc2.NetcdfFile.open(NetcdfFile.java:165)

at ucar.nc2.util.NetcdfFileCache.acquire(NetcdfFileCache.java:161) at dods.servers.netcdf.GuardedDatasetImpl.<init>(GuardedDatasetImpl.java:53) at dods.servers.netcdf.NcDODSServlet.getDataset(NcDODSServlet.java:338)

       at dods.servlet.DODSServlet.doGetHTML(DODSServlet.java:951)
       at dods.servlet.DODSServlet.doGet(DODSServlet.java:1462)

at dods.servers.netcdf.NcDODSServlet.doGet(NcDODSServlet.java:291) at javax.servlet.http.HttpServlet.service(HttpServlet.java:689) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:868) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)

       at java.lang.Thread.run(Unknown Source)

2005-11-14T11:06:21.680 -0500 [ 331406054][ 268] INFO - thredds.servlet.ServletUtil - Request Completed - 200 - -1 - 92 *************************************************************************************

Seems it can't be opened at all and  Netcdf Tool verified this too.

Yes, GRIB-2 data files directly downloaded from NDFD work fine in TDS. But the files I've got problem are from NCDC archive system. I'm puzzled because these files seem in GRIB-1 format when I use 'degrib' decoder to decode them. And current 'degrib' can't handle them too because center & subcenter are not included in local use tables.

Thanks,
Leilei

Ethan Davis wrote:

Hi Leilei,

The GRIB data is being read by the netCDF-java 2.2 library which has a number of adapters for reading various file formats. I'm CCing Robb Kambic who developed and maintains our GRIB code. I just checked on our server and looks like we are serving NDFD data. Here's the catalog for the NDFD data we're serving in case you want to take a look at that

http://motherlode.ucar.edu:8080/thredds/idd/model/NDFD/CONUS_5km/catalog.xml

Are you getting any relavent messages in the TDS logs in the ${TOMCAT_HOME}/content/thredds/logs directory?

Another thing you could do is use the netCDF Tools UI to look at one of your data files and see if it can read it in. In the "Viewer" tab you can click on the folder icon to find a local file and try to open onf the your NDFD files. That way we'll know if it is a TDS or nc/GRIB issue. If the ToolsUI can't open the file, can you make one available so Robb can take a closer look?

Let us know what kind of error messages (if any) you get from the ToolsUI or in the TDS logs.

Thanks,

Ethan

Leilei Wang wrote:

Hi Ethan:

Which decoder does TDS use to decode GRIB files? Is it degrib? We're trying to serve NCDC archived NDFD data files through TDS which in GRIB but failed.

Thanks,
Leilei

Ethan Davis wrote:

Hi all,

We will be presenting a paper on the THREDDS Data Server (TDS) at the 2006 AMS meeting in Atlanta. It is available online now if you want to take a look before January.

http://www.unidata.ucar.edu/projects/THREDDS/PublicationsAndPresentations/2006AMS-TDS/

Ethan

--
Ethan R. Davis                                Telephone: (303) 497-8155
Software Engineer                             Fax:       (303) 497-8690
UCAR Unidata Program Center                   E-mail:    edavis@xxxxxxxx
P.O. Box 3000
Boulder, CO  80307-3000                       http://www.unidata.ucar.edu/
---------------------------------------------------------------------------

• References:
  • Paper on THREDDS Data Server (TDS)
    • From: Ethan Davis
  • Re: Paper on THREDDS Data Server (TDS)
    • From: John Caron