> On Jan 17, 2019, at 5:37 AM, Christian Skarby <christians@xxxxxx> wrote:
>
> For what it's worth; from what I've seen encoding errors would result in a
> reject from tomcat before the request is passed to the TDS application, and
> it would then probably not show up in the threddsServlet.log.
This is the behavior that we have seen.
Tomcat's rejection is an empty document and response headers ala:
HTTP/1.1 400 Bad Request
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked
Date: Thu, 17 Jan 2019 13:47:48 GMT
Connection: close
>
> Perhaphs Tom, you could try to just change the application code in your
> docker image (extract war or extracted thredds servlet from one version and
> then inject it in the other). I guess this should be possible, although I
> have not looked into the official docker images for thredds. If you make this
> fly it would be possible to say if it is the servlet or the other libraries
> in the docker image that causes you problems.
>
> Regarding the possible encoding errors: Tomcat introduced the more strict
> behavior in v9.0.8 (and also v8.5.31, v8.0.52 and v7.0.87).
>
> See the corresponding tomcat changelogs:
> • 62273: Implement configuration options to work-around specification
> non-compliant user agents (including all the major browsers) that do not
> correctly %nn encode URI paths and query strings as required by RFC 7230 and
> RFC 3986. (markt)
> See this thread in the google group for opendap.org:
> https://groups.google.com/a/opendap.org/forum/#!msg/support/ixTqhDXoLZQ/IT0lvZQ7CAAJ
>
> Tomcat-documentation:
> https://tomcat.apache.org/tomcat-9.0-doc/config/http.html
>
> relaxedPathChars
> The HTTP/1.1 specification requires that certain characters are %nn encoded
> when used in URI paths. Unfortunately, many user agents including all the
> major browsers are not compliant with this specification and use these
> characters in unencoded form. To prevent Tomcat rejecting such requests, this
> attribute may be used to specify the additional characters to allow. If not
> specified, no addtional characters will be allowed. The value may be any
> combination of the following characters: " < > [ \ ] ^ ` { | } . Any other
> characters present in the value will be ignored.
>
> relaxedQueryChars
> The HTTP/1.1 specification requires that certain characters are %nn encoded
> when used in URI query strings. Unfortunately, many user agents including all
> the major browsers are not compliant with this specification and use these
> characters in unencoded form. To prevent Tomcat rejecting such requests, this
> attribute may be used to specify the additional characters to allow. If not
> specified, no addtional characters will be allowed. The value may be any
> combination of the following characters: " < > [ \ ] ^ ` { | } . Any other
> characters present in the value will be ignored.
>
>
> As the OPeNDAP spesification (see section 6.1.1.2 on page 22) requires the
> use of [ and ] in the query string, and several clients doesn't quote these
> characters, we need to tweak the relaxedQueryChars configuration for tomcat.
For completeness I will add that the DAP protocol also utilizes the < and >
characters for formulating relational constraints, however these are limited to
constraining Sequence data types. I don't know that the TDS supports this data
type so they may not be required in the relaxedQueryChars list.
> Regards,
> Christian Skarby
>
> Den tor. 17. jan. 2019 kl. 05:42 skrev Thomas Cook <tmcook@xxxxxxxx>:
> I do not know how to debug the matlab or python clients, but this is
> what shows up in the threddsServlet.log
>
> Request: "GET
> /thredds/dodsC/HFR/USWC/6km/hourly/RTV/HFRADAR_US_West_Coast_6km_Resolution_Hourly_RTV_best.ncd.dods?lat,lon,site%5flat,site%5flon,site%5fcode,site%5fnetCode,procParams
> HTTP/1.0"
>
> Thanks
> Tom
>
> On Wed, Jan 16, 2019 at 7:47 PM Dennis Heimbigner <dmh@xxxxxxxx> wrote:
> >
> > Is there anyway you can see the exact URL being sent to the server?
> >
> > On 1/16/2019 8:24 PM, Thomas Cook wrote:
> > > Thanks Sean.
> > > The server is http://hfrnet-tds.ucsd.edu/thredds (although I just
> > > downgraded back to 4.6.12-SNAPSHOT). I will see about making my test
> > > server accessible to you, but that will take a little time to
> > > configure on my end.
> > > Let me know if you think I should open an issue on the github page and
> > > move the discussion there.
> > > Tom
> > >
> > > On Wed, Jan 16, 2019 at 6:32 PM Sean Arms <sarms@xxxxxxxx> wrote:
> > >> Hi Tom,
> > >>
> > >> So you are hitting up an opendap endpoint using all three libraries /
> > >> tools (python, matlab, and panoply)? Is this particular server publicly
> > >> accessible?
> > >>
> > >> I have a bad feeling that this is due to URL encoding (or lack thereof)
> > >> on the client side and changes made in tomcat 8.5, which it looks like
> > >> the thredds-docker repository bumped up to with the 4.6.12 release.
> > >>
> > >> Cheers,
> > >>
> > >> Sean
> > >>
> > >>
> > >> On Wed, Jan 16, 2019 at 7:06 PM Thomas Cook <tmcook@xxxxxxxx> wrote:
> > >>> HI I just wanted to add some details.
> > >>>
> > >>> The functionality works when I downgraded to the docker container
> > >>> 4.6.12-SNAPSHOT
> > >>>
> > >>> My data is part of a FMRC collection.
> > >>>
> > >>> Thanks,
> > >>> Tom
> > >>>
> > >>> On Wed, Jan 16, 2019 at 4:51 PM Thomas Cook <tmcook@xxxxxxxx> wrote:
> > >>>> HI
> > >>>> I just yesterday updated my thredds-docker to 4.6.12 from
> > >>>> 4.6.12-snapshot. It seems to be running fine, except I'm getting
> > >>>> errors using matlab ncread and python netcdf4.dataset. When I try to
> > >>>> access any of the multi-dimensional (time, lat, lon) variables I get
> > >>>> the following errors
> > >>>> with matlab:
> > >>>> Error using netcdflib
> > >>>> The NetCDF library encountered an error during execution of
> > >>>> 'getVarsShort' function - 'Access failure (-77)'.
> > >>>>
> > >>>> Error in netcdf.getVar (line 136)
> > >>>> data = netcdflib(funcstr,ncid,varid,varargin{:});
> > >>>>
> > >>>> Error in internal.matlab.imagesci.nc/read (line 635)
> > >>>> data = netcdf.getVar(gid, varid, ...
> > >>>>
> > >>>> Error in ncread (line 58)
> > >>>> vardata = ncObj.read(varName, varargin{:});
> > >>>>
> > >>>> Error in readUVfromTDS (line 56)
> > >>>> u=ncread(url, 'u_mean', [1 1 1], [2 2 2]);
> > >>>>
> > >>>>
> > >>>> and with python:
> > >>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.__getitem__()
> > >>>>
> > >>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable._get()
> > >>>>
> > >>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success()
> > >>>>
> > >>>> RuntimeError: NetCDF: Access failure
> > >>>>
> > >>>> I know that the latest docker contains netcdf 4.6, where as the
> > >>>> snapshot had 4.4. Could it be a version problem with the netCD4 python
> > >>>> and matlab libraries? I should be running the latest netcdf python
> > >>>> version and not too far behind with matlab versions.
> > >>>>
> > >>>> The weird things are that the index variables are accessible (time,
> > >>>> lat, lon etc), and the multiple-dimension data is accessible just
> > >>>> fine using the OpenDap web interface and Panoply. So I'm a little
> > >>>> confused. I'm addressing this list since I think there might be some
> > >>>> matlab and python expertise here and I think there were some messages
> > >>>> about the Access failure error in the past.
> > >>>>
> > >>>> Thanks,
> > >>>> Tom
> > >>> _______________________________________________
> > >>> NOTE: All exchanges posted to Unidata maintained email lists are
> > >>> recorded in the Unidata inquiry tracking system and made publicly
> > >>> available through the web. Users who post to any of the lists we
> > >>> maintain are reminded to remove any personal information that they
> > >>> do not want to be made public.
> > >>>
> > >>>
> > >>> thredds mailing list
> > >>> thredds@xxxxxxxxxxxxxxxx
> > >>> For list information or to unsubscribe, visit:
> > >>> http://www.unidata.ucar.edu/mailing_lists/
> > > _______________________________________________
> > > NOTE: All exchanges posted to Unidata maintained email lists are
> > > recorded in the Unidata inquiry tracking system and made publicly
> > > available through the web. Users who post to any of the lists we
> > > maintain are reminded to remove any personal information that they
> > > do not want to be made public.
> > >
> > >
> > > thredds mailing list
> > > thredds@xxxxxxxxxxxxxxxx
> > > For list information or to unsubscribe, visit:
> > > http://www.unidata.ucar.edu/mailing_lists/
> >
> > _______________________________________________
> > NOTE: All exchanges posted to Unidata maintained email lists are
> > recorded in the Unidata inquiry tracking system and made publicly
> > available through the web. Users who post to any of the lists we
> > maintain are reminded to remove any personal information that they
> > do not want to be made public.
> >
> >
> > thredds mailing list
> > thredds@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe, visit:
> > http://www.unidata.ucar.edu/mailing_lists/
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
= = =
Nathan Potter ndp at opendap.org
OPeNDAP, Inc. +1.541.231.3317
