Sean,
Interesting.
I thought that mod_jk had been abandoned/superseded for/by the Apache JServ
Protocol (AJP) protocol with the introduction of Apache 2.x
N
> On Jan 18, 2019, at 9:21 AM, Sean Arms <sarms@xxxxxxxx> wrote:
>
> Hi Nathan, all,
>
> netCDF-Java does proper encoding as of 4.6.12 and 5.0.0-beta6 (runtime
> configurable), and in the past it encoded as well - there was a period of
> time in the 4.5 and 4.6 line of development in which the encoding was
> skipped, but that's no longer the case.
>
> Interestingly enough, if you run Apache with mod_jk, non-encoded urls will be
> accepted and passed along to tomcat in a usable form without issue, but I
> have no idea how long that will last.
>
> Sean
>
> On Thu, Jan 17, 2019 at 4:25 PM Nathan Potter <ndp@xxxxxxxxxxx> wrote:
> Hi Thomas,
>
> I believe that I worked with Robert Schmunk (the author of Panoply) on this
> very problem and I think the reason he is using netcdfAll-5.0.0-SNAPSHOT is
> that it solves the outgoing encoding issue on the client side. Is that right
> Robert? (We may not hear back from him as I suspect he's furloughed and
> unavailable a.t.m.)
>
> If the client cannot be cajoled into producing the correct URL encoding then
> I think the work falls back to relaxing the server's relaxedQueryChars
> configuration.
>
> Which is certainly less than ideal.
>
> And to end on a cheery note - once the netcdf-java library is release it can
> take some time (up to a year) before it gets rolled into a Matlab release.
>
> Sorry I can't be more helpful.
>
>
> Sincerely,
>
> Nathan
>
>
> > On Jan 17, 2019, at 1:39 PM, Thomas Cook <tmcook@xxxxxxxx> wrote:
> >
> > OK Given Nathan and Christian's comments, I'm focussing my
> > troubleshooting on client side.
> > What I've learned
> > 1) Matlab's native netCDF toolbox depends on netCDF 4.3.3.1. I
> > couldn't find an easy way to update the netCDF version inside my
> > Matlab install.
> > 2) I updated the netcdf4-python module to use netCDF 4.6.12. This did
> > not solve the issue.
> > 3) Panoply (which works) is using netcdfAll-5.0.0-SNAPSHOT. I couldn't
> > find an older version to download, nor was I able to run panoply with
> > an older netcdfAll.jar
> > 4) Another user has reported success using the SNTOOLS/MEXNC package
> > to read the files. This also uses a version of netcdfAll.jar for its
> > decoding. I can't verify this yet.
> >
> > So not sure there's much more I can do with on the client side. I'll
> > mess with the Dockerfile like Christian suggested, and try using some
> > non FMRC collections to test with the clients. If anyone has any
> > ideas, I'm open!
> > Thanks to all who have replied and I'll update if I make any headway.
> > Tom
> >
> > On Thu, Jan 17, 2019 at 5:53 AM Nathan Potter <ndp@xxxxxxxxxxx> wrote:
> >>
> >>
> >>
> >>> On Jan 17, 2019, at 5:37 AM, Christian Skarby <christians@xxxxxx> wrote:
> >>>
> >>> For what it's worth; from what I've seen encoding errors would result in
> >>> a reject from tomcat before the request is passed to the TDS application,
> >>> and it would then probably not show up in the threddsServlet.log.
> >>
> >> This is the behavior that we have seen.
> >>
> >> Tomcat's rejection is an empty document and response headers ala:
> >>
> >> HTTP/1.1 400 Bad Request
> >> Server: Apache-Coyote/1.1
> >> Transfer-Encoding: chunked
> >> Date: Thu, 17 Jan 2019 13:47:48 GMT
> >> Connection: close
> >>
> >>>
> >>> Perhaphs Tom, you could try to just change the application code in your
> >>> docker image (extract war or extracted thredds servlet from one version
> >>> and then inject it in the other). I guess this should be possible,
> >>> although I have not looked into the official docker images for thredds.
> >>> If you make this fly it would be possible to say if it is the servlet or
> >>> the other libraries in the docker image that causes you problems.
> >>>
> >>> Regarding the possible encoding errors: Tomcat introduced the more strict
> >>> behavior in v9.0.8 (and also v8.5.31, v8.0.52 and v7.0.87).
> >>>
> >>> See the corresponding tomcat changelogs:
> >>> • 62273: Implement configuration options to work-around
> >>> specification non-compliant user agents (including all the major
> >>> browsers) that do not correctly %nn encode URI paths and query strings as
> >>> required by RFC 7230 and RFC 3986. (markt)
> >>> See this thread in the google group for opendap.org:
> >>> https://groups.google.com/a/opendap.org/forum/#!msg/support/ixTqhDXoLZQ/IT0lvZQ7CAAJ
> >>>
> >>> Tomcat-documentation:
> >>> https://tomcat.apache.org/tomcat-9.0-doc/config/http.html
> >>>
> >>> relaxedPathChars
> >>> The HTTP/1.1 specification requires that certain characters are %nn
> >>> encoded when used in URI paths. Unfortunately, many user agents including
> >>> all the major browsers are not compliant with this specification and use
> >>> these characters in unencoded form. To prevent Tomcat rejecting such
> >>> requests, this attribute may be used to specify the additional characters
> >>> to allow. If not specified, no addtional characters will be allowed. The
> >>> value may be any combination of the following characters: " < > [ \ ] ^ `
> >>> { | } . Any other characters present in the value will be ignored.
> >>>
> >>> relaxedQueryChars
> >>> The HTTP/1.1 specification requires that certain characters are %nn
> >>> encoded when used in URI query strings. Unfortunately, many user agents
> >>> including all the major browsers are not compliant with this
> >>> specification and use these characters in unencoded form. To prevent
> >>> Tomcat rejecting such requests, this attribute may be used to specify the
> >>> additional characters to allow. If not specified, no addtional characters
> >>> will be allowed. The value may be any combination of the following
> >>> characters: " < > [ \ ] ^ ` { | } . Any other characters present in the
> >>> value will be ignored.
> >>>
> >>>
> >>> As the OPeNDAP spesification (see section 6.1.1.2 on page 22) requires
> >>> the use of [ and ] in the query string, and several clients doesn't quote
> >>> these characters, we need to tweak the relaxedQueryChars configuration
> >>> for tomcat.
> >>
> >> For completeness I will add that the DAP protocol also utilizes the < and
> >> > characters for formulating relational constraints, however these are
> >> limited to constraining Sequence data types. I don't know that the TDS
> >> supports this data type so they may not be required in the
> >> relaxedQueryChars list.
> >>
> >>
> >>> Regards,
> >>> Christian Skarby
> >>>
> >>> Den tor. 17. jan. 2019 kl. 05:42 skrev Thomas Cook <tmcook@xxxxxxxx>:
> >>> I do not know how to debug the matlab or python clients, but this is
> >>> what shows up in the threddsServlet.log
> >>>
> >>> Request: "GET
> >>> /thredds/dodsC/HFR/USWC/6km/hourly/RTV/HFRADAR_US_West_Coast_6km_Resolution_Hourly_RTV_best.ncd.dods?lat,lon,site%5flat,site%5flon,site%5fcode,site%5fnetCode,procParams
> >>> HTTP/1.0"
> >>>
> >>> Thanks
> >>> Tom
> >>>
> >>> On Wed, Jan 16, 2019 at 7:47 PM Dennis Heimbigner <dmh@xxxxxxxx> wrote:
> >>>>
> >>>> Is there anyway you can see the exact URL being sent to the server?
> >>>>
> >>>> On 1/16/2019 8:24 PM, Thomas Cook wrote:
> >>>>> Thanks Sean.
> >>>>> The server is http://hfrnet-tds.ucsd.edu/thredds (although I just
> >>>>> downgraded back to 4.6.12-SNAPSHOT). I will see about making my test
> >>>>> server accessible to you, but that will take a little time to
> >>>>> configure on my end.
> >>>>> Let me know if you think I should open an issue on the github page and
> >>>>> move the discussion there.
> >>>>> Tom
> >>>>>
> >>>>> On Wed, Jan 16, 2019 at 6:32 PM Sean Arms <sarms@xxxxxxxx> wrote:
> >>>>>> Hi Tom,
> >>>>>>
> >>>>>> So you are hitting up an opendap endpoint using all three libraries /
> >>>>>> tools (python, matlab, and panoply)? Is this particular server
> >>>>>> publicly accessible?
> >>>>>>
> >>>>>> I have a bad feeling that this is due to URL encoding (or lack
> >>>>>> thereof) on the client side and changes made in tomcat 8.5, which it
> >>>>>> looks like the thredds-docker repository bumped up to with the 4.6.12
> >>>>>> release.
> >>>>>>
> >>>>>> Cheers,
> >>>>>>
> >>>>>> Sean
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Jan 16, 2019 at 7:06 PM Thomas Cook <tmcook@xxxxxxxx> wrote:
> >>>>>>> HI I just wanted to add some details.
> >>>>>>>
> >>>>>>> The functionality works when I downgraded to the docker container
> >>>>>>> 4.6.12-SNAPSHOT
> >>>>>>>
> >>>>>>> My data is part of a FMRC collection.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Tom
> >>>>>>>
> >>>>>>> On Wed, Jan 16, 2019 at 4:51 PM Thomas Cook <tmcook@xxxxxxxx> wrote:
> >>>>>>>> HI
> >>>>>>>> I just yesterday updated my thredds-docker to 4.6.12 from
> >>>>>>>> 4.6.12-snapshot. It seems to be running fine, except I'm getting
> >>>>>>>> errors using matlab ncread and python netcdf4.dataset. When I try to
> >>>>>>>> access any of the multi-dimensional (time, lat, lon) variables I get
> >>>>>>>> the following errors
> >>>>>>>> with matlab:
> >>>>>>>> Error using netcdflib
> >>>>>>>> The NetCDF library encountered an error during execution of
> >>>>>>>> 'getVarsShort' function - 'Access failure (-77)'.
> >>>>>>>>
> >>>>>>>> Error in netcdf.getVar (line 136)
> >>>>>>>> data = netcdflib(funcstr,ncid,varid,varargin{:});
> >>>>>>>>
> >>>>>>>> Error in internal.matlab.imagesci.nc/read (line 635)
> >>>>>>>> data = netcdf.getVar(gid, varid, ...
> >>>>>>>>
> >>>>>>>> Error in ncread (line 58)
> >>>>>>>> vardata = ncObj.read(varName, varargin{:});
> >>>>>>>>
> >>>>>>>> Error in readUVfromTDS (line 56)
> >>>>>>>> u=ncread(url, 'u_mean', [1 1 1], [2 2 2]);
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> and with python:
> >>>>>>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.__getitem__()
> >>>>>>>>
> >>>>>>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable._get()
> >>>>>>>>
> >>>>>>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success()
> >>>>>>>>
> >>>>>>>> RuntimeError: NetCDF: Access failure
> >>>>>>>>
> >>>>>>>> I know that the latest docker contains netcdf 4.6, where as the
> >>>>>>>> snapshot had 4.4. Could it be a version problem with the netCD4
> >>>>>>>> python
> >>>>>>>> and matlab libraries? I should be running the latest netcdf python
> >>>>>>>> version and not too far behind with matlab versions.
> >>>>>>>>
> >>>>>>>> The weird things are that the index variables are accessible (time,
> >>>>>>>> lat, lon etc), and the multiple-dimension data is accessible just
> >>>>>>>> fine using the OpenDap web interface and Panoply. So I'm a little
> >>>>>>>> confused. I'm addressing this list since I think there might be some
> >>>>>>>> matlab and python expertise here and I think there were some messages
> >>>>>>>> about the Access failure error in the past.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Tom
> >>>>>>> _______________________________________________
> >>>>>>> NOTE: All exchanges posted to Unidata maintained email lists are
> >>>>>>> recorded in the Unidata inquiry tracking system and made publicly
> >>>>>>> available through the web. Users who post to any of the lists we
> >>>>>>> maintain are reminded to remove any personal information that they
> >>>>>>> do not want to be made public.
> >>>>>>>
> >>>>>>>
> >>>>>>> thredds mailing list
> >>>>>>> thredds@xxxxxxxxxxxxxxxx
> >>>>>>> For list information or to unsubscribe, visit:
> >>>>>>> http://www.unidata.ucar.edu/mailing_lists/
> >>>>> _______________________________________________
> >>>>> NOTE: All exchanges posted to Unidata maintained email lists are
> >>>>> recorded in the Unidata inquiry tracking system and made publicly
> >>>>> available through the web. Users who post to any of the lists we
> >>>>> maintain are reminded to remove any personal information that they
> >>>>> do not want to be made public.
> >>>>>
> >>>>>
> >>>>> thredds mailing list
> >>>>> thredds@xxxxxxxxxxxxxxxx
> >>>>> For list information or to unsubscribe, visit:
> >>>>> http://www.unidata.ucar.edu/mailing_lists/
> >>>>
> >>>> _______________________________________________
> >>>> NOTE: All exchanges posted to Unidata maintained email lists are
> >>>> recorded in the Unidata inquiry tracking system and made publicly
> >>>> available through the web. Users who post to any of the lists we
> >>>> maintain are reminded to remove any personal information that they
> >>>> do not want to be made public.
> >>>>
> >>>>
> >>>> thredds mailing list
> >>>> thredds@xxxxxxxxxxxxxxxx
> >>>> For list information or to unsubscribe, visit:
> >>>> http://www.unidata.ucar.edu/mailing_lists/
> >>>
> >>> _______________________________________________
> >>> NOTE: All exchanges posted to Unidata maintained email lists are
> >>> recorded in the Unidata inquiry tracking system and made publicly
> >>> available through the web. Users who post to any of the lists we
> >>> maintain are reminded to remove any personal information that they
> >>> do not want to be made public.
> >>>
> >>>
> >>> thredds mailing list
> >>> thredds@xxxxxxxxxxxxxxxx
> >>> For list information or to unsubscribe, visit:
> >>> http://www.unidata.ucar.edu/mailing_lists/
> >>> _______________________________________________
> >>> NOTE: All exchanges posted to Unidata maintained email lists are
> >>> recorded in the Unidata inquiry tracking system and made publicly
> >>> available through the web. Users who post to any of the lists we
> >>> maintain are reminded to remove any personal information that they
> >>> do not want to be made public.
> >>>
> >>>
> >>> thredds mailing list
> >>> thredds@xxxxxxxxxxxxxxxx
> >>> For list information or to unsubscribe, visit:
> >>> http://www.unidata.ucar.edu/mailing_lists/
> >>
> >> = = =
> >> Nathan Potter ndp at opendap.org
> >> OPeNDAP, Inc. +1.541.231.3317
> >>
>
> = = =
> Nathan Potter ndp at opendap.org
> OPeNDAP, Inc. +1.541.231.3317
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
= = =
Nathan Potter ndp at opendap.org
OPeNDAP, Inc. +1.541.231.3317
