On Fri, Jan 18, 2019 at 10:48 AM Nathan Potter <ndp@xxxxxxxxxxx
<mailto:ndp@xxxxxxxxxxx>> wrote:
Sean,
Interesting.
I thought that mod_jk had been abandoned/superseded for/by the
Apache JServ Protocol (AJP) protocol with the introduction of
Apache 2.x
N
> On Jan 18, 2019, at 9:21 AM, Sean Arms <sarms@xxxxxxxx
<mailto:sarms@xxxxxxxx>> wrote:
>
> Hi Nathan, all,
>
> netCDF-Java does proper encoding as of 4.6.12 and 5.0.0-beta6
(runtime configurable), and in the past it encoded as well - there
was a period of time in the 4.5 and 4.6 line of development in
which the encoding was skipped, but that's no longer the case.
>
> Interestingly enough, if you run Apache with mod_jk, non-encoded
urls will be accepted and passed along to tomcat in a usable form
without issue, but I have no idea how long that will last.
>
> Sean
>
> On Thu, Jan 17, 2019 at 4:25 PM Nathan Potter <ndp@xxxxxxxxxxx
<mailto:ndp@xxxxxxxxxxx>> wrote:
> Hi Thomas,
>
> I believe that I worked with Robert Schmunk (the author of
Panoply) on this very problem and I think the reason he is using
netcdfAll-5.0.0-SNAPSHOT is that it solves the outgoing encoding
issue on the client side. Is that right Robert? (We may not hear
back from him as I suspect he's furloughed and unavailable a.t.m.)
>
> If the client cannot be cajoled into producing the correct URL
encoding then I think the work falls back to relaxing the server's
relaxedQueryChars configuration.
>
> Which is certainly less than ideal.
>
> And to end on a cheery note - once the netcdf-java library is
release it can take some time (up to a year) before it gets rolled
into a Matlab release.
>
> Sorry I can't be more helpful.
>
>
> Sincerely,
>
> Nathan
>
>
> > On Jan 17, 2019, at 1:39 PM, Thomas Cook <tmcook@xxxxxxxx
<mailto:tmcook@xxxxxxxx>> wrote:
> >
> > OK Given Nathan and Christian's comments, I'm focussing my
> > troubleshooting on client side.
> > What I've learned
> > 1) Matlab's native netCDF toolbox depends on netCDF 4.3.3.1. I
> > couldn't find an easy way to update the netCDF version inside my
> > Matlab install.
> > 2) I updated the netcdf4-python module to use netCDF 4.6.12.
This did
> > not solve the issue.
> > 3) Panoply (which works) is using netcdfAll-5.0.0-SNAPSHOT. I
couldn't
> > find an older version to download, nor was I able to run
panoply with
> > an older netcdfAll.jar
> > 4) Another user has reported success using the SNTOOLS/MEXNC
package
> > to read the files. This also uses a version of netcdfAll.jar
for its
> > decoding. I can't verify this yet.
> >
> > So not sure there's much more I can do with on the client
side. I'll
> > mess with the Dockerfile like Christian suggested, and try
using some
> > non FMRC collections to test with the clients. If anyone has any
> > ideas, I'm open!
> > Thanks to all who have replied and I'll update if I make any
headway.
> > Tom
> >
> > On Thu, Jan 17, 2019 at 5:53 AM Nathan Potter <ndp@xxxxxxxxxxx
<mailto:ndp@xxxxxxxxxxx>> wrote:
> >>
> >>
> >>
> >>> On Jan 17, 2019, at 5:37 AM, Christian Skarby
<christians@xxxxxx <mailto:christians@xxxxxx>> wrote:
> >>>
> >>> For what it's worth; from what I've seen encoding errors
would result in a reject from tomcat before the request is passed
to the TDS application, and it would then probably not show up in
the threddsServlet.log.
> >>
> >> This is the behavior that we have seen.
> >>
> >> Tomcat's rejection is an empty document and response headers ala:
> >>
> >> HTTP/1.1 400 Bad Request
> >> Server: Apache-Coyote/1.1
> >> Transfer-Encoding: chunked
> >> Date: Thu, 17 Jan 2019 13:47:48 GMT
> >> Connection: close
> >>
> >>>
> >>> Perhaphs Tom, you could try to just change the application
code in your docker image (extract war or extracted thredds
servlet from one version and then inject it in the other). I guess
this should be possible, although I have not looked into the
official docker images for thredds. If you make this fly it would
be possible to say if it is the servlet or the other libraries in
the docker image that causes you problems.
> >>>
> >>> Regarding the possible encoding errors: Tomcat introduced
the more strict behavior in v9.0.8 (and also v8.5.31, v8.0.52 and
v7.0.87).
> >>>
> >>> See the corresponding tomcat changelogs:
> >>> • 62273: Implement configuration options to
work-around specification non-compliant user agents (including all
the major browsers) that do not correctly %nn encode URI paths and
query strings as required by RFC 7230 and RFC 3986. (markt)
> >>> See this thread in the google group for opendap.org
<http://opendap.org>:
> >>>
https://groups.google.com/a/opendap.org/forum/#!msg/support/ixTqhDXoLZQ/IT0lvZQ7CAAJ
> >>>
> >>> Tomcat-documentation:
> >>> https://tomcat.apache.org/tomcat-9.0-doc/config/http.html
> >>>
> >>> relaxedPathChars
> >>> The HTTP/1.1 specification requires that certain characters
are %nn encoded when used in URI paths. Unfortunately, many user
agents including all the major browsers are not compliant with
this specification and use these characters in unencoded form. To
prevent Tomcat rejecting such requests, this attribute may be used
to specify the additional characters to allow. If not specified,
no addtional characters will be allowed. The value may be any
combination of the following characters: " < > [ \ ] ^ ` { | } .
Any other characters present in the value will be ignored.
> >>>
> >>> relaxedQueryChars
> >>> The HTTP/1.1 specification requires that certain characters
are %nn encoded when used in URI query strings. Unfortunately,
many user agents including all the major browsers are not
compliant with this specification and use these characters in
unencoded form. To prevent Tomcat rejecting such requests, this
attribute may be used to specify the additional characters to
allow. If not specified, no addtional characters will be allowed.
The value may be any combination of the following characters: " <
> [ \ ] ^ ` { | } . Any other characters present in the value will
be ignored.
> >>>
> >>>
> >>> As the OPeNDAP spesification (see section 6.1.1.2 on page
22) requires the use of [ and ] in the query string, and several
clients doesn't quote these characters, we need to tweak the
relaxedQueryChars configuration for tomcat.
> >>
> >> For completeness I will add that the DAP protocol also
utilizes the < and > characters for formulating relational
constraints, however these are limited to constraining Sequence
data types. I don't know that the TDS supports this data type so
they may not be required in the relaxedQueryChars list.
> >>
> >>
> >>> Regards,
> >>> Christian Skarby
> >>>
> >>> Den tor. 17. jan. 2019 kl. 05:42 skrev Thomas Cook
<tmcook@xxxxxxxx <mailto:tmcook@xxxxxxxx>>:
> >>> I do not know how to debug the matlab or python clients, but
this is
> >>> what shows up in the threddsServlet.log
> >>>
> >>> Request: "GET
/thredds/dodsC/HFR/USWC/6km/hourly/RTV/HFRADAR_US_West_Coast_6km_Resolution_Hourly_RTV_best.ncd.dods?lat,lon,site%5flat,site%5flon,site%5fcode,site%5fnetCode,procParams
> >>> HTTP/1.0"
> >>>
> >>> Thanks
> >>> Tom
> >>>
> >>> On Wed, Jan 16, 2019 at 7:47 PM Dennis Heimbigner
<dmh@xxxxxxxx <mailto:dmh@xxxxxxxx>> wrote:
> >>>>
> >>>> Is there anyway you can see the exact URL being sent to the
server?
> >>>>
> >>>> On 1/16/2019 8:24 PM, Thomas Cook wrote:
> >>>>> Thanks Sean.
> >>>>> The server is http://hfrnet-tds.ucsd.edu/thredds (although
I just
> >>>>> downgraded back to 4.6.12-SNAPSHOT). I will see about
making my test
> >>>>> server accessible to you, but that will take a little time to
> >>>>> configure on my end.
> >>>>> Let me know if you think I should open an issue on the
github page and
> >>>>> move the discussion there.
> >>>>> Tom
> >>>>>
> >>>>> On Wed, Jan 16, 2019 at 6:32 PM Sean Arms <sarms@xxxxxxxx
<mailto:sarms@xxxxxxxx>> wrote:
> >>>>>> Hi Tom,
> >>>>>>
> >>>>>> So you are hitting up an opendap endpoint using all three
libraries / tools (python, matlab, and panoply)? Is this
particular server publicly accessible?
> >>>>>>
> >>>>>> I have a bad feeling that this is due to URL encoding (or
lack thereof) on the client side and changes made in tomcat 8.5,
which it looks like the thredds-docker repository bumped up to
with the 4.6.12 release.
> >>>>>>
> >>>>>> Cheers,
> >>>>>>
> >>>>>> Sean
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Jan 16, 2019 at 7:06 PM Thomas Cook
<tmcook@xxxxxxxx <mailto:tmcook@xxxxxxxx>> wrote:
> >>>>>>> HI I just wanted to add some details.
> >>>>>>>
> >>>>>>> The functionality works when I downgraded to the docker
container
> >>>>>>> 4.6.12-SNAPSHOT
> >>>>>>>
> >>>>>>> My data is part of a FMRC collection.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Tom
> >>>>>>>
> >>>>>>> On Wed, Jan 16, 2019 at 4:51 PM Thomas Cook
<tmcook@xxxxxxxx <mailto:tmcook@xxxxxxxx>> wrote:
> >>>>>>>> HI
> >>>>>>>> I just yesterday updated my thredds-docker to 4.6.12 from
> >>>>>>>> 4.6.12-snapshot. It seems to be running fine, except
I'm getting
> >>>>>>>> errors using matlab ncread and python netcdf4.dataset.
When I try to
> >>>>>>>> access any of the multi-dimensional (time, lat, lon)
variables I get
> >>>>>>>> the following errors
> >>>>>>>> with matlab:
> >>>>>>>> Error using netcdflib
> >>>>>>>> The NetCDF library encountered an error during execution of
> >>>>>>>> 'getVarsShort' function - 'Access failure (-77)'.
> >>>>>>>>
> >>>>>>>> Error in netcdf.getVar (line 136)
> >>>>>>>> data = netcdflib(funcstr,ncid,varid,varargin{:});
> >>>>>>>>
> >>>>>>>> Error in internal.matlab.imagesci.nc/read
<http://internal.matlab.imagesci.nc/read> (line 635)
> >>>>>>>> data = netcdf.getVar(gid, varid, ...
> >>>>>>>>
> >>>>>>>> Error in ncread (line 58)
> >>>>>>>> vardata = ncObj.read(varName, varargin{:});
> >>>>>>>>
> >>>>>>>> Error in readUVfromTDS (line 56)
> >>>>>>>> u=ncread(url, 'u_mean', [1 1 1], [2 2 2]);
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> and with python:
> >>>>>>>> netCDF4/_netCDF4.pyx in
netCDF4._netCDF4.Variable.__getitem__()
> >>>>>>>>
> >>>>>>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable._get()
> >>>>>>>>
> >>>>>>>> netCDF4/_netCDF4.pyx in
netCDF4._netCDF4._ensure_nc_success()
> >>>>>>>>
> >>>>>>>> RuntimeError: NetCDF: Access failure
> >>>>>>>>
> >>>>>>>> I know that the latest docker contains netcdf 4.6,
where as the
> >>>>>>>> snapshot had 4.4. Could it be a version problem with
the netCD4 python
> >>>>>>>> and matlab libraries? I should be running the latest
netcdf python
> >>>>>>>> version and not too far behind with matlab versions.
> >>>>>>>>
> >>>>>>>> The weird things are that the index variables are
accessible (time,
> >>>>>>>> lat, lon etc), and the multiple-dimension data is
accessible just
> >>>>>>>> fine using the OpenDap web interface and Panoply. So
I'm a little
> >>>>>>>> confused. I'm addressing this list since I think there
might be some
> >>>>>>>> matlab and python expertise here and I think there were
some messages
> >>>>>>>> about the Access failure error in the past.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Tom
> >>>>>>> _______________________________________________
> >>>>>>> NOTE: All exchanges posted to Unidata maintained email
lists are
> >>>>>>> recorded in the Unidata inquiry tracking system and made
publicly
> >>>>>>> available through the web. Users who post to any of the
lists we
> >>>>>>> maintain are reminded to remove any personal information
that they
> >>>>>>> do not want to be made public.
> >>>>>>>
> >>>>>>>
> >>>>>>> thredds mailing list
> >>>>>>> thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
> >>>>>>> For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
> >>>>> _______________________________________________
> >>>>> NOTE: All exchanges posted to Unidata maintained email
lists are
> >>>>> recorded in the Unidata inquiry tracking system and made
publicly
> >>>>> available through the web. Users who post to any of the
lists we
> >>>>> maintain are reminded to remove any personal information
that they
> >>>>> do not want to be made public.
> >>>>>
> >>>>>
> >>>>> thredds mailing list
> >>>>> thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
> >>>>> For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
> >>>>
> >>>> _______________________________________________
> >>>> NOTE: All exchanges posted to Unidata maintained email
lists are
> >>>> recorded in the Unidata inquiry tracking system and made
publicly
> >>>> available through the web. Users who post to any of the
lists we
> >>>> maintain are reminded to remove any personal information
that they
> >>>> do not want to be made public.
> >>>>
> >>>>
> >>>> thredds mailing list
> >>>> thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
> >>>> For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
> >>>
> >>> _______________________________________________
> >>> NOTE: All exchanges posted to Unidata maintained email lists are
> >>> recorded in the Unidata inquiry tracking system and made
publicly
> >>> available through the web. Users who post to any of the
lists we
> >>> maintain are reminded to remove any personal information
that they
> >>> do not want to be made public.
> >>>
> >>>
> >>> thredds mailing list
> >>> thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
> >>> For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
> >>> _______________________________________________
> >>> NOTE: All exchanges posted to Unidata maintained email lists are
> >>> recorded in the Unidata inquiry tracking system and made
publicly
> >>> available through the web. Users who post to any of the
lists we
> >>> maintain are reminded to remove any personal information
that they
> >>> do not want to be made public.
> >>>
> >>>
> >>> thredds mailing list
> >>> thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
> >>> For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
> >>
> >> = = =
> >> Nathan Potter ndp at opendap.org
<http://opendap.org>
> >> OPeNDAP, Inc. +1.541.231.3317
> >>
>
> = = =
> Nathan Potter ndp at opendap.org
<http://opendap.org>
> OPeNDAP, Inc. +1.541.231.3317
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
> For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
= = =
Nathan Potter ndp at opendap.org
<http://opendap.org>
OPeNDAP, Inc. +1.541.231.3317
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
