Hi Nathan, all,
netCDF-Java does proper encoding as of 4.6.12 and 5.0.0-beta6 (runtime
configurable), and in the past it encoded as well - there was a period of
time in the 4.5 and 4.6 line of development in which the encoding was
skipped, but that's no longer the case.
Interestingly enough, if you run Apache with mod_jk, non-encoded urls will
be accepted and passed along to tomcat in a usable form without issue, but
I have no idea how long that will last.
Sean
On Thu, Jan 17, 2019 at 4:25 PM Nathan Potter <ndp@xxxxxxxxxxx> wrote:
> Hi Thomas,
>
> I believe that I worked with Robert Schmunk (the author of Panoply) on
> this very problem and I think the reason he is using
> netcdfAll-5.0.0-SNAPSHOT is that it solves the outgoing encoding issue on
> the client side. Is that right Robert? (We may not hear back from him as I
> suspect he's furloughed and unavailable a.t.m.)
>
> If the client cannot be cajoled into producing the correct URL encoding
> then I think the work falls back to relaxing the server's relaxedQueryChars
> configuration.
>
> Which is certainly less than ideal.
>
> And to end on a cheery note - once the netcdf-java library is release it
> can take some time (up to a year) before it gets rolled into a Matlab
> release.
>
> Sorry I can't be more helpful.
>
>
> Sincerely,
>
> Nathan
>
>
> > On Jan 17, 2019, at 1:39 PM, Thomas Cook <tmcook@xxxxxxxx> wrote:
> >
> > OK Given Nathan and Christian's comments, I'm focussing my
> > troubleshooting on client side.
> > What I've learned
> > 1) Matlab's native netCDF toolbox depends on netCDF 4.3.3.1. I
> > couldn't find an easy way to update the netCDF version inside my
> > Matlab install.
> > 2) I updated the netcdf4-python module to use netCDF 4.6.12. This did
> > not solve the issue.
> > 3) Panoply (which works) is using netcdfAll-5.0.0-SNAPSHOT. I couldn't
> > find an older version to download, nor was I able to run panoply with
> > an older netcdfAll.jar
> > 4) Another user has reported success using the SNTOOLS/MEXNC package
> > to read the files. This also uses a version of netcdfAll.jar for its
> > decoding. I can't verify this yet.
> >
> > So not sure there's much more I can do with on the client side. I'll
> > mess with the Dockerfile like Christian suggested, and try using some
> > non FMRC collections to test with the clients. If anyone has any
> > ideas, I'm open!
> > Thanks to all who have replied and I'll update if I make any headway.
> > Tom
> >
> > On Thu, Jan 17, 2019 at 5:53 AM Nathan Potter <ndp@xxxxxxxxxxx> wrote:
> >>
> >>
> >>
> >>> On Jan 17, 2019, at 5:37 AM, Christian Skarby <christians@xxxxxx>
> wrote:
> >>>
> >>> For what it's worth; from what I've seen encoding errors would result
> in a reject from tomcat before the request is passed to the TDS
> application, and it would then probably not show up in the
> threddsServlet.log.
> >>
> >> This is the behavior that we have seen.
> >>
> >> Tomcat's rejection is an empty document and response headers ala:
> >>
> >> HTTP/1.1 400 Bad Request
> >> Server: Apache-Coyote/1.1
> >> Transfer-Encoding: chunked
> >> Date: Thu, 17 Jan 2019 13:47:48 GMT
> >> Connection: close
> >>
> >>>
> >>> Perhaphs Tom, you could try to just change the application code in
> your docker image (extract war or extracted thredds servlet from one
> version and then inject it in the other). I guess this should be possible,
> although I have not looked into the official docker images for thredds. If
> you make this fly it would be possible to say if it is the servlet or the
> other libraries in the docker image that causes you problems.
> >>>
> >>> Regarding the possible encoding errors: Tomcat introduced the more
> strict behavior in v9.0.8 (and also v8.5.31, v8.0.52 and v7.0.87).
> >>>
> >>> See the corresponding tomcat changelogs:
> >>> • 62273: Implement configuration options to work-around
> specification non-compliant user agents (including all the major browsers)
> that do not correctly %nn encode URI paths and query strings as required by
> RFC 7230 and RFC 3986. (markt)
> >>> See this thread in the google group for opendap.org:
> >>>
> https://groups.google.com/a/opendap.org/forum/#!msg/support/ixTqhDXoLZQ/IT0lvZQ7CAAJ
> >>>
> >>> Tomcat-documentation:
> >>> https://tomcat.apache.org/tomcat-9.0-doc/config/http.html
> >>>
> >>> relaxedPathChars
> >>> The HTTP/1.1 specification requires that certain characters are %nn
> encoded when used in URI paths. Unfortunately, many user agents including
> all the major browsers are not compliant with this specification and use
> these characters in unencoded form. To prevent Tomcat rejecting such
> requests, this attribute may be used to specify the additional characters
> to allow. If not specified, no addtional characters will be allowed. The
> value may be any combination of the following characters: " < > [ \ ] ^ ` {
> | } . Any other characters present in the value will be ignored.
> >>>
> >>> relaxedQueryChars
> >>> The HTTP/1.1 specification requires that certain characters are %nn
> encoded when used in URI query strings. Unfortunately, many user agents
> including all the major browsers are not compliant with this specification
> and use these characters in unencoded form. To prevent Tomcat rejecting
> such requests, this attribute may be used to specify the additional
> characters to allow. If not specified, no addtional characters will be
> allowed. The value may be any combination of the following characters: " <
> > [ \ ] ^ ` { | } . Any other characters present in the value will be
> ignored.
> >>>
> >>>
> >>> As the OPeNDAP spesification (see section 6.1.1.2 on page 22) requires
> the use of [ and ] in the query string, and several clients doesn't quote
> these characters, we need to tweak the relaxedQueryChars configuration for
> tomcat.
> >>
> >> For completeness I will add that the DAP protocol also utilizes the <
> and > characters for formulating relational constraints, however these are
> limited to constraining Sequence data types. I don't know that the TDS
> supports this data type so they may not be required in the
> relaxedQueryChars list.
> >>
> >>
> >>> Regards,
> >>> Christian Skarby
> >>>
> >>> Den tor. 17. jan. 2019 kl. 05:42 skrev Thomas Cook <tmcook@xxxxxxxx>:
> >>> I do not know how to debug the matlab or python clients, but this is
> >>> what shows up in the threddsServlet.log
> >>>
> >>> Request: "GET
> /thredds/dodsC/HFR/USWC/6km/hourly/RTV/HFRADAR_US_West_Coast_6km_Resolution_Hourly_RTV_best.ncd.dods?lat,lon,site%5flat,site%5flon,site%5fcode,site%5fnetCode,procParams
> >>> HTTP/1.0"
> >>>
> >>> Thanks
> >>> Tom
> >>>
> >>> On Wed, Jan 16, 2019 at 7:47 PM Dennis Heimbigner <dmh@xxxxxxxx>
> wrote:
> >>>>
> >>>> Is there anyway you can see the exact URL being sent to the server?
> >>>>
> >>>> On 1/16/2019 8:24 PM, Thomas Cook wrote:
> >>>>> Thanks Sean.
> >>>>> The server is http://hfrnet-tds.ucsd.edu/thredds (although I just
> >>>>> downgraded back to 4.6.12-SNAPSHOT). I will see about making my test
> >>>>> server accessible to you, but that will take a little time to
> >>>>> configure on my end.
> >>>>> Let me know if you think I should open an issue on the github page
> and
> >>>>> move the discussion there.
> >>>>> Tom
> >>>>>
> >>>>> On Wed, Jan 16, 2019 at 6:32 PM Sean Arms <sarms@xxxxxxxx> wrote:
> >>>>>> Hi Tom,
> >>>>>>
> >>>>>> So you are hitting up an opendap endpoint using all three libraries
> / tools (python, matlab, and panoply)? Is this particular server publicly
> accessible?
> >>>>>>
> >>>>>> I have a bad feeling that this is due to URL encoding (or lack
> thereof) on the client side and changes made in tomcat 8.5, which it looks
> like the thredds-docker repository bumped up to with the 4.6.12 release.
> >>>>>>
> >>>>>> Cheers,
> >>>>>>
> >>>>>> Sean
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Jan 16, 2019 at 7:06 PM Thomas Cook <tmcook@xxxxxxxx>
> wrote:
> >>>>>>> HI I just wanted to add some details.
> >>>>>>>
> >>>>>>> The functionality works when I downgraded to the docker container
> >>>>>>> 4.6.12-SNAPSHOT
> >>>>>>>
> >>>>>>> My data is part of a FMRC collection.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Tom
> >>>>>>>
> >>>>>>> On Wed, Jan 16, 2019 at 4:51 PM Thomas Cook <tmcook@xxxxxxxx>
> wrote:
> >>>>>>>> HI
> >>>>>>>> I just yesterday updated my thredds-docker to 4.6.12 from
> >>>>>>>> 4.6.12-snapshot. It seems to be running fine, except I'm getting
> >>>>>>>> errors using matlab ncread and python netcdf4.dataset. When I try
> to
> >>>>>>>> access any of the multi-dimensional (time, lat, lon) variables I
> get
> >>>>>>>> the following errors
> >>>>>>>> with matlab:
> >>>>>>>> Error using netcdflib
> >>>>>>>> The NetCDF library encountered an error during execution of
> >>>>>>>> 'getVarsShort' function - 'Access failure (-77)'.
> >>>>>>>>
> >>>>>>>> Error in netcdf.getVar (line 136)
> >>>>>>>> data = netcdflib(funcstr,ncid,varid,varargin{:});
> >>>>>>>>
> >>>>>>>> Error in internal.matlab.imagesci.nc/read (line 635)
> >>>>>>>> data = netcdf.getVar(gid, varid, ...
> >>>>>>>>
> >>>>>>>> Error in ncread (line 58)
> >>>>>>>> vardata = ncObj.read(varName, varargin{:});
> >>>>>>>>
> >>>>>>>> Error in readUVfromTDS (line 56)
> >>>>>>>> u=ncread(url, 'u_mean', [1 1 1], [2 2 2]);
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> and with python:
> >>>>>>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.__getitem__()
> >>>>>>>>
> >>>>>>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable._get()
> >>>>>>>>
> >>>>>>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success()
> >>>>>>>>
> >>>>>>>> RuntimeError: NetCDF: Access failure
> >>>>>>>>
> >>>>>>>> I know that the latest docker contains netcdf 4.6, where as the
> >>>>>>>> snapshot had 4.4. Could it be a version problem with the netCD4
> python
> >>>>>>>> and matlab libraries? I should be running the latest netcdf python
> >>>>>>>> version and not too far behind with matlab versions.
> >>>>>>>>
> >>>>>>>> The weird things are that the index variables are accessible
> (time,
> >>>>>>>> lat, lon etc), and the multiple-dimension data is accessible just
> >>>>>>>> fine using the OpenDap web interface and Panoply. So I'm a little
> >>>>>>>> confused. I'm addressing this list since I think there might be
> some
> >>>>>>>> matlab and python expertise here and I think there were some
> messages
> >>>>>>>> about the Access failure error in the past.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Tom
> >>>>>>> _______________________________________________
> >>>>>>> NOTE: All exchanges posted to Unidata maintained email lists are
> >>>>>>> recorded in the Unidata inquiry tracking system and made publicly
> >>>>>>> available through the web. Users who post to any of the lists we
> >>>>>>> maintain are reminded to remove any personal information that they
> >>>>>>> do not want to be made public.
> >>>>>>>
> >>>>>>>
> >>>>>>> thredds mailing list
> >>>>>>> thredds@xxxxxxxxxxxxxxxx
> >>>>>>> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
> >>>>> _______________________________________________
> >>>>> NOTE: All exchanges posted to Unidata maintained email lists are
> >>>>> recorded in the Unidata inquiry tracking system and made publicly
> >>>>> available through the web. Users who post to any of the lists we
> >>>>> maintain are reminded to remove any personal information that they
> >>>>> do not want to be made public.
> >>>>>
> >>>>>
> >>>>> thredds mailing list
> >>>>> thredds@xxxxxxxxxxxxxxxx
> >>>>> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
> >>>>
> >>>> _______________________________________________
> >>>> NOTE: All exchanges posted to Unidata maintained email lists are
> >>>> recorded in the Unidata inquiry tracking system and made publicly
> >>>> available through the web. Users who post to any of the lists we
> >>>> maintain are reminded to remove any personal information that they
> >>>> do not want to be made public.
> >>>>
> >>>>
> >>>> thredds mailing list
> >>>> thredds@xxxxxxxxxxxxxxxx
> >>>> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
> >>>
> >>> _______________________________________________
> >>> NOTE: All exchanges posted to Unidata maintained email lists are
> >>> recorded in the Unidata inquiry tracking system and made publicly
> >>> available through the web. Users who post to any of the lists we
> >>> maintain are reminded to remove any personal information that they
> >>> do not want to be made public.
> >>>
> >>>
> >>> thredds mailing list
> >>> thredds@xxxxxxxxxxxxxxxx
> >>> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
> >>> _______________________________________________
> >>> NOTE: All exchanges posted to Unidata maintained email lists are
> >>> recorded in the Unidata inquiry tracking system and made publicly
> >>> available through the web. Users who post to any of the lists we
> >>> maintain are reminded to remove any personal information that they
> >>> do not want to be made public.
> >>>
> >>>
> >>> thredds mailing list
> >>> thredds@xxxxxxxxxxxxxxxx
> >>> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
> >>
> >> = = =
> >> Nathan Potter ndp at opendap.org
> >> OPeNDAP, Inc. +1.541.231.3317
> >>
>
> = = =
> Nathan Potter ndp at opendap.org
> OPeNDAP, Inc. +1.541.231.3317
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
