On Wed, Oct 19, 2016 at 5:34 AM, Nathan Potter <ndp@xxxxxxxxxxx> wrote:
>
> I would appreciate knowing what part of the behavior/output of the
> ingria.coas.oregonstate.edu Hyrax server has led to the comments that it
> is not configured correctly.
>
that is totally speculation -- all we know for sure is that trying to
access teh contents of a variable with the netcdf library fails. Accessing
the same data with PyDAP or a simple openDAP url seems to work -- see my
original message and one from James Gallagar
I think tha the netCDF4 lib don't pass thruogh all teh error messages, so
we get a file not found error, which is probably not what's really going on .
Clearly, netCDF4 is asking for the "same" thing in a different way, so I've
tried to look at the TCP traffic, but haven't been able to make sense of it
-- I'm hoping someone with more expertise than me could debug this.
And I have no idea if the way that netCDF4 is asking for teh data is
"correct", so it could be:
A bug in the netCDF4 lib
A bug in the HYRAX server
A mis-configuration of the HYRAX server
I still don't know how to figure out which.
> By asking these questions I am trying to understand, and not in anyway
> dispute, the assertion that the server is “misconfigured”. I just want to
> tease apart the possibility that we are seeing a fundamental bug problem
> from a configuration issue.
>
> I do know that at NASA’s request and expense a significant number of
> changes and improvements have been made to Hyrax’s NcML features, but this
> work happened long after the release of version 1.9.0.
>
That certainly points to a Hyrax issue.
In light of this I think the first step should be to instigate upgrading
> the server, as version 1.9.0 is a long way back and not only does the
> latest release (1.13.1) offer an improved NcML implementation it also
> contains numerous security updates.
>
That makes lots of sense, yes. So far, we've all been doing this "blind" --
I"ll try to rach out ot the server operator, and suggest they upgrade.
> Regardless, any clarification that could be offered w.r.t. the
> mis-configuration assertion would be very helpful to me.
>
I hope that above clarified it -- we're all grasping at straws here!
-Thanks for your interest and help.
-CHB
>
> Sincerely,
>
> Nathan
>
>
>
>
> > On Oct 18, 2016, at 11:30 PM, Emilio Mayorga <emiliomayorga@xxxxxxxxx>
> wrote:
> >
> > I'm afraid I don't have much helpful insight to bring into this. The
> model behind that Hyrax server is indeed a NANOOS model, but the hyrax
> server itself was not set up and is not maintained by one of us in the
> NANOOS DMAC team. In fact, if I remember correctly the server is somewhat
> duplicative of a THREDDS server we do maintain for the same model output;
> it was setup about two years ago, I think to support an application at NOAA
> ORR that took advantage of some specific hyrax capability missing in
> THREDDS' OPeNDAP (Chris Barker would know better than me), partly through
> help and persuasion from Rich Signell; again, sketchy memory here. I was
> never thrilled with the idea of having a hyrax server in the mix when none
> of us had the capacity to invest in learning how to properly maintain it,
> and already had a THREDDS server for the same model. I'm not surprised to
> hear it's an old version, too.
> >
> > Sorry ...
> >
> > -Emilio
> >
> >
> > On Mon, Oct 17, 2016 at 5:36 PM, Chris Barker <chris.barker@xxxxxxxx>
> wrote:
> > Folks,
> >
> > I manged to do a tcpdump, and can (sortof) see what's going on -- though
> I can't really make sense of it -- maybe one of you can?
> >
> > This is running the enclosed script, to it first does the successful
> pydap request, then the unsuccessful netCDF4 request.
> >
> > (though it looking like this is a poorly configured server...)
> >
> > -CHB
> >
> >
> > On Mon, Oct 17, 2016 at 4:49 PM, Nathan Potter <ndp@xxxxxxxxxxx> wrote:
> > By which I meant - just deference the URL:
> >
> > http://ingria.coas.oregonstate.edu/opendap/aggregation/ocean_time_
> aggregation.ncml
> >
> > To get the NcML file:
> >
> > curl http://ingria.coas.oregonstate.edu/opendap/
> aggregation/ocean_time_aggregation.ncml
> >
> >
> >
> > Nathan
> >
> >
> >
> > > On Oct 17, 2016, at 4:37 PM, Nathan Potter <ndp@xxxxxxxxxxx> wrote:
> > >
> > > Hi,
> > >
> > > Seems like no one has tried to actually ask for the NcML file:
> > >
> > > http://ingria.coas.oregonstate.edu/opendap/aggregation/ocean_time_
> aggregation.ncml
> > >
> > >
> > > N
> > >
> > >
> > >> On Oct 17, 2016, at 4:35 PM, Roy Mendelssohn - NOAA Federal <
> roy.mendelssohn@xxxxxxxx> wrote:
> > >>
> > >> Hi Emilio:
> > >>
> > >> I believe this is a NaNOOS server. If you can read over the thread,,
> any insight would be appreciated, What is in the .ncml file could be of
> interest.
> > >>
> > >> Thanks,
> > >>
> > >> -roy
> > >>
> > >>> On Oct 17, 2016, at 4:20 PM, Chris Barker <Chris.Barker@xxxxxxxx>
> wrote:
> > >>>
> > >>> On Mon, Oct 17, 2016 at 3:57 PM, Roy Mendelssohn - NOAA Federal <
> roy.mendelssohn@xxxxxxxx> wrote:
> > >>> I find that if you go to the html page, many of the variables at
> the top can't be accessed requesting ascii, so there are clearly problems
> with how this is set up
> > >>>
> > >>> that does point to server issues.
> > >>>
> > >>> I am wondering if these are defined in the .ncml. It would be
> interesting to see what the .ncml file looks like.
> > >>>
> > >>> One thing to consider also is pydap does not call the C library
> IIRC, it is a pure python implementation
> > >>>
> > >>> Exactly -- so it takes the netcdf C lib out of the picture -- which
> means there is a protocol that pydap and hyrax are speaking that lets the
> data be accessed...
> > >>>
> > >>> If anyone knows how to see what actual requests are being made by
> the two, we might be able to figure out where the failure is...
> > >>>
> > >>> -CHB
> > >>>
> > >>>
> > >>> -Roy
> > >>>
> > >>>> On Oct 17, 2016, at 3:33 PM, dmh@xxxxxxxx wrote:
> > >>>>
> > >>>> The difference with pydap is probably that you adding a constraint
> > >>>> that is causing the server to ignore whatever variable
> > >>>> is causing the problem. If you ask pydap for the whole
> > >>>> dataset, does it fail?
> > >>>> (e.g. something like:
> > >>>> http://ingria.coas.oregonstate.edu/opendap/aggregation/ocean_time_
> aggregation.ncml.ascii
> > >>>> =Dennis Heimbigner
> > >>>> Unidata
> > >>>>
> > >>>> On 10/17/2016 4:16 PM, Chris Barker wrote:
> > >>>>> On Mon, Oct 17, 2016 at 2:36 PM, dmh@xxxxxxxx <mailto:dmh@xxxxxxxx
> >
> > >>>>> <dmh@xxxxxxxx <mailto:dmh@xxxxxxxx>> wrote:
> > >>>>>
> > >>>>> The log and show=fetch appear to not work under python
> > >>>>> for some reason; stdout/stderr redirect?
> > >>>>>
> > >>>>>
> > >>>>> I don't think so -- and I do get a lot of "Log:prefetch" messages,
> so
> > >>>>> stdout.stderr seems to be working.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> In any case I can duplicate the error using wget e.g.
> > >>>>> wget
> > >>>>> 'http://ingria.coas.oregonstate.edu:80/opendap/
> aggregation/ocean_time_aggregation.ncml.dods
> > >>>>> <http://ingria.coas.oregonstate.edu:80/opendap/
> aggregation/ocean_time_aggregation.ncml.dods>'
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> hmm, I don't seem to have wget, but with curl, I get the below
> results,
> > >>>>> ending in:
> > >>>>>
> > >>>>> Data:
> > >>>>>
> > >>>>> Error {
> > >>>>>
> > >>>>> code = -111;
> > >>>>>
> > >>>>> message = "libdap error transmitting DataDDS: Could not read the
> > >>>>> variable `ntimes'.";
> > >>>>>
> > >>>>> -- if that's the fatal error, then yup -- not the netcdf lib --
> and yet,
> > >>>>> we can get the data with PyDAP, and if I put this:
> > >>>>>
> > >>>>>
> > >>>>> http://ingria.coas.oregonstate.edu/opendap/aggregation/ocean_time_
> aggregation.ncml.ascii?u[0][0][0:10][0:10]
> > >>>>>
> > >>>>> into my browser, I do get data.
> > >>>>>
> > >>>>> I have asked this on support@xxxxxxxxxxx <mailto:
> support@xxxxxxxxxxx> --
> > >>>>> and they have pointed out that that is a pretty old version of
> Hyrax --
> > >>>>> _maybe_ that's the problem.
> > >>>>>
> > >>>>> -CHB
> > >>>>>
> > >>>>>
> > >>>>> $ curl
> > >>>>> 'http://ingria.coas.oregonstate.edu:80/opendap/
> aggregation/ocean_time_aggregation.ncml.dods'
> > >>>>>
> > >>>>> Dataset {
> > >>>>>
> > >>>>> Int32 ntimes;
> > >>>>>
> > >>>>> Int32 ndtfast;
> > >>>>>
> > >>>>> Float64 dt;
> > >>>>>
> > >>>>> Float64 dtfast;
> > >>>>>
> > >>>>> Float64 dstart;
> > >>>>>
> > >>>>> Int32 nHIS;
> > >>>>>
> > >>>>> Int32 ndefHIS;
> > >>>>>
> > >>>>> Int32 nRST;
> > >>>>>
> > >>>>> Int32 ntsAVG;
> > >>>>>
> > >>>>> Int32 nAVG;
> > >>>>>
> > >>>>> Int32 ndefAVG;
> > >>>>>
> > >>>>> Float64 Falpha;
> > >>>>>
> > >>>>> Float64 Fbeta;
> > >>>>>
> > >>>>> Float64 Fgamma;
> > >>>>>
> > >>>>> Float64 nl_tnu2[tracer = 2];
> > >>>>>
> > >>>>> Float64 nl_visc2;
> > >>>>>
> > >>>>> Float64 Akt_bak[tracer = 2];
> > >>>>>
> > >>>>> Float64 Akv_bak;
> > >>>>>
> > >>>>> Float64 Akk_bak;
> > >>>>>
> > >>>>> Float64 Akp_bak;
> > >>>>>
> > >>>>> Float64 rdrg;
> > >>>>>
> > >>>>> Float64 rdrg2;
> > >>>>>
> > >>>>> Float64 Zob;
> > >>>>>
> > >>>>> Float64 Zos;
> > >>>>>
> > >>>>> Float64 Znudg;
> > >>>>>
> > >>>>> Float64 M2nudg;
> > >>>>>
> > >>>>> Float64 M3nudg;
> > >>>>>
> > >>>>> Float64 Tnudg[tracer = 2];
> > >>>>>
> > >>>>> Float64 FSobc_in[boundary = 4];
> > >>>>>
> > >>>>> Float64 FSobc_out[boundary = 4];
> > >>>>>
> > >>>>> Float64 M2obc_in[boundary = 4];
> > >>>>>
> > >>>>> Float64 M2obc_out[boundary = 4];
> > >>>>>
> > >>>>> Float64 Tobc_in[boundary = 4][tracer = 2];
> > >>>>>
> > >>>>> Float64 Tobc_out[boundary = 4][tracer = 2];
> > >>>>>
> > >>>>> Float64 M3obc_in[boundary = 4];
> > >>>>>
> > >>>>> Float64 M3obc_out[boundary = 4];
> > >>>>>
> > >>>>> Float64 rho0;
> > >>>>>
> > >>>>> Float64 gamma2;
> > >>>>>
> > >>>>> Int32 LtracerSrc[tracer = 2];
> > >>>>>
> > >>>>> Int32 spherical;
> > >>>>>
> > >>>>> Float64 xl;
> > >>>>>
> > >>>>> Float64 el;
> > >>>>>
> > >>>>> Int32 Vtransform;
> > >>>>>
> > >>>>> Int32 Vstretching;
> > >>>>>
> > >>>>> Float64 theta_s;
> > >>>>>
> > >>>>> Float64 theta_b;
> > >>>>>
> > >>>>> Float64 Tcline;
> > >>>>>
> > >>>>> Float64 hc;
> > >>>>>
> > >>>>> Float64 s_rho[s_rho = 40];
> > >>>>>
> > >>>>> Float64 s_w[s_w = 41];
> > >>>>>
> > >>>>> Grid {
> > >>>>>
> > >>>>> Array:
> > >>>>>
> > >>>>> Float64 Cs_r[s_rho = 40];
> > >>>>>
> > >>>>> Maps:
> > >>>>>
> > >>>>> Float64 s_rho[s_rho = 40];
> > >>>>>
> > >>>>> } Cs_r;
> > >>>>>
> > >>>>> Grid {
> > >>>>>
> > >>>>> Array:
> > >>>>>
> > >>>>> Float64 Cs_w[s_w = 41];
> > >>>>>
> > >>>>> Maps:
> > >>>>>
> > >>>>> Float64 s_w[s_w = 41];
> > >>>>>
> > >>>>> } Cs_w;
> > >>>>>
> > >>>>> Float64 user[Nuser = 25];
> > >>>>>
> > >>>>> Float64 h[eta_rho = 522][xi_rho = 310];
> > >>>>>
> > >>>>> Float64 f[eta_rho = 522][xi_rho = 310];
> > >>>>>
> > >>>>> Float64 pm[eta_rho = 522][xi_rho = 310];
> > >>>>>
> > >>>>> Float64 pn[eta_rho = 522][xi_rho = 310];
> > >>>>>
> > >>>>> Float64 lon_rho[eta_rho = 522][xi_rho = 310];
> > >>>>>
> > >>>>> Float64 lat_rho[eta_rho = 522][xi_rho = 310];
> > >>>>>
> > >>>>> Float64 lon_u[eta_u = 522][xi_u = 309];
> > >>>>>
> > >>>>> Float64 lat_u[eta_u = 522][xi_u = 309];
> > >>>>>
> > >>>>> Float64 lon_v[eta_v = 521][xi_v = 310];
> > >>>>>
> > >>>>> Float64 lat_v[eta_v = 521][xi_v = 310];
> > >>>>>
> > >>>>> Float64 lon_psi[eta_psi = 521][xi_psi = 309];
> > >>>>>
> > >>>>> Float64 lat_psi[eta_psi = 521][xi_psi = 309];
> > >>>>>
> > >>>>> Float64 angle[eta_rho = 522][xi_rho = 310];
> > >>>>>
> > >>>>> Float64 mask_rho[eta_rho = 522][xi_rho = 310];
> > >>>>>
> > >>>>> Float64 mask_u[eta_u = 522][xi_u = 309];
> > >>>>>
> > >>>>> Float64 mask_v[eta_v = 521][xi_v = 310];
> > >>>>>
> > >>>>> Float64 mask_psi[eta_psi = 521][xi_psi = 309];
> > >>>>>
> > >>>>> Float64 ocean_time[ocean_time = 1026];
> > >>>>>
> > >>>>> Float32 zeta[ocean_time = 1026][eta_rho = 522][xi_rho = 310];
> > >>>>>
> > >>>>> Float32 ubar[ocean_time = 1026][eta_u = 522][xi_u = 309];
> > >>>>>
> > >>>>> Float32 vbar[ocean_time = 1026][eta_v = 521][xi_v = 310];
> > >>>>>
> > >>>>> Float32 u[ocean_time = 1026][s_rho = 40][eta_u = 522][xi_u = 309];
> > >>>>>
> > >>>>> Float32 v[ocean_time = 1026][s_rho = 40][eta_v = 521][xi_v = 310];
> > >>>>>
> > >>>>> Float32 w[ocean_time = 1026][s_w = 41][eta_rho = 522][xi_rho =
> 310];
> > >>>>>
> > >>>>> Float32 temp[ocean_time = 1026][s_rho = 40][eta_rho = 522][xi_rho
> =
> > >>>>> 310];
> > >>>>>
> > >>>>> Float32 salt[ocean_time = 1026][s_rho = 40][eta_rho = 522][xi_rho
> =
> > >>>>> 310];
> > >>>>>
> > >>>>> Float32 AKv[ocean_time = 1026][s_w = 41][eta_rho = 522][xi_rho =
> 310];
> > >>>>>
> > >>>>> Float32 AKt[ocean_time = 1026][s_w = 41][eta_rho = 522][xi_rho =
> 310];
> > >>>>>
> > >>>>> Float32 AKs[ocean_time = 1026][s_w = 41][eta_rho = 522][xi_rho =
> 310];
> > >>>>>
> > >>>>> Float32 tke[ocean_time = 1026][s_w = 41][eta_rho = 522][xi_rho =
> 310];
> > >>>>>
> > >>>>> Float32 shflux[ocean_time = 1026][eta_rho = 522][xi_rho = 310];
> > >>>>>
> > >>>>> Float32 latent[ocean_time = 1026][eta_rho = 522][xi_rho = 310];
> > >>>>>
> > >>>>> Float32 sensible[ocean_time = 1026][eta_rho = 522][xi_rho = 310];
> > >>>>>
> > >>>>> Float32 lwrad[ocean_time = 1026][eta_rho = 522][xi_rho = 310];
> > >>>>>
> > >>>>> Float32 swrad[ocean_time = 1026][eta_rho = 522][xi_rho = 310];
> > >>>>>
> > >>>>> } ocean_time_aggregation.ncml;
> > >>>>>
> > >>>>> Data:
> > >>>>>
> > >>>>> Error {
> > >>>>>
> > >>>>> code = -111;
> > >>>>>
> > >>>>> message = "libdap error transmitting DataDDS: Could not read the
> > >>>>> variable `ntimes'.";
> > >>>>>
> > >>>>> };
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>>
> > >>>>> Christopher Barker, Ph.D.
> > >>>>> Oceanographer
> > >>>>>
> > >>>>> Emergency Response Division
> > >>>>> NOAA/NOS/OR&R (206) 526-6959 voice
> > >>>>> 7600 Sand Point Way NE (206) 526-6329 fax
> > >>>>> Seattle, WA 98115 (206) 526-6317 main reception
> > >>>>>
> > >>>>> Chris.Barker@xxxxxxxx <mailto:Chris.Barker@xxxxxxxx>
> > >>>>
> > >>>> _______________________________________________
> > >>>> NOTE: All exchanges posted to Unidata maintained email lists are
> > >>>> recorded in the Unidata inquiry tracking system and made publicly
> > >>>> available through the web. Users who post to any of the lists we
> > >>>> maintain are reminded to remove any personal information that they
> > >>>> do not want to be made public.
> > >>>>
> > >>>>
> > >>>> netcdfgroup mailing list
> > >>>> netcdfgroup@xxxxxxxxxxxxxxxx
> > >>>> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
> > >>>
> > >>> **********************
> > >>> "The contents of this message do not reflect any position of the
> U.S. Government or NOAA."
> > >>> **********************
> > >>> Roy Mendelssohn
> > >>> Supervisory Operations Research Analyst
> > >>> NOAA/NMFS
> > >>> Environmental Research Division
> > >>> Southwest Fisheries Science Center
> > >>> ***Note new address and phone***
> > >>> 110 Shaffer Road
> > >>> Santa Cruz, CA 95060
> > >>> Phone: (831)-420-3666
> > >>> Fax: (831) 420-3980
> > >>> e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
> > >>>
> > >>> "Old age and treachery will overcome youth and skill."
> > >>> "From those who have been given much, much will be expected"
> > >>> "the arc of the moral universe is long, but it bends toward justice"
> -MLK Jr.
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>>
> > >>> Christopher Barker, Ph.D.
> > >>> Oceanographer
> > >>>
> > >>> Emergency Response Division
> > >>> NOAA/NOS/OR&R (206) 526-6959 voice
> > >>> 7600 Sand Point Way NE (206) 526-6329 fax
> > >>> Seattle, WA 98115 (206) 526-6317 main reception
> > >>>
> > >>> Chris.Barker@xxxxxxxx
> > >>
> > >> **********************
> > >> "The contents of this message do not reflect any position of the U.S .
> Government or NOAA."
> > >> **********************
> > >> Roy Mendelssohn
> > >> Supervisory Operations Research Analyst
> > >> NOAA/NMFS
> > >> Environmental Research Division
> > >> Southwest Fisheries Science Center
> > >> ***Note new address and phone***
> > >> 110 Shaffer Road
> > >> Santa Cruz, CA 95060
> > >> Phone: (831)-420-3666
> > >> Fax: (831) 420-3980
> > >> e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
> > >>
> > >> "Old age and treachery will overcome youth and skill."
> > >> "From those who have been given much, much will be expected"
> > >> "the arc of the moral universe is long, but it bends toward justice"
> -MLK Jr.
> > >>
> > >
> > > = = =
> > > Nathan Potter ndp at opendap.org
> > > OPeNDAP, Inc. +1.541.231.3317
> > >
> >
> > = = =
> > Nathan Potter ndp at opendap.org
> > OPeNDAP, Inc. +1.541.231.3317
> >
> >
> >
> >
> > --
> >
> > Christopher Barker, Ph.D.
> > Oceanographer
> >
> > Emergency Response Division
> > NOAA/NOS/OR&R (206) 526-6959 voice
> > 7600 Sand Point Way NE (206) 526-6329 fax
> > Seattle, WA 98115 (206) 526-6317 main reception
> >
> > Chris.Barker@xxxxxxxx
> >
>
> = = =
> Nathan Potter ndp at opendap.org
> OPeNDAP, Inc. +1.541.231.3317
>
>
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@xxxxxxxx
