Re: [netcdfgroup] [support] Re: Failure accessing an HYRAX OpenDAP server...

Thanks EMilio,

I'll reach out to Alexandre.

-Chris


On Wed, Oct 19, 2016 at 8:39 AM, Emilio Mayorga <emiliomayorga@xxxxxxxxx>
wrote:

> Just to clarify, I made references to potential misconfigurations on the
> hyrax server only b/c I saw statements to that effect in the previous
> emails on this thread. I'm not in a position to make assessments myself.
> And as I mentioned, NANOOS is also not in a real position to offer
> significant changes to that hyrax server.
>
> Any recommendations to upgrade to a more recent version should be made
> directly to the server maintainer; I don't remember her name, but the model
> PI is Alexandre Kurapov, kurapov@xxxxxxxxxxxxxxxxxxxxx Please cc me.
>
> Also, Chris Barker replied separately saying that their needs our almost
> always met by THREDDS servers, so he thought our operational and actively
> maintained NANOOS THREDDS server may meet his needs. If the hyrax server in
> question was in fact originally deployed only to meet NOAA ORR needs, and
> those needs can now be met by our THREDDS server, the decision might be to
> discontinue the hyrax server. But that'll be up to Alexandre.
>
> Cheers,
> -Emilio
>
>
> On Wed, Oct 19, 2016 at 5:34 AM, Nathan Potter <ndp@xxxxxxxxxxx> wrote:
>
>>
>> I would appreciate knowing what part of the behavior/output of the
>> ingria.coas.oregonstate.edu Hyrax server has led to the comments that it
>> is not configured correctly.
>>
>> - Is it simply that the server is producing an error when trying to
>> transmit the ântimesâ variable?
>> - Is there something specific in the NcML content that speaks directly to
>> this?
>> - In the context of this thread does âconfiguration problem/issueâ mean
>> an improperly constructed NcML file?
>>
>> Can anyone provide some insight?
>>
>> By asking these questions I am trying to understand, and not in anyway
>> dispute, the assertion that the server is âmisconfiguredâ. I just want to
>> tease apart the possibility that we are seeing a fundamental bug problem
>> from a configuration issue.
>>
>> I do know that at NASAâs request and expense a significant number of
>> changes and improvements have been made to Hyraxâs NcML features, but this
>> work happened long after the release of version 1.9.0.
>>
>> In light of this I think the first step should be to instigate upgrading
>> the server, as version 1.9.0 is a long way back and not only does the
>> latest release (1.13.1) offer an improved NcML implementation it also
>> contains numerous security updates.
>>
>> Regardless, any clarification that could be offered w.r.t. the
>> mis-configuration assertion would be very helpful to me.
>>
>>
>> Sincerely,
>>
>> Nathan
>>
>>
>>
>>
>> > On Oct 18, 2016, at 11:30 PM, Emilio Mayorga <emiliomayorga@xxxxxxxxx>
>> wrote:
>> >
>> > I'm afraid I don't have much helpful insight to bring into this. The
>> model behind that Hyrax server is indeed a NANOOS model, but the hyrax
>> server itself was not set up and is not maintained by one of us in the
>> NANOOS DMAC team. In fact, if I remember correctly the server is somewhat
>> duplicative of a THREDDS server we do maintain for the same model output;
>> it was setup about two years ago, I think to support an application at NOAA
>> ORR that took advantage of some specific hyrax capability missing in
>> THREDDS' OPeNDAP (Chris Barker would know better than me), partly through
>> help and persuasion from Rich Signell; again, sketchy memory here. I was
>> never thrilled with the idea of having a hyrax server in the mix when none
>> of us had the capacity to invest in learning how to properly maintain it,
>> and already had a THREDDS server for the same model. I'm not surprised to
>> hear it's an old version, too.
>> >
>> > Sorry ...
>> >
>> > -Emilio
>> >
>> >
>> > On Mon, Oct 17, 2016 at 5:36 PM, Chris Barker <chris.barker@xxxxxxxx>
>> wrote:
>> > Folks,
>> >
>> > I manged to do a tcpdump, and can (sortof) see what's going on --
>> though I can't really make sense of it -- maybe one of you can?
>> >
>> > This is running the enclosed script, to it first does the successful
>> pydap request, then the unsuccessful netCDF4 request.
>> >
>> > (though it looking like this is a poorly configured server...)
>> >
>> > -CHB
>> >
>> >
>> > On Mon, Oct 17, 2016 at 4:49 PM, Nathan Potter <ndp@xxxxxxxxxxx> wrote:
>> > By which I meant - just deference the URL:
>> >
>> >     http://ingria.coas.oregonstate.edu/opendap/aggregation/
>> ocean_time_aggregation.ncml
>> >
>> > To get the NcML file:
>> >
>> >     curl http://ingria.coas.oregonstate.edu/opendap/aggregation/
>> ocean_time_aggregation.ncml
>> >
>> >
>> >
>> > Nathan
>> >
>> >
>> >
>> > > On Oct 17, 2016, at 4:37 PM, Nathan Potter <ndp@xxxxxxxxxxx> wrote:
>> > >
>> > > Hi,
>> > >
>> > > Seems like no one has tried to actually ask for the NcML file:
>> > >
>> > > http://ingria.coas.oregonstate.edu/opendap/aggregation/
>> ocean_time_aggregation.ncml
>> > >
>> > >
>> > > N
>> > >
>> > >
>> > >> On Oct 17, 2016, at 4:35 PM, Roy Mendelssohn - NOAA Federal <
>> roy.mendelssohn@xxxxxxxx> wrote:
>> > >>
>> > >> Hi Emilio:
>> > >>
>> > >> I believe this is a NaNOOS server.  If you can read over the
>> thread,, any insight would be appreciated,  What is in the .ncml file could
>> be of interest.
>> > >>
>> > >> Thanks,
>> > >>
>> > >> -roy
>> > >>
>> > >>> On Oct 17, 2016, at 4:20 PM, Chris Barker <Chris.Barker@xxxxxxxx>
>> wrote:
>> > >>>
>> > >>> On Mon, Oct 17, 2016 at 3:57 PM, Roy Mendelssohn - NOAA Federal <
>> roy.mendelssohn@xxxxxxxx> wrote:
>> > >>> I find that if you go to the html page,  many of the variables at
>> the top can't be accessed requesting ascii, so there are clearly problems
>> with how this is set up
>> > >>>
>> > >>> that does point to server issues.
>> > >>>
>> > >>> I am wondering if these are defined in the .ncml.  It would be
>> interesting to see what the .ncml file looks like.
>> > >>>
>> > >>> One thing to consider also is pydap does not call the C library
>> IIRC, it is a pure python implementation
>> > >>>
>> > >>> Exactly -- so it takes the netcdf C lib out of the picture -- which
>> means there is a protocol that pydap and hyrax are speaking that lets the
>> data be accessed...
>> > >>>
>> > >>> If anyone knows how to see what actual requests are being made by
>> the two, we might be able to figure out where the failure is...
>> > >>>
>> > >>> -CHB
>> > >>>
>> > >>>
>> > >>> -Roy
>> > >>>
>> > >>>> On Oct 17, 2016, at 3:33 PM, dmh@xxxxxxxx wrote:
>> > >>>>
>> > >>>> The difference with pydap is probably that you adding a constraint
>> > >>>> that is causing the server to ignore whatever variable
>> > >>>> is causing the problem. If you ask pydap for the whole
>> > >>>> dataset, does it fail?
>> > >>>> (e.g. something like:
>> > >>>> http://ingria.coas.oregonstate.edu/opendap/aggregation/
>> ocean_time_aggregation.ncml.ascii
>> > >>>> =Dennis Heimbigner
>> > >>>> Unidata
>> > >>>>
>> > >>>> On 10/17/2016 4:16 PM, Chris Barker wrote:
>> > >>>>> On Mon, Oct 17, 2016 at 2:36 PM, dmh@xxxxxxxx <mailto:
>> dmh@xxxxxxxx>
>> > >>>>> <dmh@xxxxxxxx <mailto:dmh@xxxxxxxx>> wrote:
>> > >>>>>
>> > >>>>>  The log and show=fetch appear to not work under python
>> > >>>>>  for some reason; stdout/stderr redirect?
>> > >>>>>
>> > >>>>>
>> > >>>>> I don't think so -- and I do get a lot of "Log:prefetch"
>> messages, so
>> > >>>>> stdout.stderr seems to be working.
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>>  In any case I can duplicate the error using wget e.g.
>> > >>>>>    wget
>> > >>>>>  'http://ingria.coas.oregonstate.edu:80/opendap/aggregation/
>> ocean_time_aggregation.ncml.dods
>> > >>>>>  <http://ingria.coas.oregonstate.edu:80/opendap/aggregation/
>> ocean_time_aggregation.ncml.dods>'
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>> hmm, I don't seem to have wget, but with curl, I get the below
>> results,
>> > >>>>> ending in:
>> > >>>>>
>> > >>>>> Data:
>> > >>>>>
>> > >>>>> Error {
>> > >>>>>
>> > >>>>>  code = -111;
>> > >>>>>
>> > >>>>>  message = "libdap error transmitting DataDDS: Could not read the
>> > >>>>> variable `ntimes'.";
>> > >>>>>
>> > >>>>> -- if that's the fatal error, then yup -- not the netcdf lib --
>> and yet,
>> > >>>>> we can get the data with PyDAP, and if I put this:
>> > >>>>>
>> > >>>>>
>> > >>>>> http://ingria.coas.oregonstate.edu/opendap/aggregation/
>> ocean_time_aggregation.ncml.ascii?u[0][0][0:10][0:10]
>> > >>>>>
>> > >>>>> into my browser, I do get data.
>> > >>>>>
>> > >>>>> I have asked this on support@xxxxxxxxxxx <mailto:
>> support@xxxxxxxxxxx> --
>> > >>>>> and they have pointed out that that is a pretty old version of
>> Hyrax --
>> > >>>>> _maybe_ that's the problem.
>> > >>>>>
>> > >>>>> -CHB
>> > >>>>>
>> > >>>>>
>> > >>>>> $ curl
>> > >>>>> 'http://ingria.coas.oregonstate.edu:80/opendap/aggregation/
>> ocean_time_aggregation.ncml.dods'
>> > >>>>>
>> > >>>>> Dataset {
>> > >>>>>
>> > >>>>>  Int32 ntimes;
>> > >>>>>
>> > >>>>>  Int32 ndtfast;
>> > >>>>>
>> > >>>>>  Float64 dt;
>> > >>>>>
>> > >>>>>  Float64 dtfast;
>> > >>>>>
>> > >>>>>  Float64 dstart;
>> > >>>>>
>> > >>>>>  Int32 nHIS;
>> > >>>>>
>> > >>>>>  Int32 ndefHIS;
>> > >>>>>
>> > >>>>>  Int32 nRST;
>> > >>>>>
>> > >>>>>  Int32 ntsAVG;
>> > >>>>>
>> > >>>>>  Int32 nAVG;
>> > >>>>>
>> > >>>>>  Int32 ndefAVG;
>> > >>>>>
>> > >>>>>  Float64 Falpha;
>> > >>>>>
>> > >>>>>  Float64 Fbeta;
>> > >>>>>
>> > >>>>>  Float64 Fgamma;
>> > >>>>>
>> > >>>>>  Float64 nl_tnu2[tracer = 2];
>> > >>>>>
>> > >>>>>  Float64 nl_visc2;
>> > >>>>>
>> > >>>>>  Float64 Akt_bak[tracer = 2];
>> > >>>>>
>> > >>>>>  Float64 Akv_bak;
>> > >>>>>
>> > >>>>>  Float64 Akk_bak;
>> > >>>>>
>> > >>>>>  Float64 Akp_bak;
>> > >>>>>
>> > >>>>>  Float64 rdrg;
>> > >>>>>
>> > >>>>>  Float64 rdrg2;
>> > >>>>>
>> > >>>>>  Float64 Zob;
>> > >>>>>
>> > >>>>>  Float64 Zos;
>> > >>>>>
>> > >>>>>  Float64 Znudg;
>> > >>>>>
>> > >>>>>  Float64 M2nudg;
>> > >>>>>
>> > >>>>>  Float64 M3nudg;
>> > >>>>>
>> > >>>>>  Float64 Tnudg[tracer = 2];
>> > >>>>>
>> > >>>>>  Float64 FSobc_in[boundary = 4];
>> > >>>>>
>> > >>>>>  Float64 FSobc_out[boundary = 4];
>> > >>>>>
>> > >>>>>  Float64 M2obc_in[boundary = 4];
>> > >>>>>
>> > >>>>>  Float64 M2obc_out[boundary = 4];
>> > >>>>>
>> > >>>>>  Float64 Tobc_in[boundary = 4][tracer = 2];
>> > >>>>>
>> > >>>>>  Float64 Tobc_out[boundary = 4][tracer = 2];
>> > >>>>>
>> > >>>>>  Float64 M3obc_in[boundary = 4];
>> > >>>>>
>> > >>>>>  Float64 M3obc_out[boundary = 4];
>> > >>>>>
>> > >>>>>  Float64 rho0;
>> > >>>>>
>> > >>>>>  Float64 gamma2;
>> > >>>>>
>> > >>>>>  Int32 LtracerSrc[tracer = 2];
>> > >>>>>
>> > >>>>>  Int32 spherical;
>> > >>>>>
>> > >>>>>  Float64 xl;
>> > >>>>>
>> > >>>>>  Float64 el;
>> > >>>>>
>> > >>>>>  Int32 Vtransform;
>> > >>>>>
>> > >>>>>  Int32 Vstretching;
>> > >>>>>
>> > >>>>>  Float64 theta_s;
>> > >>>>>
>> > >>>>>  Float64 theta_b;
>> > >>>>>
>> > >>>>>  Float64 Tcline;
>> > >>>>>
>> > >>>>>  Float64 hc;
>> > >>>>>
>> > >>>>>  Float64 s_rho[s_rho = 40];
>> > >>>>>
>> > >>>>>  Float64 s_w[s_w = 41];
>> > >>>>>
>> > >>>>>  Grid {
>> > >>>>>
>> > >>>>>    Array:
>> > >>>>>
>> > >>>>>      Float64 Cs_r[s_rho = 40];
>> > >>>>>
>> > >>>>>    Maps:
>> > >>>>>
>> > >>>>>      Float64 s_rho[s_rho = 40];
>> > >>>>>
>> > >>>>>  } Cs_r;
>> > >>>>>
>> > >>>>>  Grid {
>> > >>>>>
>> > >>>>>    Array:
>> > >>>>>
>> > >>>>>      Float64 Cs_w[s_w = 41];
>> > >>>>>
>> > >>>>>    Maps:
>> > >>>>>
>> > >>>>>      Float64 s_w[s_w = 41];
>> > >>>>>
>> > >>>>>  } Cs_w;
>> > >>>>>
>> > >>>>>  Float64 user[Nuser = 25];
>> > >>>>>
>> > >>>>>  Float64 h[eta_rho = 522][xi_rho = 310];
>> > >>>>>
>> > >>>>>  Float64 f[eta_rho = 522][xi_rho = 310];
>> > >>>>>
>> > >>>>>  Float64 pm[eta_rho = 522][xi_rho = 310];
>> > >>>>>
>> > >>>>>  Float64 pn[eta_rho = 522][xi_rho = 310];
>> > >>>>>
>> > >>>>>  Float64 lon_rho[eta_rho = 522][xi_rho = 310];
>> > >>>>>
>> > >>>>>  Float64 lat_rho[eta_rho = 522][xi_rho = 310];
>> > >>>>>
>> > >>>>>  Float64 lon_u[eta_u = 522][xi_u = 309];
>> > >>>>>
>> > >>>>>  Float64 lat_u[eta_u = 522][xi_u = 309];
>> > >>>>>
>> > >>>>>  Float64 lon_v[eta_v = 521][xi_v = 310];
>> > >>>>>
>> > >>>>>  Float64 lat_v[eta_v = 521][xi_v = 310];
>> > >>>>>
>> > >>>>>  Float64 lon_psi[eta_psi = 521][xi_psi = 309];
>> > >>>>>
>> > >>>>>  Float64 lat_psi[eta_psi = 521][xi_psi = 309];
>> > >>>>>
>> > >>>>>  Float64 angle[eta_rho = 522][xi_rho = 310];
>> > >>>>>
>> > >>>>>  Float64 mask_rho[eta_rho = 522][xi_rho = 310];
>> > >>>>>
>> > >>>>>  Float64 mask_u[eta_u = 522][xi_u = 309];
>> > >>>>>
>> > >>>>>  Float64 mask_v[eta_v = 521][xi_v = 310];
>> > >>>>>
>> > >>>>>  Float64 mask_psi[eta_psi = 521][xi_psi = 309];
>> > >>>>>
>> > >>>>>  Float64 ocean_time[ocean_time = 1026];
>> > >>>>>
>> > >>>>>  Float32 zeta[ocean_time = 1026][eta_rho = 522][xi_rho = 310];
>> > >>>>>
>> > >>>>>  Float32 ubar[ocean_time = 1026][eta_u = 522][xi_u = 309];
>> > >>>>>
>> > >>>>>  Float32 vbar[ocean_time = 1026][eta_v = 521][xi_v = 310];
>> > >>>>>
>> > >>>>>  Float32 u[ocean_time = 1026][s_rho = 40][eta_u = 522][xi_u =
>> 309];
>> > >>>>>
>> > >>>>>  Float32 v[ocean_time = 1026][s_rho = 40][eta_v = 521][xi_v =
>> 310];
>> > >>>>>
>> > >>>>>  Float32 w[ocean_time = 1026][s_w = 41][eta_rho = 522][xi_rho =
>> 310];
>> > >>>>>
>> > >>>>>  Float32 temp[ocean_time = 1026][s_rho = 40][eta_rho =
>> 522][xi_rho =
>> > >>>>> 310];
>> > >>>>>
>> > >>>>>  Float32 salt[ocean_time = 1026][s_rho = 40][eta_rho =
>> 522][xi_rho =
>> > >>>>> 310];
>> > >>>>>
>> > >>>>>  Float32 AKv[ocean_time = 1026][s_w = 41][eta_rho = 522][xi_rho =
>> 310];
>> > >>>>>
>> > >>>>>  Float32 AKt[ocean_time = 1026][s_w = 41][eta_rho = 522][xi_rho =
>> 310];
>> > >>>>>
>> > >>>>>  Float32 AKs[ocean_time = 1026][s_w = 41][eta_rho = 522][xi_rho =
>> 310];
>> > >>>>>
>> > >>>>>  Float32 tke[ocean_time = 1026][s_w = 41][eta_rho = 522][xi_rho =
>> 310];
>> > >>>>>
>> > >>>>>  Float32 shflux[ocean_time = 1026][eta_rho = 522][xi_rho = 310];
>> > >>>>>
>> > >>>>>  Float32 latent[ocean_time = 1026][eta_rho = 522][xi_rho = 310];
>> > >>>>>
>> > >>>>>  Float32 sensible[ocean_time = 1026][eta_rho = 522][xi_rho = 310];
>> > >>>>>
>> > >>>>>  Float32 lwrad[ocean_time = 1026][eta_rho = 522][xi_rho = 310];
>> > >>>>>
>> > >>>>>  Float32 swrad[ocean_time = 1026][eta_rho = 522][xi_rho = 310];
>> > >>>>>
>> > >>>>> } ocean_time_aggregation.ncml;
>> > >>>>>
>> > >>>>> Data:
>> > >>>>>
>> > >>>>> Error {
>> > >>>>>
>> > >>>>>  code = -111;
>> > >>>>>
>> > >>>>>  message = "libdap error transmitting DataDDS: Could not read the
>> > >>>>> variable `ntimes'.";
>> > >>>>>
>> > >>>>> };
>> > >>>>>
>> > >>>>>
>> > >>>>> --
>> > >>>>>
>> > >>>>> Christopher Barker, Ph.D.
>> > >>>>> Oceanographer
>> > >>>>>
>> > >>>>> Emergency Response Division
>> > >>>>> NOAA/NOS/OR&R            (206) 526-6959   voice
>> > >>>>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> > >>>>> Seattle, WA  98115       (206) 526-6317   main reception
>> > >>>>>
>> > >>>>> Chris.Barker@xxxxxxxx <mailto:Chris.Barker@xxxxxxxx>
>> > >>>>
>> > >>>> _______________________________________________
>> > >>>> NOTE: All exchanges posted to Unidata maintained email lists are
>> > >>>> recorded in the Unidata inquiry tracking system and made publicly
>> > >>>> available through the web.  Users who post to any of the lists we
>> > >>>> maintain are reminded to remove any personal information that they
>> > >>>> do not want to be made public.
>> > >>>>
>> > >>>>
>> > >>>> netcdfgroup mailing list
>> > >>>> netcdfgroup@xxxxxxxxxxxxxxxx
>> > >>>> For list information or to unsubscribe,  visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>> > >>>
>> > >>> **********************
>> > >>> "The contents of this message do not reflect any position of the
>> U.S. Government or NOAA."
>> > >>> **********************
>> > >>> Roy Mendelssohn
>> > >>> Supervisory Operations Research Analyst
>> > >>> NOAA/NMFS
>> > >>> Environmental Research Division
>> > >>> Southwest Fisheries Science Center
>> > >>> ***Note new address and phone***
>> > >>> 110 Shaffer Road
>> > >>> Santa Cruz, CA 95060
>> > >>> Phone: (831)-420-3666
>> > >>> Fax: (831) 420-3980
>> > >>> e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
>> > >>>
>> > >>> "Old age and treachery will overcome youth and skill."
>> > >>> "From those who have been given much, much will be expected"
>> > >>> "the arc of the moral universe is long, but it bends toward
>> justice" -MLK Jr.
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>> --
>> > >>>
>> > >>> Christopher Barker, Ph.D.
>> > >>> Oceanographer
>> > >>>
>> > >>> Emergency Response Division
>> > >>> NOAA/NOS/OR&R            (206) 526-6959   voice
>> > >>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> > >>> Seattle, WA  98115       (206) 526-6317   main reception
>> > >>>
>> > >>> Chris.Barker@xxxxxxxx
>> > >>
>> > >> **********************
>> > >> "The contents of this message do not reflect any position of the
>> U.S. Government or NOAA."
>> > >> **********************
>> > >> Roy Mendelssohn
>> > >> Supervisory Operations Research Analyst
>> > >> NOAA/NMFS
>> > >> Environmental Research Division
>> > >> Southwest Fisheries Science Center
>> > >> ***Note new address and phone***
>> > >> 110 Shaffer Road
>> > >> Santa Cruz, CA 95060
>> > >> Phone: (831)-420-3666
>> > >> Fax: (831) 420-3980
>> > >> e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
>> > >>
>> > >> "Old age and treachery will overcome youth and skill."
>> > >> "From those who have been given much, much will be expected"
>> > >> "the arc of the moral universe is long, but it bends toward justice"
>> -MLK Jr.
>> > >>
>> > >
>> > > = = =
>> > > Nathan Potter                        ndp at opendap.org
>> > > OPeNDAP, Inc.                        +1.541.231.3317
>> > >
>> >
>> > = = =
>> > Nathan Potter                        ndp at opendap.org
>> > OPeNDAP, Inc.                        +1.541.231.3317
>> >
>> >
>> >
>> >
>> > --
>> >
>> > Christopher Barker, Ph.D.
>> > Oceanographer
>> >
>> > Emergency Response Division
>> > NOAA/NOS/OR&R            (206) 526-6959   voice
>> > 7600 Sand Point Way NE   (206) 526-6329   fax
>> > Seattle, WA  98115       (206) 526-6317   main reception
>> >
>> > Chris.Barker@xxxxxxxx
>> >
>>
>> = = =
>> Nathan Potter                        ndp at opendap.org
>> OPeNDAP, Inc.                        +1.541.231.3317
>>
>>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker@xxxxxxxx