Re: [netcdfgroup] [support] Re: Failure accessing an HYRAX OpenDAP server...

  • To: Chris Barker <Chris.Barker@xxxxxxxx>
  • Subject: Re: [netcdfgroup] [support] Re: Failure accessing an HYRAX OpenDAP server...
  • From: Roy Mendelssohn - NOAA Federal <roy.mendelssohn@xxxxxxxx>
  • Date: Mon, 17 Oct 2016 16:45:54 -0700
Looking at pydap source,  it uses the .dds or .das to get the shape of the 
array, and that is defined in the those files.  If however, I assume but don't 
know for certain that the dimension ocean_times is size ntimes, which is 
undefined  (check the .html page), if a program uses that to figure out the 
size and location of the array, it will fail.

Don't know for certain this is the case.....

-Roy

> On Oct 17, 2016, at 4:37 PM, Nathan Potter <ndp@xxxxxxxxxxx> wrote:
> 
> Hi,
> 
> Seems like no one has tried to actually ask for the NcML file:
> 
> http://ingria.coas.oregonstate.edu/opendap/aggregation/ocean_time_aggregation.ncml
> 
> 
> N
> 
> 
>> On Oct 17, 2016, at 4:35 PM, Roy Mendelssohn - NOAA Federal 
>> <roy.mendelssohn@xxxxxxxx> wrote:
>> 
>> Hi Emilio:
>> 
>> I believe this is a NaNOOS server.  If you can read over the thread,, any 
>> insight would be appreciated,  What is in the .ncml file could be of 
>> interest.
>> 
>> Thanks,
>> 
>> -roy
>> 
>>> On Oct 17, 2016, at 4:20 PM, Chris Barker <Chris.Barker@xxxxxxxx> wrote:
>>> 
>>> On Mon, Oct 17, 2016 at 3:57 PM, Roy Mendelssohn - NOAA Federal 
>>> <roy.mendelssohn@xxxxxxxx> wrote:
>>> I find that if you go to the html page,  many of the variables at the top 
>>> can't be accessed requesting ascii, so there are clearly problems with how 
>>> this is set up 
>>> 
>>> that does point to server issues.
>>> 
>>> I am wondering if these are defined in the .ncml.  It would be interesting 
>>> to see what the .ncml file looks like.
>>> 
>>> One thing to consider also is pydap does not call the C library IIRC, it is 
>>> a pure python implementation
>>> 
>>> Exactly -- so it takes the netcdf C lib out of the picture -- which means 
>>> there is a protocol that pydap and hyrax are speaking that lets the data be 
>>> accessed...
>>> 
>>> If anyone knows how to see what actual requests are being made by the two, 
>>> we might be able to figure out where the failure is...
>>> 
>>> -CHB
>>> 
>>> 
>>> -Roy
>>> 
>>>> On Oct 17, 2016, at 3:33 PM, dmh@xxxxxxxx wrote:
>>>> 
>>>> The difference with pydap is probably that you adding a constraint
>>>> that is causing the server to ignore whatever variable
>>>> is causing the problem. If you ask pydap for the whole
>>>> dataset, does it fail?
>>>> (e.g. something like:
>>>> http://ingria.coas.oregonstate.edu/opendap/aggregation/ocean_time_aggregation.ncml.ascii
>>>> =Dennis Heimbigner
>>>> Unidata
>>>> 
>>>> On 10/17/2016 4:16 PM, Chris Barker wrote:
>>>>> On Mon, Oct 17, 2016 at 2:36 PM, dmh@xxxxxxxx <mailto:dmh@xxxxxxxx>
>>>>> <dmh@xxxxxxxx <mailto:dmh@xxxxxxxx>> wrote:
>>>>> 
>>>>>  The log and show=fetch appear to not work under python
>>>>>  for some reason; stdout/stderr redirect?
>>>>> 
>>>>> 
>>>>> I don't think so -- and I do get a lot of "Log:prefetch" messages, so
>>>>> stdout.stderr seems to be working.
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>  In any case I can duplicate the error using wget e.g.
>>>>>    wget
>>>>>  
>>>>> 'http://ingria.coas.oregonstate.edu:80/opendap/aggregation/ocean_time_aggregation.ncml.dods
>>>>>  
>>>>> <http://ingria.coas.oregonstate.edu:80/opendap/aggregation/ocean_time_aggregation.ncml.dods>'
>>>>> 
>>>>> 
>>>>> 
>>>>> hmm, I don't seem to have wget, but with curl, I get the below results,
>>>>> ending in:
>>>>> 
>>>>> Data:
>>>>> 
>>>>> Error {
>>>>> 
>>>>>  code = -111;
>>>>> 
>>>>>  message = "libdap error transmitting DataDDS: Could not read the
>>>>> variable `ntimes'.";
>>>>> 
>>>>> -- if that's the fatal error, then yup -- not the netcdf lib -- and yet,
>>>>> we can get the data with PyDAP, and if I put this:
>>>>> 
>>>>> 
>>>>> http://ingria.coas.oregonstate.edu/opendap/aggregation/ocean_time_aggregation.ncml.ascii?u[0][0][0:10][0:10]
>>>>> 
>>>>> into my browser, I do get data.
>>>>> 
>>>>> I have asked this on support@xxxxxxxxxxx <mailto:support@xxxxxxxxxxx> --
>>>>> and they have pointed out that that is a pretty old version of Hyrax --
>>>>> _maybe_ that's the problem.
>>>>> 
>>>>> -CHB
>>>>> 
>>>>> 
>>>>> $ curl
>>>>> 'http://ingria.coas.oregonstate.edu:80/opendap/aggregation/ocean_time_aggregation.ncml.dods'
>>>>> 
>>>>> Dataset {
>>>>> 
>>>>>  Int32 ntimes;
>>>>> 
>>>>>  Int32 ndtfast;
>>>>> 
>>>>>  Float64 dt;
>>>>> 
>>>>>  Float64 dtfast;
>>>>> 
>>>>>  Float64 dstart;
>>>>> 
>>>>>  Int32 nHIS;
>>>>> 
>>>>>  Int32 ndefHIS;
>>>>> 
>>>>>  Int32 nRST;
>>>>> 
>>>>>  Int32 ntsAVG;
>>>>> 
>>>>>  Int32 nAVG;
>>>>> 
>>>>>  Int32 ndefAVG;
>>>>> 
>>>>>  Float64 Falpha;
>>>>> 
>>>>>  Float64 Fbeta;
>>>>> 
>>>>>  Float64 Fgamma;
>>>>> 
>>>>>  Float64 nl_tnu2[tracer = 2];
>>>>> 
>>>>>  Float64 nl_visc2;
>>>>> 
>>>>>  Float64 Akt_bak[tracer = 2];
>>>>> 
>>>>>  Float64 Akv_bak;
>>>>> 
>>>>>  Float64 Akk_bak;
>>>>> 
>>>>>  Float64 Akp_bak;
>>>>> 
>>>>>  Float64 rdrg;
>>>>> 
>>>>>  Float64 rdrg2;
>>>>> 
>>>>>  Float64 Zob;
>>>>> 
>>>>>  Float64 Zos;
>>>>> 
>>>>>  Float64 Znudg;
>>>>> 
>>>>>  Float64 M2nudg;
>>>>> 
>>>>>  Float64 M3nudg;
>>>>> 
>>>>>  Float64 Tnudg[tracer = 2];
>>>>> 
>>>>>  Float64 FSobc_in[boundary = 4];
>>>>> 
>>>>>  Float64 FSobc_out[boundary = 4];
>>>>> 
>>>>>  Float64 M2obc_in[boundary = 4];
>>>>> 
>>>>>  Float64 M2obc_out[boundary = 4];
>>>>> 
>>>>>  Float64 Tobc_in[boundary = 4][tracer = 2];
>>>>> 
>>>>>  Float64 Tobc_out[boundary = 4][tracer = 2];
>>>>> 
>>>>>  Float64 M3obc_in[boundary = 4];
>>>>> 
>>>>>  Float64 M3obc_out[boundary = 4];
>>>>> 
>>>>>  Float64 rho0;
>>>>> 
>>>>>  Float64 gamma2;
>>>>> 
>>>>>  Int32 LtracerSrc[tracer = 2];
>>>>> 
>>>>>  Int32 spherical;
>>>>> 
>>>>>  Float64 xl;
>>>>> 
>>>>>  Float64 el;
>>>>> 
>>>>>  Int32 Vtransform;
>>>>> 
>>>>>  Int32 Vstretching;
>>>>> 
>>>>>  Float64 theta_s;
>>>>> 
>>>>>  Float64 theta_b;
>>>>> 
>>>>>  Float64 Tcline;
>>>>> 
>>>>>  Float64 hc;
>>>>> 
>>>>>  Float64 s_rho[s_rho = 40];
>>>>> 
>>>>>  Float64 s_w[s_w = 41];
>>>>> 
>>>>>  Grid {
>>>>> 
>>>>>    Array:
>>>>> 
>>>>>      Float64 Cs_r[s_rho = 40];
>>>>> 
>>>>>    Maps:
>>>>> 
>>>>>      Float64 s_rho[s_rho = 40];
>>>>> 
>>>>>  } Cs_r;
>>>>> 
>>>>>  Grid {
>>>>> 
>>>>>    Array:
>>>>> 
>>>>>      Float64 Cs_w[s_w = 41];
>>>>> 
>>>>>    Maps:
>>>>> 
>>>>>      Float64 s_w[s_w = 41];
>>>>> 
>>>>>  } Cs_w;
>>>>> 
>>>>>  Float64 user[Nuser = 25];
>>>>> 
>>>>>  Float64 h[eta_rho = 522][xi_rho = 310];
>>>>> 
>>>>>  Float64 f[eta_rho = 522][xi_rho = 310];
>>>>> 
>>>>>  Float64 pm[eta_rho = 522][xi_rho = 310];
>>>>> 
>>>>>  Float64 pn[eta_rho = 522][xi_rho = 310];
>>>>> 
>>>>>  Float64 lon_rho[eta_rho = 522][xi_rho = 310];
>>>>> 
>>>>>  Float64 lat_rho[eta_rho = 522][xi_rho = 310];
>>>>> 
>>>>>  Float64 lon_u[eta_u = 522][xi_u = 309];
>>>>> 
>>>>>  Float64 lat_u[eta_u = 522][xi_u = 309];
>>>>> 
>>>>>  Float64 lon_v[eta_v = 521][xi_v = 310];
>>>>> 
>>>>>  Float64 lat_v[eta_v = 521][xi_v = 310];
>>>>> 
>>>>>  Float64 lon_psi[eta_psi = 521][xi_psi = 309];
>>>>> 
>>>>>  Float64 lat_psi[eta_psi = 521][xi_psi = 309];
>>>>> 
>>>>>  Float64 angle[eta_rho = 522][xi_rho = 310];
>>>>> 
>>>>>  Float64 mask_rho[eta_rho = 522][xi_rho = 310];
>>>>> 
>>>>>  Float64 mask_u[eta_u = 522][xi_u = 309];
>>>>> 
>>>>>  Float64 mask_v[eta_v = 521][xi_v = 310];
>>>>> 
>>>>>  Float64 mask_psi[eta_psi = 521][xi_psi = 309];
>>>>> 
>>>>>  Float64 ocean_time[ocean_time = 1026];
>>>>> 
>>>>>  Float32 zeta[ocean_time = 1026][eta_rho = 522][xi_rho = 310];
>>>>> 
>>>>>  Float32 ubar[ocean_time = 1026][eta_u = 522][xi_u = 309];
>>>>> 
>>>>>  Float32 vbar[ocean_time = 1026][eta_v = 521][xi_v = 310];
>>>>> 
>>>>>  Float32 u[ocean_time = 1026][s_rho = 40][eta_u = 522][xi_u = 309];
>>>>> 
>>>>>  Float32 v[ocean_time = 1026][s_rho = 40][eta_v = 521][xi_v = 310];
>>>>> 
>>>>>  Float32 w[ocean_time = 1026][s_w = 41][eta_rho = 522][xi_rho = 310];
>>>>> 
>>>>>  Float32 temp[ocean_time = 1026][s_rho = 40][eta_rho = 522][xi_rho =
>>>>> 310];
>>>>> 
>>>>>  Float32 salt[ocean_time = 1026][s_rho = 40][eta_rho = 522][xi_rho =
>>>>> 310];
>>>>> 
>>>>>  Float32 AKv[ocean_time = 1026][s_w = 41][eta_rho = 522][xi_rho = 310];
>>>>> 
>>>>>  Float32 AKt[ocean_time = 1026][s_w = 41][eta_rho = 522][xi_rho = 310];
>>>>> 
>>>>>  Float32 AKs[ocean_time = 1026][s_w = 41][eta_rho = 522][xi_rho = 310];
>>>>> 
>>>>>  Float32 tke[ocean_time = 1026][s_w = 41][eta_rho = 522][xi_rho = 310];
>>>>> 
>>>>>  Float32 shflux[ocean_time = 1026][eta_rho = 522][xi_rho = 310];
>>>>> 
>>>>>  Float32 latent[ocean_time = 1026][eta_rho = 522][xi_rho = 310];
>>>>> 
>>>>>  Float32 sensible[ocean_time = 1026][eta_rho = 522][xi_rho = 310];
>>>>> 
>>>>>  Float32 lwrad[ocean_time = 1026][eta_rho = 522][xi_rho = 310];
>>>>> 
>>>>>  Float32 swrad[ocean_time = 1026][eta_rho = 522][xi_rho = 310];
>>>>> 
>>>>> } ocean_time_aggregation.ncml;
>>>>> 
>>>>> Data:
>>>>> 
>>>>> Error {
>>>>> 
>>>>>  code = -111;
>>>>> 
>>>>>  message = "libdap error transmitting DataDDS: Could not read the
>>>>> variable `ntimes'.";
>>>>> 
>>>>> };
>>>>> 
>>>>> 
>>>>> --
>>>>> 
>>>>> Christopher Barker, Ph.D.
>>>>> Oceanographer
>>>>> 
>>>>> Emergency Response Division
>>>>> NOAA/NOS/OR&R            (206) 526-6959   voice
>>>>> 7600 Sand Point Way NE   (206) 526-6329   fax
>>>>> Seattle, WA  98115       (206) 526-6317   main reception
>>>>> 
>>>>> Chris.Barker@xxxxxxxx <mailto:Chris.Barker@xxxxxxxx>
>>>> 
>>>> _______________________________________________
>>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>>> recorded in the Unidata inquiry tracking system and made publicly
>>>> available through the web.  Users who post to any of the lists we
>>>> maintain are reminded to remove any personal information that they
>>>> do not want to be made public.
>>>> 
>>>> 
>>>> netcdfgroup mailing list
>>>> netcdfgroup@xxxxxxxxxxxxxxxx
>>>> For list information or to unsubscribe,  visit: 
>>>> http://www.unidata.ucar.edu/mailing_lists/
>>> 
>>> **********************
>>> "The contents of this message do not reflect any position of the U.S. 
>>> Government or NOAA."
>>> **********************
>>> Roy Mendelssohn
>>> Supervisory Operations Research Analyst
>>> NOAA/NMFS
>>> Environmental Research Division
>>> Southwest Fisheries Science Center
>>> ***Note new address and phone***
>>> 110 Shaffer Road
>>> Santa Cruz, CA 95060
>>> Phone: (831)-420-3666
>>> Fax: (831) 420-3980
>>> e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
>>> 
>>> "Old age and treachery will overcome youth and skill."
>>> "From those who have been given much, much will be expected"
>>> "the arc of the moral universe is long, but it bends toward justice" -MLK 
>>> Jr.
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> 
>>> Christopher Barker, Ph.D.
>>> Oceanographer
>>> 
>>> Emergency Response Division
>>> NOAA/NOS/OR&R            (206) 526-6959   voice
>>> 7600 Sand Point Way NE   (206) 526-6329   fax
>>> Seattle, WA  98115       (206) 526-6317   main reception
>>> 
>>> Chris.Barker@xxxxxxxx
>> 
>> **********************
>> "The contents of this message do not reflect any position of the U.S. 
>> Government or NOAA."
>> **********************
>> Roy Mendelssohn
>> Supervisory Operations Research Analyst
>> NOAA/NMFS
>> Environmental Research Division
>> Southwest Fisheries Science Center
>> ***Note new address and phone***
>> 110 Shaffer Road
>> Santa Cruz, CA 95060
>> Phone: (831)-420-3666
>> Fax: (831) 420-3980
>> e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
>> 
>> "Old age and treachery will overcome youth and skill."
>> "From those who have been given much, much will be expected" 
>> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
>> 
> 
> = = =
> Nathan Potter                        ndp at opendap.org
> OPeNDAP, Inc.                        +1.541.231.3317
> 

**********************
"The contents of this message do not reflect any position of the U.S. 
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new address and phone***
110 Shaffer Road
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected" 
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.