HI all,
It looks like this is an issue with the built-in Mac curl library dropping
support for openSSL in favor of SecureTransport. SecureTransport will not
read certificates, keys, etc. off of disk. If you have a .dodsrc file
telling netCDF-C to use certs, keys, etc. from disk, it looks like you will
trigger this issue. If you move .dodsrc, or comment the entries out, ncdump
works. (note: I'm not sure which specific entry/entries are causing the
root issue).
In a previous version of OSX, I was able to successfully use the homebrew
netCDF-C library to get data from an ESG node by loading my
credentials.pem file into the system keychain, but as of now, that no
longer works. The only way I can get data that requires a certificate is to
build the entire netCDF-C stack from scratch, making sure I build curl
against openSSL.
I had submitted a homebrew issue and pull request about this:
https://github.com/Homebrew/homebrew-science/pull/4847
which would enable homebrew to build netCDF-C against a curl with openSSL.
Looks like I need to let the homebrew devs know that the keychain trick
does not work anymore, and so the only way to allow users to do
authorization/authentication on the latest MacOS would be to build against
curl with openSSL enabled.
Sean
On Thu, May 4, 2017 at 7:33 AM, Capehart, William J <
William.Capehart@xxxxxxxxx> wrote:
> Is this an issue then with something interesting Xcode?
>
> Sent from my iPhone
>
> On May 4, 2017, at 00:12, Larry Baker <baker@xxxxxxxx> wrote:
>
> See https://github.com/Unidata/netcdf-c/issues/122 around July 10, 2015
> for a similar/the same bug? My previous inference about strcmp() being the
> culprit is not quite right. The failing code in the thread I found (below)
> is just before the strcmp() call. Looks like the same problem to me. The
> cause at that time was "c" had a value of 67, while the nameindices[] array
> was declared [26].
>
> doutriaux1@maryam:[build]:[10587]> lldb ~/build//install/Externals/bin/ncdump
> http://test.opendap.org/opendap/hyrax/netcdf/examples/
> ECMWF_ERA-40_subset.nc
> (lldb) target create "/Users/doutriaux1/build//
> install/Externals/bin/ncdump"
> Current executable set to
> '/Users/doutriaux1/build//install/Externals/bin/ncdump'
> (x86_64).
> (lldb) settings set -- target.run-args "http://test.opendap.org/
> opendap/hyrax/netcdf/examples/ECMWF_ERA-40_subset.nc"
> (lldb) r
> Process 45302 launched:
> '/Users/doutriaux1/build//install/Externals/bin/ncdump'
> (x86_64)
> Process 45302 stopped
> * thread #1: tid = 0xebf1, 0x00000001000b09c7 libnetcdf.7.dylib`
> occurlflagbyname(name=0x0000000100186749) + 327 at occurlflags.c:301,
> queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1,
> address=0x2)
> frame #0: 0x00000001000b09c7
> libnetcdf.7.dylib`occurlflagbyname(name=0x0000000100186749)
> + 327 at occurlflags.c:301
> 298
> 299 if(nameindices[c] == NULL)
> 300 return NULL; /* no possible match */
> -> 301 for(f=nameindices[c];f->name;f++) {
> 302 int cmp = strcmp(name,f->name);
> 303 if(cmp > 0) break; /* We assume sorted */
> 304 if(cmp == 0) return f;
>
>
> Larry Baker
> US Geological Survey
> 650-329-5608 <(650)%20329-5608>
> baker@xxxxxxxx
>
>
>
>
> On May 3, 2017, at 10:57 PM, Larry Baker <baker@xxxxxxxx> wrote:
>
> Looks to me like an illegal memory access in a strcmp() inside
> libnetcdf.11.dylib. The "occurlflagbyname()" is the failing function
> inside libnetcdf?
>
> Larry Baker
> US Geological Survey
> 650-329-5608 <(650)%20329-5608>
> baker@xxxxxxxx
>
>
>
>
> On May 3, 2017, at 10:44 PM, Capehart, William J <
> William.Capehart@xxxxxxxxx> wrote:
>
> OK. I am not familiar with how to use this but here is the output that I
> have for lldb usng …
>
> lldb --debug /usr/local/bin/ncdump http://test.opendap.org/opendap/data/
> nc/3fnoc.nc
>
> https://gist.github.com/wjcapehart/8cc8068a6b6951ee98a2a48dbae02520
>
> And for my output for the fortran program that I wrote.
>
> https://gist.github.com/9a46b7750bc5a9c3ebfdf1916941f490
>
>
> For the command
>
> % otool -L /usr/local/bin/ncdump http://test.opendap.org/opendap/data/
> nc/3fnoc.nc
> /usr/local/bin/ncdump:
> @rpath/libnetcdf.11.dylib (compatibility version 11.0.0,
> current version 11.4.0)
> /usr/local/opt/hdf5/lib/libhdf5_hl.100.dylib
> (compatibility version 101.0.0, current version 101.0.0)
> /usr/local/opt/hdf5/lib/libhdf5.100.dylib (compatibility
> version 101.0.0, current version 101.1.0)
> /usr/local/opt/szip/lib/libsz.2.dylib (compatibility
> version 3.0.0, current version 3.0.0)
> /usr/local/lib/libz.1.dylib (compatibility version 1.0.0,
> current version 1.2.8)
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0,
> current version 1238.50.2)
> /usr/lib/libcurl.4.dylib (compatibility version 7.0.0,
> current version 9.0.0)
> /Applications/Xcode.app/Contents/Developer/Toolchains/
> XcodeDefault.xctoolchain/usr/bin/objdump: 'http://test.opendap.org/
> opendap/data/nc/3fnoc.nc': No such file or directory
>
>
> ------------------------------------------------
> Bill Capehart <William.Capehart@xxxxxxxxx>
> Atmospheric and Environmental Sciences Program Coordinator
> Civil and Environmental Engineering
> 201 Mineral Industries Building
> South Dakota School of Mines and Technology
> 501 East St Joseph Street
> Rapid City, SD 57701-3995
> Ph: +1-605-394-1994 <(605)%20394-1994> Mobile: +1-605-484-4692
> <(605)%20484-4692>
>
> *From: *<netcdfgroup-bounces@xxxxxxxxxxxxxxxx> on behalf of Larry Baker <
> baker@xxxxxxxx>
> *Date: *Wednesday, May 3, 2017 at 16:45 MDT
> *To: *"dmh@xxxxxxxx" <dmh@xxxxxxxx>
> *Cc: *"netcdfgroup@xxxxxxxxxxxxxxxx" <netcdfgroup@xxxxxxxxxxxxxxxx>
> *Subject: *Re: [netcdfgroup] Using NetCDF with OPeNDAP on MacOS Sierra
> (homebrew- and netCDF-dependant software)
>
> Good idea: lldb
>
> Larry Baker
> US Geological Survey
> 650-329-5608 <(650)%20329-5608>
> baker@xxxxxxxx
>
>
>
>
>
> On 3 May 2017, at 3:35:19 PM, dmh@xxxxxxxx wrote:
>
> Is there a osx equivalent of gdb? If so, you could
> probably run the ncdump command using that debugger
> and it will probably catch the seg fault. This would
> allow some idea of where the failure is occurring.
> =Dennis Heimbigner
> Unidata
>
> On 5/3/2017 4:25 PM, Capehart, William J wrote:
>
> To be desperate ;-P is there a way to test things with just the libcurl to
> see where the problem is? I am assuming that most people are getting
> satisfaction with the resident version with Sierra. I'm really on my last
> leg here. I'm not even getting core files with my seg faults on ncdump or
> my simple Fortran program inbuilt to test it... both run just fine on
> Centos .
> Sent from my iPhone
>
> On May 3, 2017, at 15:04, "dmh@xxxxxxxx" <dmh@xxxxxxxx> wrote:
>
> To be pedantic, ncdump (via the netcdf-c library) uses libcurl.
> =Dennis Heimbigner
> Unidata
>
>
> On 5/3/2017 2:31 PM, Roy Mendelssohn - NOAA Federal wrote:
> Hmm - I may be wrong but i thought the ncdump for a remote file just uses
> curl to get the .das and .dds and forms the .cdl response from those.
> Have you tried a different server?
> Say,
> ncdump -h http://coastwatch.pfeg.noaa.gov/erddap/griddap/
> jplAquariusSSSMonthlyV4
> -Roy
>
> On May 3, 2017, at 1:25 PM, Capehart, William J <
> William.Capehart@xxxxxxxxx> wrote:
>
> Roy for the curl I get the followingf
>
> % curl -v http://test.opendap.org/opendap/data/nc/3fnoc.nc.das
> * Trying 52.44.0.41...
> * TCP_NODELAY set
> * Connected to test.opendap.org (52.44.0.41) port 80 (#0)
>
> GET /opendap/data/nc/3fnoc.nc.das HTTP/1.1
> Host: test.opendap.org
> User-Agent: curl/7.51.0
> Accept: */*
>
> < HTTP/1.1 200 OK
> < Date: Wed, 03 May 2017 20:22:21 GMT
> < X-FRAME-OPTIONS: DENY
> < Last-Modified: Tue, 10 Jan 2006 18:32:26 GMT
> < Set-Cookie: JSESSIONID=63E4296077ED5DBE9CFE2C49B169D4ED; Path=/opendap;
> HttpOnly
> < XDODS-Server: dods/3.2
> < XOPeNDAP-Server: bes/3.17.4, csv_handler/1.1.6, dap-server/ascii/4.1.5,
> dap-server/usage/4.2.6, dap-server/www/4.1.5, dapreader_module/0.0.1,
> fileout_gdal/0.10.1, fileout_json/1.0.6, fileout_netcdf/1.4.3,
> fits_handler/1.0.18, freeform_handler/3.9.6, functions/1.1.0,
> gateway_module/1.1.9, gdal_handler/1.0.7, hdf4_handler/3.12.3,
> hdf5_handler/2.3.5, libdap/3.18.3, ncml_moddule/1.4.4,
> netcdf_handler/3.11.6, w10n_handler/1.0.6, xml_data_handler/1.1.2
> < X-DAP: 3.2
> < Content-Description: dods_das
> < Content-Type: text/plain
> < Connection: close
> < Transfer-Encoding: chunked
> <
> Attributes {
> u {
> String units "meter per second";
> String long_name "Vector wind eastward component";
> String missing_value "-32767";
> String scale_factor "0.005";
> }
> v {
> String units "meter per second";
> String long_name "Vector wind northward component";
> String missing_value "-32767";
> String scale_factor "0.005";
> }
> lat {
> String units "degree North";
> }
> lon {
> String units "degree East";
> }
> time {
> String units "hours from base_time";
> }
> NC_GLOBAL {
> String base_time "88-245-00:00:00";
> String title " FNOC UV wind components from 1988-245 to 1988-247.";
> }
> DODS_EXTRA {
> String Unlimited_Dimension "time_a";
> }
> }
> * Curl_http_done: called premature == 0
> * Closing connection 0
>
>
> ------------------------------------------------
> Bill Capehart <William.Capehart@xxxxxxxxx>
> Atmospheric and Environmental Sciences Program Coordinator
> Civil and Environmental Engineering
> 201 Mineral Industries Building
> South Dakota School of Mines and Technology
> 501 East St Joseph Street
> Rapid City, SD 57701-3995
> Ph: +1-605-394-1994 <(605)%20394-1994> Mobile: +1-605-484-4692
> <(605)%20484-4692>
>
> On 5/3/17, 14:19 MDT, "Roy Mendelssohn - NOAA Federal" <
> roy.mendelssohn@xxxxxxxx> wrote:
>
> More specifically, what do you get when you give:
>
> curl -v http://test.opendap.org/opendap/data/nc/3fnoc.nc.das
>
>
> -Roy
>
>
>
> On May 3, 2017, at 1:01 PM, Roy Mendelssohn - NOAA Federal <
> roy.mendelssohn@xxxxxxxx> wrote:
>
> What happens if you just do a curl to get the address but ask for the .das
> instead, and see if your curl and access the site.
>
> -Roy
>
>
> On May 3, 2017, at 12:57 PM, Capehart, William J <
> William.Capehart@xxxxxxxxx> wrote:
>
> Did that with both fink and homebrew including compiling all from source
> each time. Still no traction. :-(
>
> ------------------------------------------------
> Bill Capehart <William.Capehart@xxxxxxxxx>
> Atmospheric and Environmental Sciences Program Coordinator
> Civil and Environmental Engineering
> 201 Mineral Industries Building
> South Dakota School of Mines and Technology
> 501 East St Joseph Street
> Rapid City, SD 57701-3995
> Ph: +1-605-394-1994 <(605)%20394-1994> Mobile: +1-605-484-4692
> <(605)%20484-4692>
>
> On 5/3/17, 08:06 MDT, "Roy Mendelssohn - NOAA Federal" <
> roy.mendelssohn@xxxxxxxx> wrote:
>
> try installing the fink curl.
>
> -roy
>
>
> On May 3, 2017, at 6:47 AM, Capehart, William J <
> William.Capehart@xxxxxxxxx> wrote:
>
> I am attaching my gist of the build
>
> https://gist.github.com/anonymous/4db828fcd4384c7883a8687147ced1f7
>
> The C tst is 3.make
> The C++4 Test is 7.make
> The Fortran Test is 11.make
> The C++ test is 16.make
>
> Hi Sean: And the output of my nc-config –alll
>
> https://gist.github.com/733ad6330523a83cee204f9f7f34c061
>
>
> The curl version on my mac (10.12.4) is the on-board version which I
> presume is what I need
>
> % curl --version
> curl 7.51.0 (x86_64-apple-darwin16.0) libcurl/7.51.0 SecureTransport
> zlib/1.2.8
> Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3
> pop3s rtsp smb smbs smtp smtps telnet tftp
> Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB
> SSL libz UnixSockets
>
> And curl -Oing my the target file produces the following
>
> https://gist.github.com/wjcapehart/6f49640a075f47b1040c48917ebd631b
>
> Sorry Larry: valgrind is reporting from homebrew that is in compatible
> with MacOS post El Capitan.
>
> Roy: from the client side I’m not seeing any significant redflags in the
> console logs
>
> Really really out of clues at this point. Thanks to everyone or their
> input though.
>
> Bill
>
>
> From: Sean Arms <sarms@xxxxxxxx>
> Date: Tuesday, May 2, 2017 at 17:50 MDT
> To: William Capehart <William.Capehart@xxxxxxxxx>
> Cc: "netcdfgroup@xxxxxxxxxxxxxxxx" <netcdfgroup@xxxxxxxxxxxxxxxx>
> Subject: Re: [netcdfgroup] Using NetCDF with OPeNDAP on MacOS Sierra
> (homebrew- and netCDF-dependant software)
>
> Greetings Bill,
>
> I have a suspicion this has to do with the way curl was built. What does
>
> nc-config --libs
>
> give you?
>
> Sean
>
>
>
> On Sun, Apr 30, 2017 at 10:42 AM, Capehart, William J <
> William.Capehart@xxxxxxxxx> wrote:
> Hi All
>
> I have been having a particularly time working with NetCDF with OPeNDAP on
> my Macs (right now both are on MacOS 10.12.4). I’ve mostly been using
> homebrew but this also happens with NCL 6.3 and 6.4 as downloaded from
> Earth System Grid.
>
> When testing code using just the NetCDF fortran libraries as well as just
> using ncdump I get segmentation faults when I try to access
> http://test.opendap.org/opendap/data/nc/3fnoc.nc
>
> Is anyone else having any luck with their MacOS Sierra NetCDF builds and
> the software that leverages NetCDF?
>
>
> ------------------------------------------------
> Bill Capehart <William.Capehart@xxxxxxxxx>
> Atmospheric and Environmental Sciences Program Coordinator
> Civil and Environmental Engineering
> 201 Mineral Industries Building
> South Dakota School of Mines and Technology
> 501 East St Joseph Street
> Rapid City, SD 57701-3995
> Ph: +1-605-394-1994 <(605)%20394-1994> Mobile: +1-605-484-4692
> <(605)%20484-4692>
>
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit: http://www.unidata.
> ucar.edu/mailing_lists/
>
>
>
>
> **********************
> "The contents of this message do not reflect any position of the U.S.
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> ***Note new street address***
> 110 McAllister Way
> Santa Cruz, CA 95060
> Phone: (831)-420-3666 <(831)%20420-3666>
> Fax: (831) 420-3980
> e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
>
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected"
> "the arc of the moral universe is long, but it bends toward justice" -MLK
> Jr.
>
>
>
> **********************
> "The contents of this message do not reflect any position of the U.S.
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> ***Note new street address***
> 110 McAllister Way
> Santa Cruz, CA 95060
> Phone: (831)-420-3666 <(831)%20420-3666>
> Fax: (831) 420-3980
> e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
>
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected"
> "the arc of the moral universe is long, but it bends toward justice" -MLK
> Jr.
>
>
> **********************
> "The contents of this message do not reflect any position of the U.S.
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> ***Note new street address***
> 110 McAllister Way
> Santa Cruz, CA 95060
> Phone: (831)-420-3666 <(831)%20420-3666>
> Fax: (831) 420-3980
> e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
>
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected"
> "the arc of the moral universe is long, but it bends toward justice"
> -MLK Jr.
>
>
> **********************
> "The contents of this message do not reflect any position of the U.S.
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> ***Note new street address***
> 110 McAllister Way
> Santa Cruz, CA 95060
> Phone: (831)-420-3666 <(831)%20420-3666>
> Fax: (831) 420-3980
> e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected"
> "the arc of the moral universe is long, but it bends toward justice" -MLK
> Jr.
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit: http://www.unidata.
> ucar.edu/mailing_lists/
>
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit: http://www.unidata.
> ucar.edu/mailing_lists/
>
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit: http://www.unidata.
> ucar.edu/mailing_lists/
>
>
>
>
>
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
