Hi Dennis,you are right about the problem about dual duty NC_NETCDF4. It was never declared in the way I've been using it for NETCDF4 detection.
The more I think about it, the less I like the NC_NETCDF4 idea. Not only the NC_NETCDF4 solution, but more generally the detectability of Netcdf4 in the header-file. The library might change, but the header should be the same, i.e. one might first add the library with nc3 features, and later exchange that with a library with nc4 features - since the ABI doesn't change it would only require a restart of the final program (dyn. libraries), but no re-compilation.
The main reason I used NC_NETCDF4 for, wasa) to be sure I have all the required functions - this is no longer needed since all functions are always there.
b) to write a runtime-function giving netcdf4 status - this might be added much easier to add the netcdf-library than to external programs. This is not needed in compile-time. (e.g. int nc_can_nc4(); int nc_can_dap();)
I still would like the API version to be part of the netcdf-header, though. Currently, I need to write test-programs (autotools) to detect existence of functions, e.g. nc_can_nc4() ...
Heiko On 2012-05-22 18:49, Dennis Heimbigner wrote:
Heiko- I have been thinking about this issue and am concerned about one point. If we undefine NC_NETCDF4 when netcdf-4 functionality is not in the library, then it makes that flag have dual duty. On one hand it is just a flag like any other, and on the other hand it signals the presence/absence of the netcdf-4 functionality. As far as I can tell, there is no other flag that has that dual function. So, as a purist, I would prefer to leave NC_NETCDF4 always defined, and use some other compile-time #define to determine the presence/absence of netcdf-4 functionality. =Dennis Heimbigner Unidata Heiko Klein wrote:Hi Dennis and Russ, Defining NC_NETCDF4 will break code running nicely in <= 4.1.1, because that code might not check the error-code correctly against NC_ENOTNC4 (which didn't exist then). (ususal code which will break: if (type == 'nc4') { #ifdef NC_NETCDF4 .... #else printf("netcdf4 type not supported by netcdf-library"); #endif } ) So, I still think you shouldn't define NC_NETCDF4 if netcdf4 filetype is not supported since this might break older code. That this is broken in 4.1.2-4.2 is the problem I currently want to see solved. I don't see any advantage in NC_FEATURE_DAP2. Is there anything in the API which might use this feature? Any flags I should set because of DAP2 support? Otherwise, this is only useful for runtime. The NC_FEATURE_NETCDF4 is just another name for the previously used NC_NETCDF4, so yes, it is useful, but why introduce a new name? So, this doesn't help now: netcdf.h currently does not contain a VERSION pragma. The version pragma like #define NC_VERSION 400100200 (e.g. 4.1.2) might help in the future to differ between inconsistencies between APIs. Best regards, Heiko On 2012-05-02 20:55, Dennis Heimbigner wrote:Russ and I have discussed one other alternative. 1. we leave NC_NETCDF4 always defined in netcdf.h 2. we add "feature" flags to netcdf.h to indicate which features are actually supported by the corresponding library. For example, we might have the following flags: NC_FEATURE_NETCDF4 NC_FEATURE_DAP2 etc. =Dennis Heimbigner Doug Hunt wrote:Hi Heiko: I'm fine with your proposal. This will make the netcdf interface more useful (ability to check HDF presence at compile- and run-times) at the cost of a little bit of extra complexity in PDL-NetCDF. Could you add the extra logic to check for netcdf versions between 4.1.2 and 4.2 to the (not released to CPAN) PDL-NetCDF-4.16 that I sent you and test it with HDF? Once you've got that working, send it back and I'll test it in my environment and then release it to CPAN. How does that sound? --Doug dhunt@xxxxxxxx Software Engineer UCAR - COSMIC, Tel. (303) 497-2611 On Wed, 2 May 2012, Heiko Klein wrote:Hi Dennis, Doug, I really like Dennis proposal. It will give both a compile and a run-time ability to check for netcdf4 support. Concernings Dougs comment: Yes, we need a test for the broken versions 4.1.2 to 4.2. Since NC_NETCDF4 is correct before 4.1.2 and will be correct after 4.2, a way to go would be to just add #undef NC_NETCDF4 right after #include "netcdf.h", depending of the result of a test_nc_netcdf4_wrongly_declared. The pre-build test code (e.g. autotools/configure) for that function is then something like: #include "netcdf.h" int main () { #ifdef NC_NETCDF4 nc_inq_var_deflate(0, 0, NULL, NULL, NULL); # else this_fails(); # endif } Doug, I can put that into PDL::NetCDF during that week, if you like. I see you have already a framework for pre-build tests in the current PDL::NetCDF 4.16 Heiko On 2012-05-01 17:49, Doug Hunt wrote:Hi Dennis: After sleeping on this, I find I may have spoken too soon. It may be best (at least from my perspective as maintainer of PDL::NetCDF) to leave things as they are. As logical as your change is, it would still introduce a 3rd behavior regarding netcdf-4 detection. PDL::NetCDF will have to support all 3 behaviors: 1) (version 4.0.1) NC_NETCDF4 defined means netcdf-4 is supported. If 'nc_inq_var_deflate' is present at run-time then netcdf-4 is supported. 2) (version 4.2) NC_NETCDF4 always defined. If 'nc_inq_var_deflate' is present at run-time then netcdf-4 is supported. 3) (your proposed) NC_NETCDF4 defined means netcdf-4 is supported. 'nc_inq_var_deflate' is always defined, but returns NC_ENOTNC4 if netcdf-4 is not supported. Dealing with three of these behaviors will complicate the logic used to detect netcdf-4 support. The new version of PDL::NetCDF that I'm working on right now (version 4.16) will do the run-time check and so catch cases 1) and 2), but would have to be modified to catch case 3). So, since netcdf version 4.2 is already out there and needs to be supported, I would think just leaving things the way they are might be wisest. Heiko--What do you think? Regards, Doug dhunt@xxxxxxxx Software Engineer UCAR - COSMIC, Tel. (303) 497-2611 On Mon, 30 Apr 2012, Doug Hunt wrote:Hi Dennis: This sounds good to me! Regards, Doug Hunt dhunt@xxxxxxxx Software Engineer UCAR - COSMIC, Tel. (303) 497-2611 On Mon, 30 Apr 2012, Dennis Heimbigner wrote:Unidata proposes to change this as follows. 1. All the netcdf-4 functions will exist in the netcdf-3 only library, but will return the error NC_ENOTNC4 if invoked. This allows runtime testing for netcdf-4 support. 2. The NC_NETCDF4 flag in netcdf.h will only be defined it the corresponding library has netcdf-4 support enabled. This is so compile-time testing for netcdf-4 can occur. Comments? =Dennis Heimbigner Unidata Doug Hunt wrote:Hi Heiko: I made a fix to the logic PDL-NetCDF uses to detect if netcdf-4 support is available. In the previous version, a simple check for the pre-processor symbol NC_NETCDF4 being defined is made--if it is, then netcdf-4 support is found. Recent netcdf releases, however, have been defining NC_NETCDF4 whether or not netcdf-4 support is actually present (meaning netcdf is compiled with HDF5). The change I made includes a test compile and link of nc_inq_var_deflate to determine if netcdf-4 support is available. The logic is in Makefile.PL, not netcdf.pd. I've used this procedure before in my PDL-Graphics-PLplot interface and it works well. I'm sending this to you instead of just putting it on CPAN because (1) I can't get netcdf-4 support compiled in on my machine, so I can't test it properly and (2) you added the netcdf-4 support so I'd like you to look it over. Could you take a look and see if this makes sense to you? If you've got a netcdf-4 enabled netcdf on one of your systems, it would be great if you could test it out. Thanks much, Doug dhunt@xxxxxxxx Software Engineer UCAR - COSMIC, Tel. (303) 497-2611 On Fri, 27 Apr 2012, Heiko Klein wrote:Russ, I'll try to give a better example: I don't have hdf-libraries installed (at least not in default pathes). First, from a pure ABI point of view: --------------------------------------------------------------- With netcdf-4.1.2: ./configure --prefix=/disk1/netcdf-4.1.2_install make make install In /disk1/netcdf-4.1.2_install/include: $ grep nc_def_var_deflate netcdf.h * function nc_def_var_deflate. These defines are used there. */ nc_def_var_deflate(int ncid, int varid, int shuffle, int deflate, In /disk1/netcdf-4.1.2_install/lib: $ nm libnetcdf.so | grep nc_def_var_deflate Nothing found, symbol not defined. -------------------------------------------------------------- New try, all directories removed and new untar: ./configure --prefix=/disk1/netcdf-4.1.2_install --disable-netcdf4 make make install In /disk1/netcdf-4.1.2_install/lib: $ nm libnetcdf.so | grep nc_def_var_deflate Nothing found, symbol not defined. Still the same. --------------------------------------------------------------- New try, now 4.2: $ ./configure --prefix=/disk1/netcdf-4.2_install --disable-netcdf-4 make make install In /disk1/netcdf-4.2_install/lib $ nm libnetcdf.so | grep nc_def_var_deflate Nothing found, symbol not defined. --------------------------------------------------------------- A previous build, with hdf5 enabled: $ nm /disk1/heiko/local/netcdf4.2-rc1/lib/libnetcdf.so | grep nc_def_var_deflate 000299b0 T nc_def_var_deflate ---------------------------------------------------------------- Now, from a API point of view: a slightly modified simple_xy_wr.c (from http://www.unidata.ucar.edu/software/netcdf/examples/programs/), just enabling compression if netcdf4 is supported by the library: #ifdef NC_NETCDF4 /* compress all variables in netcdf4 */ if ((retval = nc_def_var_deflate(ncid, varid, 0, 1, 3))) ERR(retval); #endif The program works nicely with netcdf-4.1.1, independently of netcdf4 is enabled or disabled. $ gcc -c -I/opt/netcdf4.1.1/include/ simple_xy_wr.c $ gcc -o simple_xy_wr simple_xy_wr.o -L/opt/netcdf4.1.1/lib/ -lnetcdf No problems With above 4.2: $ gcc -c -I/disk1/netcdf-4.2_install/include/ simple_xy_wr.c No problems. $ gcc -o simple_xy_wr simple_xy_wr.o -L/disk1/netcdf-4.2_install/lib -lnetcdf simple_xy_wr.o: In function `main': simple_xy_wr.c:(.text+0x27d): undefined reference to `nc_def_var_deflate' collect2: ld returned 1 exit status With 4.2, but with enabled hdf/netcdf4 support (ok, that's 4.2rc1, but that not the point here): $ gcc -c -I/disk1/heiko/local/netcdf4.2-rc1/include simple_xy_wr.c No problems $ gcc -o simple_xy_wr simple_xy_wr.o -L/disk1/heiko/local/netcdf4.2-rc1/lib -lnetcdf No problems, generating netcdf4 files. I see the code of nc_def_var_deflate defined in libdispatch/dvar.c, but it doesn't seem to make it to the library. Best regards, Heiko On 2012-04-26 18:51, Russ Rew wrote:Heiko,since netcdf-4.1.2, the netcdf.h file declares all netcdf4 functions, like nc_def_var_deflate and NC_NETCDF4, even if the library was compiled without hdf5 support, and does not have these functions. I have code like #ifdef NC_NETCDF4 nc_def_var_deflate(...) #endif which worked perfectly with netcdf3 and netcdf4 until 4.1.1. Since netcdf 4.1.2, this code compiles still, but when the netcdf-library was not compiled with hdf5 support, the code won't link with the library. This means, that netcdf since 4.1.2 has a single API, but has different ABIs depending on netcdf-configure flags. I would suggest adding the 'Extra netcdf4 stuff.' function calls to the netcdf3 code, even if they are empty, e.g. int nc_def_var_deflate(int ncid, int varid, int shuffle, int deflate, int deflate_level) { return NC_NOERR; } This means, that netcdf since 4.1.2 has a single API, but has different ABIs depending on netcdf-configure flags. I would suggest adding the 'Extra netcdf4 stuff.' function calls to the netcdf3 code, even if they are empty, e.g. int nc_def_var_deflate(int ncid, int varid, int shuffle, int deflate, int deflate_level) { return NC_NOERR; }Sorry, but I can't duplicate the problem. This is supposed to be handled by the "dispatch layer" in netCDF-4.x. For example, when you run "make check" on a netCDF-4.2 distribution that has been configured with "--disable-netcdf-4", it will link and run netCDF-3 tests against the library, so this would indicate all the public functions declared in netcdf.h must be implemented. The following function appears in libdispatch/dvar.c, even when configured with "--disable-netcdf-4": int nc_def_var_deflate(int ncid, int varid, int shuffle, int deflate, int deflate_level) { NC* ncp; int stat = NC_check_id(ncid,&ncp); if(stat != NC_NOERR) return stat; return ncp->dispatch->def_var_deflate(ncid,varid,shuffle,deflate,deflate_level); } I've also just tried building the 4.2 library with --disable-netcdf-4 and --disable-shared, and verified that a netCDF-3 program compiles and links with the resulting static library. However, I had to remove previously built shared libraries from the directory in which I installed the static library, because the dynamic linker/loader will use an older installed shared library in preference to a new static library. I also built a new shared library with --disable-netcdf-4 but without --disable-shared and verified that a netCDF-3 program could be successfully linked and run against the resulting shared library. For this, I set LD_LIBRARY_PATH to run the resulting program, but other alternatives would also work, assuming the library is installed in LIBDIR: - add LIBDIR to the `LD_RUN_PATH' environment variable during linking - use the `-Wl,-rpath -Wl,LIBDIR' linker flag - have system administrator add LIBDIR to `/etc/ld.so.conf' This was all on a Linux system, but our cross-platform tests should catch such problems on some other systems, such as Solaris, MacOS, and AIX. It's possible I've misunderstood the problem. If so, I'll need more information to reproduce it. For now, I think the dispatch layer implementation already provides the stub functions for 'extra netCDF-4 stuff' that you're requesting ... --Russ Note: as Dennis Heimbigner pointed out last week on this list, for now it's necessary to build netCDF-4.2 with "--disable-dap-remote-tests", at least until we resolve a problem with the test server, which was moved from port 8080 to 8081.-- Dr. Heiko Klein Tel. + 47 22 96 32 58 Development Section / IT Department Fax. + 47 22 69 63 55 Norwegian Meteorological Institute http://www.met.no P.O. Box 43 Blindern 0313 Oslo NORWAY------------------------------------------------------------------------ _______________________________________________ netcdfgroup mailing list netcdfgroup@xxxxxxxxxxxxxxxx For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/_______________________________________________ netcdfgroup mailing list netcdfgroup@xxxxxxxxxxxxxxxx For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/-- Dr. Heiko Klein Tel. + 47 22 96 32 58 Development Section / IT Department Fax. + 47 22 69 63 55 Norwegian Meteorological Institute http://www.met.no P.O. Box 43 Blindern 0313 Oslo NORWAY
-- Dr. Heiko Klein Tel. + 47 22 96 32 58 Development Section / IT Department Fax. + 47 22 69 63 55 Norwegian Meteorological Institute http://www.met.no P.O. Box 43 Blindern 0313 Oslo NORWAY
netcdfgroup archives: