Re: [netcdfgroup] what is status of netCDF3 vs netCDF4?

On 11/07/2014 07:13 AM, Ed Hartnett wrote:
Howdy all!

Before Ward's time, I was the one who split out the Fortran libraries in
netCDF.

Although this represents some extra inconvenience for Fortran
programmers who want to build from source, it has a couple of benefits.

1 - Users who only want the C library don't get confused with the
Fortran build. This was a significant problem when the releases were
combined. Although the netCDF Fortran community is strong, there is an
even larger set of tools and APIs that only need the C library. The
Fortran build generated many more support questions than the C build. By
providing a C-only library distribution we cut support questions pretty
dramatically.


So, you orphaned us poor Fortran dinosaur programmers ...
... too bad because besides the (nice, high quality)
tools that are based in NetCDF-C, virtually all ocean/atmosphere/climate models (and probably more software out there) is written in Fortran (of various generations).
How could the more expert programmers of those tools get more confused
than we dinosaur Fortranists did after Fortran was dismembered from
NetCDF?  :)

2 - Splitting the libraries by language fits a lot better with the
GNU/Linux worldview, and makes them easier and more attractive for
dedicated volunteers to distribute via the Linux package management
tools.

The argument isn't so strong, because the Fortran libraries are
hierarchically dependent on the C ones (and in a dependence chain).
Besides, the releases became totally independent, and who knows which
Fortran library versions are consistent with such and such C library
versions?
The build became much more cumbersome.

On my stock Fedora machine I can type "yum install
netcdf-fortran" and I will get the fortran build, and all the libraries
it depends on.

Not everybody wears Fedora.
RHEL/CentOS/SL, which are the stable distributions used
in many servers and clusters are ages late w.r.t. NetCDF packages
(4.1.1 I think, from Epel).
Not sure about the situation with SLES, Debian/Ubuntu, and friends.

But more important, packages are not always the best fit,
actually the worst in many cases.
Many programs depend on having libraries built with commercial
compilers (eg. Intel, PGI, etc), won't link programs otherwise.
So, one must have the ability to build from source with the compilers
of choice, to have this flexibility.

I (and many others) go through the trouble of building zlib, HDF5, the
whole chain of NetCDF dependencies in order to provide
a consistent and rich
environment for compilation and linking.
And on top of this build also some of the tools in the NetCDF constellation (NCO, CDO, etc).
That abiltiy is becoming hindered by the disconnected versions of
the various NetCDF library interfaces distributed to the public.

That can be easily remedied without hindering the developers preferences, by bundling them all into public releases with an
easy to use "configure/make/make-install" build system.
Please, don't throw the baby with the bath water.

Have you tried to compile NCAR, GFDL,
MaxPlanckInstitute (ECHAM and friends),
and other models with gfortran, and link them with NetCDF from packages?
Please, give the a try ...
F90 modules are not compatible across compilers, and if one uses
NetCDF packages there is no game when the programs require commercial
compilers.


This is by far the easiest way to install Netcdf, and it
does not generate support questions.

Easy, sure.
Effective, it is not, at least not for many applications.

NetCDF libraries are ported to most
of the mainstream Linux distros I've encountered, and even on Windows
with Cygwin. (But to use the Microsoft development tools you must use
Ward's CMake-based windows build.)

Best advice is to build from source only if you need the very latest
release, and it is not yet offered by your package management system.

Not necessarily for the latest release.
We can live with versions that are a tad old, but we need
flexibility to build from source because not all programs will work otherwise.

Otherwise use the package management system and you will get all
dependencies and everything properly installed in the right place,
without having to read documentation or install multiple packages by
hand. I note that the Fedora distro helpfully includes versions of
netCDF built for parallel I/O as well. With all these helpful
distributions, the average netCDF user should not have to build from source.

Thanks,
Ed


Thanks,
Gus Correa

On Thu, Nov 6, 2014 at 8:55 PM, Gus Correa <gus@xxxxxxxxxxxxxxxxx
<mailto:gus@xxxxxxxxxxxxxxxxx>> wrote:

    Thank you, Ward.

    On 11/06/2014 06:06 PM, Ward Fisher wrote:

        Thanks all for the input re: bundling the different interfaces; it’s
        clear this is more convenient. I would argue that the benefit is not
        strictly to us developers at the expense of the poor users (as
        somebody
        put it :) ); the split makes it much easier to provide support for
        individual interfaces, as well as faster bug fixes as previously
        mentioned.

        There are significant technical hurdles to recombining the
        interfaces
        into a single project, as it was for versions 4.1.3 and prior.


    However, the various interfaces have a hierarchical dependence,
    all of them - except for Java - depending on the C interface.
    This gives them some sort of unity, at least when one has to *use* them.

    Nobody can get the F90 interface to work without the Fortran (77) one,
    and the latter doesn't work without the C interface either.
    Hence, from the user/programmer standpoint, I think breaking this
    unit into smaller pieces is artificial and not helpful.

    Additional difficulties happen also when one has to build more tools
    in the NetCDF universe (NCO, NCL, CDO, etc) which depend on NetCDF
    (and sometimes on specific library versions).
    Those difficulties may also be reduced by having a clear and unified
    public distribution.

    However, I guess there may be a way to reconcile the two,
    keeping the developement in separate projects,
    and yet provide a bundled distribution for
    users/programmers as it was before.

        There may
        be avenues for making distribution more transparent and easier
        to keep
        track of from the end user point-of-view, however. I may start a new
        thread once I’ve explored a couple of ideas.


    That would really be great for NetCDF users/programmers,
    and hopefully won't be a hurdle for the developers.

    As others list subscribers certainly did, I built NetCDF C 4.3.0,
    along with Fortran 4.2 and C++4.4.2,
    which are available from Unidata as separate tarballs.
    I keep them under a single environment module in our clusters (actually
    one per compiler), which is very convenient from the user point of view,
    and seems to be all we need to compile and run correctly a
    large number of programs (from NCAR, GFDL, MPII, MIT,
    some inhouse codes, etc).

    But what took some time for me to setup, may not be so
    hard for the developers.
    So, maybe Unidata could bundle the three main interfaces
    into a single public/external/vanilla distribution,
    perhaps under a single release name
    (NetCDF.all.X.Y.Z perhaps?), which would build from a single
    configure/make/make_install action, as it was done until 4.3.1,
    and likewise for subsequent public releases.

    The existence of a single bundled release also reassures the end user
    that he is dealing with a consistent software package.
    (S)he doesn't need to go crazy to find out what set of specific
    interface releases are consistent with each other, or to spend too
    much time building/testing each one separately.

    Anyway, I'm glad that somehow my hijacking of this thread
    was not just an annoyance to the list,
    but had also a productive side.

    Thank you
    Gus Correa

        Regarding the question below about binary distributions for OSX and
        Windows. We provide Windows binaries because, frankly, building with
        Visual Studio can be a bit of a mess, and providing the libraries
        packaged with dependencies seemed like the easiest way to head
        off a lot
        of problems. These can be downloaded here:

           *
        http://www.unidata.ucar.edu/__software/netcdf/docs/winbin.__html
        <http://www.unidata.ucar.edu/software/netcdf/docs/winbin.html>

        I wasn’t under the impression that there was much need for OSX
        binary
        distributions, since OSX is essentially BSD and works with autotools
        and/or CMake. I know that the popular package managers
        |homebrew| and
        |macports| have netcdf packages (which we do not maintain), and had
        always thought these must be sufficient, as nobody has said
        otherwise.
        I’d be really interested to know if these were insufficient!

        Thanks all,

        -Ward

        On Thu, Nov 6, 2014 at 3:34 PM, Chris Barker
        <chris.barker@xxxxxxxx <mailto:chris.barker@xxxxxxxx>
        <mailto:chris.barker@xxxxxxxx <mailto:chris.barker@xxxxxxxx>>__>
        wrote:

             On Thu, Nov 6, 2014 at 12:57 PM, Gus Correa
        <gus@xxxxxxxxxxxxxxxxx <mailto:gus@xxxxxxxxxxxxxxxxx>
             <mailto:gus@xxxxxxxxxxxxxxxxx
        <mailto:gus@xxxxxxxxxxxxxxxxx>>__> wrote:

                 PS - Yes, I do understand the initial thread was about
                 NetCDF3 vs NetCDF4, which is a separate discussion in
        itself.


             Bringing it back ---

             One of the main reasons users don't use netcdf4 is that it's
             substantially harder to build (and more heavy weight). As
        (at least
             for the c libs) the netcdf4 libs fully support netcdf3,
        there really
             isn't any reason for all client code to use the netcdf4
        libraries,
             regardless of whether they are actually using netcdf4 files.

             So anything Unidata can do to make it easier for end users
        to use
             netcdf4 will really help:

             Easier to build

             Good binary distributions (for Window and probably OS-X anyway)


             ???

             -Chris


             --

             Christopher Barker, Ph.D.
             Oceanographer

             Emergency Response Division
             NOAA/NOS/OR&R (206) 526-6959 <tel:%28206%29%20526-6959>
        <tel:%28206%29%20526-6959>   voice
             7600 Sand Point Way NE (206) 526-6329
        <tel:%28206%29%20526-6329> <tel:%28206%29%20526-6329>   fax
             Seattle, WA  98115 (206) 526-6317
        <tel:%28206%29%20526-6317> <tel:%28206%29%20526-6317>   main
             reception

        Chris.Barker@xxxxxxxx <mailto:Chris.Barker@xxxxxxxx>
        <mailto:Chris.Barker@xxxxxxxx <mailto:Chris.Barker@xxxxxxxx>>


    _________________________________________________
    netcdfgroup mailing list
    netcdfgroup@xxxxxxxxxxxxxxxx <mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
    For list information or to unsubscribe,  visit:
    http://www.unidata.ucar.edu/__mailing_lists/
    <http://www.unidata.ucar.edu/mailing_lists/>





  • 2014 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: