NOTE: The netcdf-hdf mailing list is no longer active. The list archives are made available for historical reasons.
> Having seen the arguments from the HDF side and the netCDF side, I still
> agree, wholeheartedly, with Ed. If we say HDF5 supports a certain kind of
> compression, there should be no if's about it. It has to be complete
> support, not 95% support.
I'm not terribly opposed to requiring zlib in order to build the HDF5
distribution (at least by default), it's a fairly pervasive technology by this
point. The main point I was trying to make was about not shipping zlib with
the HDF5 (or netCDF-4) distribution and building/installing our own version.
Quincey
> Mike
>
> At 12:23 PM 3/16/2006, Albert Cheng wrote:
> >Hi, Ed and everyone,
> >
> >I would like to share some of my experience installing HDF5 on
> >various platforms, some classic like Linux/AIX/Sun and some one
> >of a kind new platforms like IBM Blue Gene, Cray XT3.
> >
> >On the classics platforms, nearly all hosts of them would zlib
> >installed. So, for them, it is a non-issue. So those that don't
> >have zlib installed, some have a policy of no non-supported software.
> >Gnu Zlib is considered non-supported according to them. In those cases,
> >I usually just build HDF5 without zlib since their policy is no non-supported
> >software. (You may ask "Is HDF5 software supported?" For them, yes, because
> >they can contact the HDF group for support.)
> >
> >On some new platforms, things are still in flux. Zlib compression is
> >a luxury for them at the moment. If it works, great. If not, well,
> >they have other higher priorities like file systems or compilers. :-)
> >I would have to build HDF5 without zlib in order for HDF5 users to
> >test their software on these new platforms.
> >
> >Yes, it is true that HDF5 files with data compression produced in some
> >other systems will not be quite usable in the above platforms. (Actually,
> >one can still access parts of the file that are not compressed, just not
> >able to read the data of any compressed datasets.) On the other hand,
> >HDF5 files without compression will be totally useable in the above
> >platforms.
> >
> >Then there is a different dimension of the zlib due to its own success.
> >Since zlib is so popular now, 95% of machines I run into would have a
> >version of zlib installed by the system. If HDF5 brings in its
> >own version, HDF5 users now have a new confusion--which version
> >of zlib is my application linking with--the system version or the HDF5
> >version?
> >
> >There are also applications that already use zlib on their
> >own and has been linking with their system version of zlib. Now they
> >want to use HDF5 library in their application and if HDF5 insists on
> >"inserting" their own version of Zlib, these groups of users would be
> >annoyed. To further complicate the issue, if shared lib versions of
> >zlib is available, the user applications code will use different versions
> >of zlib depending on how their runtime environment such as $LD_LIBRARY_PATH.
> >It is confusing. (Honestly, the shared lib, though a great tool, could be
> >confusing at times.)
> >
> >There are valid pros and cons on each approach. I think the current HDF5
> >setting
> >is a reasonable compromise for the current conditions. Since netCDF-4
> >must run
> >with zlib compression, would it work for netCDF4 to be distributed with a
> >version
> >of zlib code? I know, it is not quite right for netCDF4, being the upper
> >layer,
> >to provide the source code of a library two layers down. But is it a
> >possible
> >solution?
> >
> >-Albert
> >
> >At 09:47 AM 3/16/2006, you wrote:
> >>Quincey Koziol <koziol@xxxxxxxxxxxxx> writes:
> >>
> >>Howdy Quincey!
> >>
> >> >> I agree with Ed's concern about the netCDF library not being able to
> >> read
> >> >> netCDF files that are created using zlib. Same goes for HDF5
> >> >> itself. However it's done, the libraries should have zlib available.
> >> >
> >> > If we want to make zlib a requirement, we should decide that for
> >> > certain.
> >>
> >>By "we" in this, you mean the HDF5 team, right?
> >>
> >>Because for netCDF-4, zlib is a requirement. We have the following
> >>netCDF-4 requirement:
> >>
> >>"Compression can be applied on a per-variable basis."
> >>
> >>I think this is an important feature of netCDF-4. I don't think we
> >>want the user confusion that would go with some installations having
> >>zlib, others not. So I don't want to build netCDF-4 without zlib. It's
> >>a firm requirement for netCDF-4.
> >>
> >>For HDF5, my opinion would be that you are making it harder for the
> >>users. With the current situation, it is possible to write a HDF5 file
> >>on one system, and be unable to read it on another. It is possible to
> >>write a HDF5 program that works great on one platform, but won't
> >>compile on another. Does this not cause a lot of user confusion?
> >>
> >>I suspect most people just install HDF5 without zlib. But per-variable
> >>compression is a *GREAT* feature, and one that I think most
> >>researchers would very much like to use (my opinion).
> >>
> >> > Then, if it's a requirement, we should make the absolute best attempt
> >> we can
> >> > to use the system's installed zlib - that's what it's there for. Only
> >> if there
> >> > is no zlib installed on the system should we attempt to help the user
> >> install
> >> > a copy of zlib from the _zlib_ web-site. We should _not_ be in the
> >> "business"
> >> > of distributing zlib.
> >>
> >>Here I must disagree. Your current method makes it considerably harder
> >>for the user to ensure that zlib is installed, and the HDF5 is built
> >>with the --with-zlib option. In fact, I would speculate that the vast
> >>majority of users will screw this part up. (I often do myself!)
> >>
> >>It is usual in autotools to provide what cannot be found on the
> >>system. For example, if I want to use the "strtok" function, the
> >>autotools way would be to look for it on the host system, and, if not
> >>found, to provide one, *not* to have my code work one way if it's
> >>there, and another way if it's not.
> >>
> >>Similarly, it would make sense for HDF5 to look for zlib, and, if not
> >>found, to provide it.
> >>
> >>The amount of extra code in your distribution is trivial. The amount
> >>of extra work to get it working this way is zero for you, since I am
> >>going to do it anyway, and you can just take my solution if you want
> >>it.
> >>
> >>In general, this also guards against versioning problems. What if the
> >>user has a version of zlib other than 1.2.3 installed? Surely you
> >>don't test against every possible version of zlib.
> >>
> >>I understand your concerns about taking on some responsibility for
> >>zlib. But that is already the case - it is already an important
> >>component of our products. Making it harder to install does not really
> >>help this problem. If there is a bug is zlib, we all will be hurting
> >>- no matter how it was installed.
> >>
> >>In fact, by taking a known, tested version of zlib inside our
> >>products, we guard against zlib risks. If version 1.2.4 of zlib comes
> >>out, and it sucks, we will be protected. All our users will be using
> >>1.2.3, which they got with their tarball.
> >>
> >> >> At first blush, Ed's solution seems to me like a good one.
> >> >
> >> > It's _very much_ against all the current open source "package"
> >> installation
> >> > systems' philosophy and I don't think it's a good idea. All of package
> >> > installation systems allow each package in the system to list other
> >> packages as
> >> > prerequisites that must be installed on the system before the package the
> >> > user is interested in is installed. We just _finished_ removing zlib and
> >> > libjpeg from the HDF4 distribution... :-)
> >>
> >>In the world of package distribution, if zlib is required for HDF5,
> >>that means that HDF5 will not build without zlib.
> >>
> >>The problem is that HDF5 *will* build without zlib, yielding a HDF5
> >>build which is not usable for netCDF-4. To fit the current package
> >>model, you need to make the HDF5 configure script dumb enough to just
> >>give up when zlib is not present. There there would be only one way to
> >>build HDF5 - with zlib.
> >>
> >>Also, package distribution, while wonderful, is not the usual way for
> >>people to get either HDF5 or netCDF - they get these as tarballs from
> >>our site. So we cannot rely on every user having a package manager to
> >>hide this complexity.
> >>
> >>I would be interested to hear why you removed zlib from HDF4. Was
> >>there a problem?
> >>
> >> >
> >> > Shipping someone else's library with our package and installing it
> >> > on a user's system (most likely _in place of_ the system version of that
> >> > library) has caused _lots_ of headaches for us in the past.
> >>
> >>Installation is different from building.
> >>
> >>I would not suggest that zlib be installed from the HDF5 build. You
> >>can build it and use it internal to the HDF5 package, use it within
> >>HDF5 but not expose it to anyone else.
> >>
> >>Then you have not installed anything on the user's system other than
> >>HDF5.
> >>
> >> > Frankly, I think this is the course that netCDF-4 should take
> >> > with HDF5 also, but that'll ultimately be up to them.
> >>
> >>I would love to. Instead of programming autoconf and automake I would
> >>be out on a bike ride!
> >>
> >>But that's just going to kill uptake of netCDF-4. Our users are Earth
> >>scientists, not computer scientists.
> >>
> >>I intend that users will be able to install netCDF-4 in one step, just
> >>like netCDF-3. It makes extra work for me up front, but also saves me
> >>hundreds or thousands of support emails later. (Recall - we don't have
> >>a large support team here at netCDF - just me and Russ!)
> >>
> >>In this new build method, users will not get HDF5 from you and
> >>netCDF-4 from me, they will get the whole tarball from me, and build
> >>it all at once.
> >>
> >>(There will also be provision for users who want to
> >>use an already-installed version of HDF5 - but this will be a small
> >>minority of users. However, for them, everything can work just as it
> >>does now.)
> >>
> >>It was also not my intention to install HDF5 on their machines with
> >>netCDF-4, but just to use it internally, and end up installing
> >>netCDF-4 only.
> >>
> >>(This also would mean that the user would not have to provide the link
> >>arguments "-lhdf5 -lhdf5_hl -lz -lnetcdf", just "-lnetcdf". It will
> >>make the netcdf library files larger, but so what?)
> >>
> >>If the HDF5 build can take on the task of building zlib when not
> >>present that would make things a lot easier for me. I can give you the
> >>changes that you would need for that to happen.
> >>
> >>If not, then I will possibly have to hack every HDF5 release before
> >>including it in the master tarball with netCDF-4. This will add some
> >>work for me every time you do a release, and some delay in adopting
> >>new HDF5 versions in netCDF-4.
> >>
> >>My goal is to make life easier for the netCDF user. Current netCDF4
> >>installation is far too complicated - it *must* be
> >>simplified. Otherwise we will lose half our users or more before they
> >>even get to try it.
> >>
> >>Thanks,
> >>
> >>Ed
> >>
> >>
> >>--
> >>Ed Hartnett -- ed@xxxxxxxxxxxxxxxx
> >
> >--
> >Mike Folk, Scientific Data Tech (HDF) http://hdf.ncsa.uiuc.edu
> >NCSA/U of Illinois at Urbana-Champaign Voice: 217-244-0647
> >1205 W. Clark St, Urbana IL 61801 Fax: 217-244-5521
>
netcdf-hdf archives: