NOTE: The netcdf-hdf mailing list is no longer active. The list archives are made available for historical reasons.
Hi Ed,
Interesting... I'm curious about why HDF5 is slower for both tests
below. Can you send me your benchmarks? I'll take a look at them and see
if there are any ways to speed them up...
Quincey
> Howdy all!
>
> Russ suggested to me yesterday that I post these timing results. In
> the tests below I write, and then read, 4 files. The first is in HDF5,
> with no netCDF stuff at all. (Writing BE on a LE system, because
> that's what netcdf does). The second file is pure netcdf-3, with no
> netcdf-4 code involved. The third uses the netcdf-4 library to write a
> file in netcdf classic format. Finally, the last file is created with
> netcdf-4, with HDF5 as a storage layer. This is faster because it is
> writing LE on a LE system (i.e. using HDF "native" format).
>
> The CPU time is the combined total of time spent by the CPU on
> user/library code, and the User time is wall clock time.
>
> Russ would like to include some of these results in an AMS paper, but
> I think we all need to give a thought to what we are measuring first.
>
> About to write pure HDF5 file, one dataset, record by record: x 2000 y 300 z
> 500...
> avg CPU time = 15.17 secs.
> avg User Time = 43 secs.
>
> About to write pure netcdf-3 file, record by record...
> avg CPU time = 17.15 secs.
> avg User Time = 45 secs.
>
> About to write netcdf-3 file thru netcdf-4, record by record...
> avg CPU time = 17.26 secs.
> avg User Time = 45 secs.
>
> About to write netcdf-4 (i.e. HDF5) file, record by record...
> avg CPU time = 15.87 secs.
> avg User Time = 36 secs.
>
> About to read pure HDF5 file...
> avg CPU time = 13.70 secs.
> avg User Time = 34 secs.
>
> About to read pure netcdf-3 file...
> avg CPU time = 9.18 secs.
> avg User Time = 29 secs.
>
> About to read netcdf-3 file, created with netcdf-4...
> avg CPU time = 11.64 secs.
> avg User Time = 20 secs.
>
> About to read netcdf-4 (i.e. HDF5) file...
> avg CPU time = 12.49 secs.
> avg User Time = 20 secs.
>
>From owner-netcdf-hdf@xxxxxxxxxxxxxxxx 29 2003 Dec -0700 11:08:19
Message-ID: <wrxhdzjwl7w.fsf@xxxxxxxxxxxxxxxxxxxxxxx>
Date: 29 Dec 2003 11:08:19 -0700
From: Ed Hartnett <ed@xxxxxxxxxxxxxxxx>
In-Reply-To: <200312291717.hBTHHWuW099245@xxxxxxxxxxxxxxxxxxxxxx>
To: netcdf-hdf@xxxxxxxxxxxxxxxx
Subject: Re: some performance benchmarks for netcdf-4
Received: (from majordo@localhost)
by unidata.ucar.edu (UCAR/Unidata) id hBTI8Kwj007341
for netcdf-hdf-out; Mon, 29 Dec 2003 11:08:20 -0700 (MST)
Received: from rodney.unidata.ucar.edu (rodney.unidata.ucar.edu
[128.117.140.88])
by unidata.ucar.edu (UCAR/Unidata) with ESMTP id hBTI8Jp2007335
for <netcdf-hdf@xxxxxxxxxxxxxxxx>; Mon, 29 Dec 2003 11:08:19 -0700 (MST)
Organization: UCAR/Unidata
Keywords: 200312291808.hBTI8Jp2007335
References: <200312291717.hBTHHWuW099245@xxxxxxxxxxxxxxxxxxxxxx>
Lines: 87
User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.2
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Sender: owner-netcdf-hdf@xxxxxxxxxxxxxxxx
Precedence: bulk
Reply-To: netcdf-hdf@xxxxxxxxxxxxxxxx
Quincey Koziol <koziol@xxxxxxxxxxxxx> writes:
> Hi Ed,
> Interesting... I'm curious about why HDF5 is slower for both tests
> below. Can you send me your benchmarks? I'll take a look at them and see
> if there are any ways to speed them up...
I will be posting my first netcdf-4 release package (hopefully!)
sometime today. Then you can run these on your machine.
HDF5 is slower in tests 1 and 5 because I am using BE on a LE machine
for that. (To try and get a head-to-head comparison with netcdf-3 which
always uses BE.)
For the netcdf-4/HDF5 file I'm using NATIVE types, which, predictably,
yield a faster performance. I'm sure if I switched tests 1 and 5 to
NATIVE formats, we would see a similar gain in performance.
I have to iron out a bunch of release issues (which will take
several weeks, as we've just decided on a major restructuring of the
codebase to combine the netcdf-3 and netcdf-4 cvs repositories). Once
that is complete, I'll take another iteration through this timing
program to add some more tests and learn some more things.
(Any additional timing tests you might like to propose would be
welcome. I'm at least going to add some floating point stuff. The
times below also don't represent the different ways we're going to
allow the user to specify chunking algorithms.)
The goal at this time was just to ensure that netcdf-4 was not such a
dog that it would never work for anyone. At least we know that it is
in the same ballpark as netcdf-3.
Ed
>
> Quincey
>
> > Howdy all!
> >
> > Russ suggested to me yesterday that I post these timing results. In
> > the tests below I write, and then read, 4 files. The first is in HDF5,
> > with no netCDF stuff at all. (Writing BE on a LE system, because
> > that's what netcdf does). The second file is pure netcdf-3, with no
> > netcdf-4 code involved. The third uses the netcdf-4 library to write a
> > file in netcdf classic format. Finally, the last file is created with
> > netcdf-4, with HDF5 as a storage layer. This is faster because it is
> > writing LE on a LE system (i.e. using HDF "native" format).
> >
> > The CPU time is the combined total of time spent by the CPU on
> > user/library code, and the User time is wall clock time.
> >
> > Russ would like to include some of these results in an AMS paper, but
> > I think we all need to give a thought to what we are measuring first.
> >
> > About to write pure HDF5 file, one dataset, record by record: x 2000 y 300
> > z 500...
> > avg CPU time = 15.17 secs.
> > avg User Time = 43 secs.
> >
> > About to write pure netcdf-3 file, record by record...
> > avg CPU time = 17.15 secs.
> > avg User Time = 45 secs.
> >
> > About to write netcdf-3 file thru netcdf-4, record by record...
> > avg CPU time = 17.26 secs.
> > avg User Time = 45 secs.
> >
> > About to write netcdf-4 (i.e. HDF5) file, record by record...
> > avg CPU time = 15.87 secs.
> > avg User Time = 36 secs.
> >
> > About to read pure HDF5 file...
> > avg CPU time = 13.70 secs.
> > avg User Time = 34 secs.
> >
> > About to read pure netcdf-3 file...
> > avg CPU time = 9.18 secs.
> > avg User Time = 29 secs.
> >
> > About to read netcdf-3 file, created with netcdf-4...
> > avg CPU time = 11.64 secs.
> > avg User Time = 20 secs.
> >
> > About to read netcdf-4 (i.e. HDF5) file...
> > avg CPU time = 12.49 secs.
> > avg User Time = 20 secs.
> >
netcdf-hdf archives: