Re: performance degrades with filesize

Konrad,

> I think I found what slows it down. In my Python interface, every
> write operation to an array with an unlimited dimension is followed by
> a call to nc_inq_dimlen() in order to keep the internal dimension
> count up to date. I had assumed that this call would be cheap and
> independent of the file size, since it ought to be no more than an
> access to some data structure that the netCDF library should keep in
> memory.

The implementation of nc_inq_dimlen() is a bit more complicated, since
changes were added to support multiprocessing on Cray T3E systems
(September 1999).  However, it still should take constant time,
independent of the file size.  So I was puzzled by your findings and
tried to reproduce them.

Since the Python, C++, and Fortran netCDF interfaces all use the C
interface to do the I/O, I tried to duplicate the reported performance
problem in the C interface.  Translating the Python example to C and
running it still shows no apparent performance problem, even when I
add a call to nc_inq_dimlen() after each write operation.

Specifically, the following C program:

  http://www.unidata.ucar.edu/packages/netcdf/jg1.c

accepts a single command-line argument for how many records to write,
creates a netCDF file with the same structure as John Galbraith's
Python script, and prints the time required to append each additional
batch of 5000 records to the initially empty file.

The times are nearly constant even when the file has grown to 1.2
Gbytes:

  $ ./jg1 300000
  record 5000:      1.010 secs
  record 10000:      0.950 secs
  record 15000:      0.940 secs
  record 20000:      1.020 secs
  record 25000:      0.930 secs
  record 30000:      1.030 secs
  record 35000:      1.080 secs
  record 40000:      1.060 secs
  record 45000:      1.040 secs
  record 50000:      0.940 secs
   ...
  record 240000:      1.160 secs
  record 245000:      1.150 secs
  record 250000:      1.130 secs
  record 255000:      1.120 secs
  record 260000:      1.180 secs
  record 265000:      1.160 secs
  record 270000:      1.140 secs
  line 107 of jg1.c: No space left on device

If those reporting the problem could please compile and run this C
program on their systems and report the circumstances under which they
observe performance degrading with file size, or modify the program to
demonstrate the problem, that would assist us in determining the cause
of the problem and fixing it.

--Russ

P.S.  I'm trying to continue doing my job in the face of the terrible
tragedies that occurred earlier today.  It has been very difficult to
focus on work, but not working feels like giving in to those who would
demand that we instead focus on their acts of terror.

  • 2001 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: