Peter Halvorson <pjh@xxxxxxxxx> wrote:
> The list has been kind of quiet, is it time to start the time debate again?
> Just kidding.
For those not familiar with the classic "time debate" that erupts here every
year or two, or in case you're nostalgic and are thinking of starting
another time discussion, please first see the collected messages from the
previous time discussions, available (for a limited time only!) as a
hypermail archive at
http://www.unidata.ucar.edu/packages/netcdf/time/
I've also added a new Frequently Asked Question about handling time to the
netCDF FAQ list.
...
> As my program iterates along, it will write all the 0d, 1d, and 2d data
> at each step, but only writes the large 3 dimensional arrays at selected
> steps (to conserve disk space). There seems to be a built in constraint
> at the core of the library that the amount of data for each step in the
> unlimited dimension will be the same (which means low overhead and fast
> calculation of the location of data). The best you can do is to not
> write or initialize a variable for a particular step, which leaves a
> hole in the file. Almost any sort of file copy will then fill out this
> hole with nulls. This could make the difference between a 5 MByte
> output file and a 100 MByte output file (and we do a lot of runs).
In addition to Steven Walker's suggestion to use two output files to get two
unlimited dimensions, there is another possibility: use an artificial
"subrecord" dimension for the more frequent time steps and the "record"
dimension for the occasional writes of the large 3D arrays. For example, if
you write all the 0d, 1d, and 2d data 20 times as often as you write the 3d
data, add an extra dimension, "subrecord" of length 20, to all the variables
except the 3d variables, and increment that when you are writing each time
step. Increment the record dimension and reset the subrecord dimension to 0
when you write the 3d data. That way the most space you waste is a small
multiple of the space used by the smaller arrays, which presumably is much
less than the space taken by the 3d arrays.
...
> Am I missing something? Is there some way to do what I want with the
> current netCDF? Is the rumored multi-unlimited dimension version of
> netCDF close to ready? beta? something being considered for next year?
There is no way to do what you want without wasting some space with the
current version of netCDF. Although we are currently working on an
implementation of netCDF version 3, it does not have multiple unlimited
dimensions. We have discussed a way to implement that capability in a
future version, but providing packed data types is currently a higher
priority.
Let me take this opportunity to plug our "What's New with NetCDF?" web page
at
http://www.unidata.ucar.edu/packages/netcdf/whatsnew.html
which we maintain as an archive of announcements about netCDF developments.
I was remiss in forwarding the last minor release announcement to the
netcdfgroup mailing list, but you'll find it on the What's New page. We'll
be announcing the next minor release (2.4.2) as soon as we've had time to
test it thoroughly.
--Russ
______________________________________________________________________________
Russ Rew UCAR Unidata Program
russ@xxxxxxxxxxxxxxxx http://www.unidata.ucar.edu
To: netcdfgroup@xxxxxxxxxxxxxxxx
Subject: Re: limits of netCDF library
Cc: bert@xxxxxxxxxxxx
Sender: owner-netcdfgroup@xxxxxxxxxxxxxxxx
Precedence: bulk
Reply-To: netcdfgroup@xxxxxxxxxxxxxxxx
Russ,
It would be helpful to fans of netCDF in other branches of physics if netCDF
were to include complex*4 and complex*8 as a data type. Complex variables
have been very useful for a couple of centuries in applied math..
altho complex analysis may not have worked out well for the Unibomber suspect,
and the weather people may wish their numbers were real..
;-) [I wish MINE were *real* too]
Bruce
