I think that lack of compression is the major failing of the netcdf format. I
have
been looking at the "WMO sanctioned" grib format lately and I am appreciating
what
a major advance netcdf represents over those older forms.
I think that compression deserves more thought. It may be that by adding a few
reasonable restrictions, it could be supported in a straightforward manner. If
the major
problem is to support random write capability, then restrict that capability.
It is
much more common to write once and read many times. Typically if you write
again to a file, it is a time series, and you are appending. In any case it
would "be alright" to
have a seperate call or even a seperate program that would compress after the
file
was completely written.
Obviously using a unix compress gets you the same thing. However, you want
netcdf to
manage the uncompress "on the fly" so that you dont have to worry about it.
Here, I dont
see where the difficulty would be, unless you are worried about efficiency,
e.g. having to
decompress the entire file to get a a single cross section. I will say that if
your
compression is good enough, you can easily save more in I/O time than you spend
in
decompression.
I suppose you could do it "outside" netcdf, and implement a daemon that grabs
the ncopen
call, check if the file exists, if not, checks if file.Z exists, and
uncompresses.
Give the daemon a fixed amount of disk space so that when it has used all of
its space,
and needs to uncompress another file, it deletes the oldest files in its
subdirectory, etc.
