Hi,
Dave Allured wrote:
> ... Consider these options to reduce archival file size:
>
> 1. Keep your current Netcdf format, but store your files gzip'ed. Make
> uncompressing a standard part of opening the file. Many application
> languages will allow you to call the shell to gunzip and delete a
> temporary file, so you can automate this. gunzip is rather fast, as I
> recall. As you stated, your file size is reduced by 99%.
>
> 2. Netcdf 16-bit packed format. Reduce file size by 50%. You get 16
> bits for your combined precision and dynamic range.
>
> 3. Netcdf 8-bit packed format. Reduce file size by 75%. You get 8
> bits for your combined precision and dynamic range.
That's good advice, and here's two additional options:
4. Switch to using HDF5, which supports a form of access called
"chunking", a multidimensional tiling of arrays that permits
accessing data a tile at a time, with each tile compressed
separately. Thus if you only access a small subset of the data in
a large file, only a small part of the data is uncompressed, so
you don't pay the penalty of uncompressing the whole file on open.
With chunking, you can also choose which variables to compress
instead of compressing the entire file.
5. Wait for netCDF-4, which also supports chunking and compression by
delegating it to the HDF5 storage layer. Currently the release of
netCDF-4 is being delayed until the corresponding HDF5 1.8
software on which it depends is ready (estimated December 2006),
but if you can wait, it may be easier to convert your code to use
netCDF-4 than HDF5. You can also test the use of chunking in the
current netCDF-4 alpha release, but note that it uses artifacts
that will not be supported in the real netCDF-4 software.
--Russ
_____________________________________________________________________
Russ Rew UCAR Unidata Program
russ@xxxxxxxxxxxxxxxx http://www.unidata.ucar.edu
==============================================================================
To unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================
