I do this kind of thing all the time by generating a netcdf file using
a public domain tool called yorick. The resulting files are truely half
the size of the same file written with floats. Yorick does not use the
unidata software to read and write the files. It manipulates the
netcdf directly.
For testing you might be interested in a a small 100 line program that
converts a generic netcdf file containing floats or doubles (mixed
with anything else) to one containing shorts replacing those
fields. You are welcome to if you want it. Let me know. I could also
try converting your files, or giving you one of mine, if you want a
look at the output.
I am in Japan this week, so there will be a 1-day lag if you try to
contact me.
Phil
On Fri, Apr 06, 2001 at 04:06:20PM -0700, Alexey Goldin wrote:
>
> Sorry if this message is repeated --- I had trouble with majordomo.
>
> I have question about packing 1-d NC_SHORT arrays with unlimited
> dimension.
>
> We have gigabytes of telescope data, stored as 2 byte integer time
> traces. I am trying to move our data acquisition and archival system
> from homegrown format to NetCDF.
>
> When playing with NetCDF, I found that files usually take twice as
> much space as I would expect. A close examinationg with od -x and
> lessdemonstrates that half of the space is not used.
>
> I realize that every record should be aligned at 4-byte boundary, but it
> looks like every member of record structure is aligned at 4-byte
> boundary as well.
>
> Here is the a small file demostrating the problem:
>
> netcdf t2 {
> dimensions:
> time = UNLIMITED ; // (100 currently)
> variables:
> short array1(time) ;
> short array2(time) ;
> data:
>
> array1 = 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
> 1, 1, 1, 1, 1, 1 ;
>
> array2 = 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
> 2, 2,
> 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
> 2, 2,
> 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
> 2, 2,
> 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
> 2, 2,
> 2, 2, 2, 2, 2, 2 ;
> }
>
>
> and here is hex dump of this 924 byte (I'd expect it to be 524 byte)
> file:
>
> 0000000 4443 0146 0000 6400 0000 0a00 0000 0100
> 0000020 0000 0400 6974 656d 0000 0000 0000 0000
> 0000040 0000 0000 0000 0b00 0000 0200 0000 0600
> 0000060 7261 6172 3179 0000 0000 0100 0000 0000
> 0000100 0000 0000 0000 0000 0000 0300 0000 0400
> 0000120 0000 7c00 0000 0600 7261 6172 3279 0000
> 0000140 0000 0100 0000 0000 0000 0000 0000 0000
> 0000160 0000 0300 0000 0400 0000 8000 0100 0180
> 0000200 0200 0180 0100 0180 0200 0180 0100 0180
> 0000240 0200 0180 0100 0180 0200 0180 0100 0180
> 0000280 0200 0180 0100 0180 0200 0180 0100 0180
> *
> 0001620 0200 0180 0100 0180 0200 0180
> 0001634
> ^^^^ ^^^^ ^^^^ ^^^^
>
>
> As you see, half of the space is filled by 0x0180 --- -32767,
> standartNC_SHORT fill value.
>
>
> Is it possible to do something about it as wasting half of disk space
> is not really an option?
>
> Software: netcdf-3.5b3 on Intel Redhat-6.2
>
> Thanks a lot for your attention!
--
Phil Rasch, Climate Modeling Section, National Center for Atmospheric Research
Mail --> P.O. Box 3000, Boulder CO 80307
Shipping --> 1850 Table Mesa Dr, Boulder, CO 80305
email: pjr@xxxxxxxx, Web: http://www.cgd.ucar.edu/cms/pjr Phone: 303-497-1368,
FAX: 303-497-1324
