Hi John:
On Apr 25, 2011, at 12:42 PM, John Caron wrote:
> On 4/25/2011 1:37 PM, Roy Mendelssohn wrote:
>> yes, internal compression. All the files were made from netcdf3 files using
>> NCO with the options:
>>
>> ncks -4 -L 1
>>
>> The results so far show a decrease in file size from 40% of original to
>> 1/100 th of the original file size. If the internally compressed data
>> requests are cached differently than request to netcdf3 files, we want to
>> take that into account when we do the tests, so that we do not just see the
>> affect of differential cacheing.
>>
>> When we have done tests on just local files, the reads where about 8 times
>> slower from a compressed file. But Rich Signell has found that the
>> combination of serialization/bandwidth is the bottleneck, and you hardly
>> notice the difference in a remote access situation. That is what we want to
>> find out, because we run on very little money and with compression as
>> mentioned above our RAIDS would go a lot farther, as long the hit to the
>> access time is not too great.
>>
>> Thanks,
>>
>> -Roy
>
> in netcdf4/hdf5, compression is tied to the chunking. Each chunk is
> individually compressed, and must be completely decompressed to retrieve even
> one value from that chunk. So the trick is to make your chunks correspond to
> your "common cases" of data access. If thats possible, you should find that
> compressed access is faster than non-compressed access, because IO is
> smaller. but it will be highly dependent on that.
Hi John:
But do you cache the entire chunk that you decompress, or do you toss it? So
if I make a second request that has data in that chunk i it saved or is it
reread from the file.
-Thanks,
-Roy
**********************
"The contents of this message do not reflect any position of the U.S.
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097
e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/
"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
