NetCDF-4 AR-4 Performance Data With One Time Series
31 December 2009
Another concern that Russ and Dennis and I had was that by taking 5 time
series, we were involving the cache too much. Might the data not be
already pre-loaded after the first time series is retrieved?
So in this run I do one one time series read. This is all on the 3D precip flux data that Gary gave us.
cs[0] cs[1] cs[2] cache(MB) deflate shuffle read_hor(us) read_time_ser(us)
0 0 0 0 0 0 247 5073
256 64 128 4 0 0 238 2162
256 64 128 32 0 0 172 47050
256 64 128 128 0 0 165 42516
256 64 256 4 0 0 89 2061
256 64 256 32 0 0 136 83352
256 64 256 128 0 0 112 80282
256 128 128 4 0 0 217 2119
256 128 128 32 0 0 153 83400
256 128 128 128 0 0 128 80055
256 128 256 4 0 0 78 2250
256 128 256 32 0 0 188 188474
256 128 256 128 0 0 133 177679
1024 64 128 4 0 0 230 1952
1024 64 128 32 0 0 108690 52045
1024 64 128 128 0 0 216 49018
1024 64 256 4 0 0 88 2100
1024 64 256 32 0 0 87 1964
1024 64 256 128 0 0 175 95524
1024 128 128 4 0 0 218 2064
1024 128 128 32 0 0 218 1991
1024 128 128 128 0 0 194 95595
1024 128 256 4 0 0 76 2128
1024 128 256 32 0 0 76 2041
1024 128 256 128 0 0 198 197824
1560 64 128 4 0 0 229 1973
1560 64 128 32 0 0 229 1915
1560 64 128 128 0 0 161078 37119
1560 64 256 4 0 0 87 2178
1560 64 256 32 0 0 87 2105
1560 64 256 128 0 0 160058 72695
1560 128 128 4 0 0 214 2048
1560 128 128 32 0 0 213 1980
1560 128 128 128 0 0 159984 73765
1560 128 256 4 0 0 78 2284
1560 128 256 32 0 0 76 1954
1560 128 256 128 0 0 76 1947
Sorry about the alignment of the columns, but they show up fine for me
in emacs. Obviously this fancy web technology is only a few years away
from doing what emacs could do in 1981...
Posted by $entry.creator.screenName
NetCDF-4 AR-4 Performance Data With Horizonatal and Time Series Reversed
31 December 2009
Russ, Dennis and I discussed some of the chunking results yesterday. We
were concerned that the horizontal reads were causing all the data to be
preloaded into cache for the subsequent time series read. So I swapped
the order - now the time series read is done first.
Here's some results. The first line are the results for reading a classic netCDF file.
cs[0] cs[1] cs[2] cache(MB) deflate shuffle read_hor(us) read_time_ser(us)
0 0 0 0 0 0 247 5908
256 64 128 4 0 0 241 2039
256 64 128 32 0 0 168 31384
256 64 128 128 0 0 140 17096
256 64 256 4 0 0 93 2548
256 64 256 32 0 0 136 55722
256 64 256 128 0 0 106 26892
256 128 128 4 0 0 216 2035
256 128 128 32 0 0 152 55488
256 128 128 128 0 0 121 26698
256 128 256 4 0 0 79 2392
256 128 256 32 0 0 188 191120
256 128 256 128 0 0 136 186396
1024 64 128 4 0 0 236 1945
1024 64 128 32 0 0 108356 53812
1024 64 128 128 0 0 220 19551
1024 64 256 4 0 0 89 1930
1024 64 256 32 0 0 89 1864
1024 64 256 128 0 0 209 40942
1024 128 128 4 0 0 222 2065
1024 128 128 32 0 0 220 1833
1024 128 128 128 0 0 227 41183
1024 128 256 4 0 0 77 1894
1024 128 256 32 0 0 76 1839
1024 128 256 128 0 0 199 207533
1560 64 128 4 0 0 234 1885
1560 64 128 32 0 0 233 1850
1560 64 128 128 0 0 161596 14921
1560 64 256 4 0 0 88 1969
1560 64 256 32 0 0 87 1929
1560 64 256 128 0 0 160939 30848
1560 128 128 4 0 0 218 1924
1560 128 128 32 0 0 218 1875
1560 128 128 128 0 0 161316 30876
1560 128 256 4 0 0 77 1857
1560 128 256 32 0 0 76 1797
1560 128 256 128 0 0 76 1796
Again, there are many chunk size selections which beat the classic netCDF file performance.
Posted by $entry.creator.screenName