Kevin L Manross wrote:
Greetings!
For our CONUS grids, we make use of "sparse grids" in our netCDF files.
From <http://cimms.ou.edu/~lakshman/WDSS2/dataformat.shtml> (about
halfway down the page), our/the netCDF sparse grid format can be
described as:
If your data are sparse, you can save a whole bunch of space by saving
only the non-missing data in a simple run-length encoded format. The
global attributes and the other variables are the same as the regular
formats. However, the dimensions are provided by pixel_x, pixel_y for
the first two dimensions and run_length for the number of values in a
run (default=1 if run_length is omitted). The original dimensions are
also required even if no variables actually use them. The default
background value is MISSING_DATA unless a different value is provided
(see the SparseRadialSet example below).
Example of a sparse lat-lon-grid:
dimensions:
Lat = 650 ;
Lon = 700 ;
pixel = 23541 ;
variables:
float Reflectivity_0C(pixel) ;
Reflectivity_0C:Units = "dBZ" ;
short pixel_x(pixel) ;
short pixel_y(pixel) ;
int pixel_count(pixel) ;
// global attributes:
:TypeName = "Reflectivity_0C" ;
:DataType = "SparseLatLonGrid" ;
:Latitude = 37. ;
:Longitude = -100. ;
:Height = 0. ;
:Time = 990376569 ;
:FractionalTime = 0.584999999999127 ;
:attributes = " ColorMap Unit" ;
:ColorMap-unit = "dimensionless" ;
:ColorMap-value = "Reflectivity" ;
:Unit-unit = "dimensionless" ;
:Unit-value = "dBZ" ;
:LatGridSpacing = 0.01 ;
:LonGridSpacing = 0.01 ;
:MissingData = -99900.f ;
:RangeFolded = -99901.f ;
I haven't dug far enough yet to see much on netCDF sparse grids.
Considering that we were able to reduce a CONUS grid from 71 Mb to 4 Mb,
we would like to continue to make use of sparse grids if we can.
Can anyone offer some guidance whether we can use THREDDS with sparse
grid data? If we can, do I attempt to specify my coordinate
transformations in the header, or is there more manipulation that I need
to do on the data?
Thanks!!
-kevin.
Hi Kevin:
TDS does not support sparse arrays. Have you considered using netcdf-4? I would
expect the compression facilities there would give you similar size reduction,
and it is transparent to all applications, including TDS. You should be able to
relink your application with it, and make some minor modifications to your
nc_create() call, and see what you get.
John
