Hi all,
To add more weight to this argument if more is needed, we are working
more and more with high resolution 3D chemistry/climate models in the
UK with resolutions of 1 degree or better. I could quote similar
numbers to the ones Keith gives. Storage capacity used to be a
limiting factor for us but this is starting to disappear. I have
resorted to integer packing some fields and splitting model output
across multiple netcdf files to overcome netcdf size limitations.
I would be loathe to abandon netCDF. It has been extremely useful to
use in terms of handling model data across different platforms and
using analysis software. Even our data centre seems to be (at last
8-) recognizing the value of netCDF.
Glenn
At 9:45 am -0600 2/5/07, Keith Lindsay wrote:
Hi,
I'll jump in too.
I work with global ocean modeling of biogeochemical tracers. In the
past, because of computational cost, we have worked with low resolution
models, which translated to horizontal dimensions of 100x116 and 25
vertical levels. We are now moving to higher resolutions, with our
current target have a horizontal dimension of 320x384 with 60 vertical
levels. Each 3D single precision field at this resolution uses ~28Mb, so
72 such fields fit into 2Gb and 145 fields fit into 4Gb. Or output
module is designed to put all output fields into a single file that
includes a record dimension, which is the time axis. Having the time
axis be a record dimension eases later manipulation of the files with
the NCO suite of tools. In classic netCDF format, we can only fit 72 3D
fields into a record and this only doubles when moving to CDF2. Our
biogeochemical module has 24 tracers and there are typically numerous
diagnostic fields describing fluxes between each tracer field. When I
started doing integrations with the higher resolution, I had to remove
some diagnostic fields in order to fit within the 2Gb record limit. I
resisted moving to CDF2 because of concern with typical tools being
compatible with the new format.
I am very concerned that switched to CDF2 only gains us a factor of 2.
Our grid has an approximate resolution of 1 degree. Colleagues of mine
regularly use 0.4 degree grids and 0.1 degree grids, and are forced to
use binary formatted output. Given our current output architecture of
having all fields in a single file, which our analysis tools are
constructed for, CDF2 just won't get us much beyond where we are. If
NetCDF doesn't transition allow larger records, in the near future we
will either have to redesign our output and analysis tools, which is
time consuming, or we won't be able to use NetCDF. I'm not looking
forward to either option.
Keith
--
******************************************************************
Keith Lindsay http://www.cgd.ucar.edu/oce/klindsay/
email: klindsay@xxxxxxxx phone: 303-497-1722 fax: 303-497-1700
==============================================================================
To unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================
--
Dr. Glenn Carver, NCAS-Climate, ACMSU, Centre for Atmospheric Science,
Univ. of Cambridge, Chemistry Dept., Lensfield Road, Cambridge, CB2 1EW, UK.
mailto:Glenn.Carver@xxxxxxxxxxxxxxxx http://www.atm.ch.cam.ac.uk/~glenn/
Phone: +44 (1223) 763827
Fax: +44 (1223) 763823
==============================================================================
To unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================
