NOTE: The netcdf-hdf mailing list is no longer active. The list archives are made available for historical reasons.
Hi John, You wrote: > Greetings, > > I am sending this e-mail to the group of people who I think might > be able to give me some insight into the current status of > parallel netcdf (or parallel hdf/netcdf). I currently have a couple > of months to investigate implementing some sort of parallel netcdf > capability in our code. Note that this is not meant to be a research > endeavor, but to actually introduce some new technology / > improved capabilities. > > A couple of notes on the code itself. It is a parallel global > Atmospheric Chemical Transport Model (ACTM) that uses MPI to > communicate between the processors. It is written in F77/F90 and > uses netCDF for much of its I/O. It is a mature code, highly portable, > and has been run on a variety of MP platforms over the years. Of > course like most models, it is continues to be "enhanced" regularly. > We do parallel reads of large amounts of netCDF data, but currently all > significant netCDF output is funneled through a single Master processor. > This output can be many gigabytes as well. > > After looking into this for a short time and trying to figure out if > there was a particular direction I should go in, the following choices / > issues seemed to emerge: > > * Wait a couple of years for the newly funded proposal to merge netCDF > with HDF to be completed? I wish I could, but we really would like > something now, and I also have much of my time funded to do this > between now and 9/30/03. As the PI of this effort, I agree that you should try one of the other parallel netCDF implementations, with your time constraints. From what I've seen, the Northwestern Univ./ANL effort <http://www-unix.mcs.anl.gov/parallel-netcdf/> has achieved some impressive performance results and is closest to ready to try at this point, although I haven't kept up with recent progress of the NERSC/LBL effort. > * Work with NERSC/LBL and try to use the parallel netCDF capability > that they have developed. Not sure if this just runs on a single > platform, if it has a Fortran interface, what state it is in, etc.? > > * Ditto for Northwestern Univ./ANL. It was actually a tutorial I took > at SuperComputing from Rob Ross, et.al. that got me interested in this. > > * Somehow use HDF to provide the parallel netCDF capability. Convert > the code to use HDF (probably a significant amount of work) and convert > the netCDF input files to HDF? Or maybe HDF can read netCDF files? Would > probably want to convert the HDF output files as well, but not sure. If > this sort of format conversion is possible, how much time would it take? > We have lots and lots of netCDF input files. > > * Other possibilities? > > Any guidance any of you can give me would be much appreciated. > > Regards, > John > > -- > =========================== > John R. Tannahill > Lawrence Livermore Nat. Lab. > P.O. Box 808, M/S L-103 > Livermore, CA 94551 > 925-423-3514 > Fax: 925-423-4908 > =========================== --Russ _____________________________________________________________________ Russ Rew UCAR Unidata Program russ@xxxxxxxxxxxxxxxx http://my.unidata.ucar.edu
netcdf-hdf archives: