NOTE: The netcdf-hdf mailing list is no longer active. The list archives are made available for historical reasons.
> "Robert E. McGrath" <mcgrath@xxxxxxxxxxxxx> writes:
>
> > On Thu, 30 Jun 2005, Ed Hartnett wrote:
> >
> >> As I understand it, using fixed-length data, my different processes
> >> could write their data and go on their merry way, without waiting for
> >> anything.
> >
> > This can be done only if you know that the different processors are
> > not updating the same chunk.
>
> Surely this is easy to arrange.
Actually, the HDF5 library handles the case where the processors are
updating the same chunk correctly, even for independent writes.
> > Fixed vs extended is not relevant to the basic issue: you must
> > coordinate all writes in parallel.
>
> But aren't writes that don't call extend independent? Doesn't that
> mean processes writing to them don't have to coordinate?
>
> Or did you mean they have to coordinate in that they each have to know
> what chunks they can write, but they don't have to wait for the other
> processes to do their writing.
They do have to coordinate in the sense that the processes shouldn't
expect any ordering to their independent writes (which is only a problem if
they overlap the elements they are writing). Otherwise, as I said above, it
is possible for two processes to write to the same (non-compressed) chunk
independently or collectively.
> >> As currently implemented, H5Dextend is called when needed as you write
> >> the data in netcdf. That is, if you are writing a record at a time,
> >> H5Dextend is called for each record.
> >
> > This will be quite slow in parallel. But it will work.
> >
> > Presumably, user's can control this by batching the writes.
> >
>
> Well, if we batched the extends, that would help, right?
>
> For example, instead of extending it one record at a time, I could
> extend it 10 records at a time, and then for 9 out of 10 writes, I
> wouldn't have to call extend.
>
> Or am I barking up the wrong tree?
No, that's a reasonable thing to try. It will require some tuning and
keeping of state in the netCDF4 layer though.
Quincey
> Thanks!
>
> Ed
>
>
> --
> Ed Hartnett -- ed@xxxxxxxxxxxxxxxx
>
netcdf-hdf archives: