Hi Paul,
Well, you don't say what the size of each timestep is, but as the size of
each timestep becomes small (< 50 MB maybe?) I would think that doing each
timestep as a separate group (if that's what you're doing) would, for a
5000 timestep array, take ~5000 times as long. That's since the set up time
is very considerable, and the incremental time for a second timestep after
you've set up for a first timestep is small (unless each timestep is quite
large).
For someone who doesn't know just what you're doing this part is pretty
hard to parse:
"I did this so each group can have different "base" dimensions for the data
arrays."
Maybe you could give the specific example? Not knowing the details it's
hard to see why it would be desirable to take the multi-group approach, or
to think about alternate approaches that would accomplish your goal but
might be more efficient.
Regards,
--Dave
On Wed, Oct 30, 2013 at 8:00 AM, Paul van Delst <paul.vandelst@xxxxxxxx>wrote:
> Hello,
>
> I've just converted some of my netCDF writing code to write/read multiple
> groups rather than use an unlimited dimension. I did this so each group can
> have different "base" dimensions for the data arrays.
>
> I have one data set where the unlimited dimension is 5000. The read/write
> of this data in netCDF3 format is almost instantaneous. When I use the
> netCDF4 approach (reading and writing 5000 separate groups) the reads and
> write can take upwards of 10minutes (I started the program at 10:33am. It
> is now 10:51am and the read of the created file is still going on).
>
> I realise there's going to be additional overhead using the "groups"
> approach (defining dimensions and variables for each group) but I presume
> I'm doing something very wrong/stupid to cause the I/O to be as slow as it
> is. Before I start posting code snippets, does anyone have any experience
> hints as to what could be causing this supa slow I/O?
>
> Thanks for any info.
>
> cheers,
>
> paulv
>
> p.s. It's now 11:00am and the dataset reading is still going on...
>
> ______________________________**_________________
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/**mailing_lists/<http://www.unidata.ucar.edu/mailing_lists/>
--
David W. Pierce
Division of Climate, Atmospheric Science, and Physical Oceanography
Scripps Institution of Oceanography, La Jolla, California, USA
(858) 534-8276 (voice) / (858) 534-8561 (fax) dpierce@xxxxxxxx
--
David W. Pierce
Division of Climate, Atmospheric Science, and Physical Oceanography
Scripps Institution of Oceanography, La Jolla, California, USA
(858) 534-8276 (voice) / (858) 534-8561 (fax) dpierce@xxxxxxxx
