Breaking up into GB sized chunks will not be a significant performance
hit. If you expect your users to run close to the memory limit, you'll
be answering messages about this error all the time. Better to
implement the solution now IMHO. -- Rob
On Nov 29, 2010, at 10:47 AM, Jennifer Adams wrote:
My nc_put_vara_double() call writes the entire array at once; in
this case, it sounds like the library does more or less the same
thing as my algorithm outlined in the original message.
Breaking up the write into pieces would probably be a performance
hit, and would make my code a bit more complicated, since deciding
if splitting is necessary and how many splits to make is highly
system dependent.
I guess I will have to live with being a memory hog -- if I document
this feature, then it becomes the user's problem, not mine. But it
seems to me that it is still better to let the library do the type
conversion. Will the library return a meaningful error if it is
unable to allocate the required memory? Have I overlooked any other
costs/benefits?
--Jennifer
On Nov 29, 2010, at 10:51 AM, Rob Ross wrote:
So you will want to split up the writes into multiple writes to
keep memory allocation down. Perhaps 1GB at a time, given what you
describe below. Still a lot easier than doing the conversion by hand!
Rob
On Nov 29, 2010, at 9:45 AM, Ed Hartnett wrote:
Jennifer Adams <jma@xxxxxxxxxxxxx> writes:
That's great! Does the library use a chunk of memory to do this?
If my
array of doubles is very large, say 20Gb, and I'm running on a
system with
24GB of memory, will it work?
--Jennifer
On Nov 29, 2010, at 9:32 AM, Jim Edwards wrote:
Memory will be allocated, but only enough to convert data for each
nc_put_vara call. So if you are writing 1 MB of values at a time, it
will allocated 1 MB at a time to convert, and free that 1 MB after
the
write operation is complete.
So it doesn't matter how large your variables are, only how much
you try
and write at one time.
Thanks,
Ed
--
Ed Hartnett -- ed@xxxxxxxxxxxxxxxx
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
--
Jennifer M. Adams
IGES/COLA
4041 Powder Mill Road, Suite 302
Calverton, MD 20705
jma@xxxxxxxxxxxxx
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
