Peyush Jain wrote:
Hi John,
I am still having memory problems (java.lang.OutOfMemoryError). I have 480
(to be precise) 1-D arrays with data and an additional 1-D array with time
data. I get this information every few seconds. By moving the offset (for
all 481 arrays) and not writing to the file every time data is received, I
run out of memory. So, I am wondering if there is a way to write to the disk
after each time data is received.
you want to write the data out to disk which is what ncfile.write() does.
do this each time step instead of accumulating data.
in java, you have to be careful to release all references to objects,
else you run out of memory.
in your case, you probably can reuse your data arrays.
some comments below
I am not sure how to do this: "not store more than one time step at a time".
So, here is what I do... I create 481 variables (with unlimited length) and
every time I get new data I move the offset by 200 (length of each incoming
array). The following is a very simple version of what I am trying to do:
----------------------------------------------------------------------------
Dimension[] dims;
Dimension timeDim;
private static boolean createFile = true;
private static int offset = 0;
public void processData(Data time, Data[] d)
{
if(createFile)
{
timeDim = ncfile.addDimension("time", -1);
ncfile.addVariable("time", double.class, new Dimension[]{timeDim});
dims = new Dimension[d.length];
for(int i = 0; i < d.length; i++)
{
dims[i] = ncfile.addDimension(d[i].getName(), -1);
this line should have failed, probably a bug if it doesnt. You can only
have one unlimited dimension, which should be time.
ncfile.addVariable(d[i].getName(), int.class, new
Dimension[]{dims[i]});
storing this a 1-D is probably not what you want to do. better to store
it as > 1D, with time the outer dimension, and a dimension of length 200
as the inner dimension perhaps.
}
try
{
ncfile.create();
}
catch(IOException e)
{
System.err.println("ERROR creating file");
}
createFile = false;
}
int origin[] = new int[1];
origin[0] = offset;
try
{
ncfile.write("time", origin,
ArrayAbstract.factory(time.getArray()));
for(int i = 0; i < d.length; i++)
{
ncfile.write(d[i].getName(), origin,
ArrayAbstract.factory(d[i].getArray()));
}
offset = offset + 200;
ncfile.flush();
}
catch(IOException e)
{
System.err.println("ERROR writing file");
}
}
----------------------------------------------------------------------------
Thanks a lot,
Peyush
if you still have trouble, you might want to get the example working
that i sent you, then modify it for your needs.
