Hi Jean-Michel:
ok, finally found it - an overflow of an int variable in some obscure place. i
will get a new release out on monday that will fix.
BTW, you can cut your time approx in half by using:
NetcdfFileWriteable.setFill(false);
thanks for debugging!
Jean-Michel.Perraud@xxxxxxxx wrote:
Hi John,
I was not using setLargeFile(true) indeed. I still get the same exception with
it added. I attach the test program I use. I ran it both from a WinXP 32 bit
workstation and a Windows Server 2003 R2 x64 with the same behavior. The JVM is:
java version "1.6.0_11"
Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
Java HotSpot(TM) Client VM (build 11.0-b16, mixed mode, sharing)
Thanks for your help.
Cheers,
Jean-Michel
-----Original Message-----
From: John Caron [mailto:caron@xxxxxxxxxxxxxxxx]
Sent: Thursday, 30 April 2009 9:37 PM
To: Perraud, Jean-Michel (CLW, Black Mountain)
Cc: netcdf-java@xxxxxxxxxxxxxxxx
Subject: Re: [netcdf-java] ArrayIndexOutOfBoundException when generating a
large file
Hi Jean-Michel:
Are you setting NetcdfFileWriteable.setLargeFile(true) ?
Can i see your createLatLonTimeDataCube() method?
Jean-Michel.Perraud@xxxxxxxx wrote:
Hi,
I want to generate a lat*lon*time data file which I estimate will be ~16GB. I
am using the latitude as an unlimited dimension to (1) lay out the file for
efficient reading of time series and (2) as I understand, overcome the 4GB size
limit. Things work fine with small data. Testing with lat=300, lon=300 and
time=40000 the file fails to grow beyond 8GB. This is only my second
significant use of netcdf and I am probably missing something obvious to
seasoned netCDF users.
I am using netcdfAll-4.0.jar. The stack trace in Eclipse is from the close()
method, but things must go pear-shaped before as only half the records are
output (179 out of 300 latitudes).
Any advice would be appreciated. Sample code/netcdf structure follows.
Cheers
-------------------------------------------
Jean-Michel Perraud
Prediction and Reporting Technologies, CSIRO Land and Water
Thread [main] (Suspended (exception ArrayIndexOutOfBoundsException))
RandomAccessFile.write(int) line: 747 //
ArrayIndexOutOfBoundsException: -2013020
RandomAccessFile.writeInt(int) line: 1438
N3header.writeNumrecs() line: 900
N3raf(N3iosp).flush() line: 869
NetcdfFileWriteable.flush() line: 731
NetcdfFileWriteable.close() line: 741
TestLargeGeneration.main(String[]) line: 55
netcdf file://XXXX/tmp/test.nc {
dimensions:
time = 40000;
lat = UNLIMITED; // (179 currently)
lon = 300;
variables:
float testVar(lat=179, lon=300, time=40000);
:long_name = "testVar";
:units = "mm";
:missing_value = -9999.0f; // float
:_FillValue = -9999.0f; // float
char cellId(lat=179, lon=300);
:long_name = "Cell ID";
float lat(lat=179);
:units = "degrees_north";
:axis = "Y";
:standard_name = "latitude";
:_CoordinateAxisType = "Lat";
float lon(lon=300);
:units = "degrees_east";
:axis = "X";
:standard_name = "longitude";
:_CoordinateAxisType = "Lon";
int time(time=40000);
:units = "days since 1889-01-01";
:axis = "T";
:standard_name = "time";
:long_name = "time";
:_CoordinateAxisType = "Time";
}
#################################
try {
writeableFile = createLatLonTimeDataCube(filename);
// The outer dimension has shape 1, since we will write
one record
// at a time, and one
// record corresponds to one latitude ordinate
ArrayFloat.D3 variableData = new ArrayFloat.D3(1,
LON_LEN, TIME_LEN);
// ArrayObject.D2 cellIdData = new
// ArrayObject.D2(class(java.lang.String),
latDim.getLength(),
// lonDim.getLength());
// An array to record the latitude ordinate
ucar.ma2.Array latData =
ucar.ma2.Array.factory(DataType.FLOAT,
new int[] { 1 });
// The origin to use to write the runoff record
int[] origin = new int[] { 0, 0, 0 }; // lat, lon, time
// The origin to use to write the latitude for each
record
int[] lat_origin = new int[] { 0 };
for (int lat = 0; lat < LAT_LEN; lat++) {
latData.setFloat(latData.getIndex(), (float)
(lat / 10.0));
for (int lon = 0; lon < LON_LEN; lon++) {
for (int time = 0; time < TIME_LEN;
time++) {
variableData.set(0, lon, time,
(float) time);
}
}
// write the data out for this record
origin[0] = lat;
lat_origin[0] = lat;
writeableFile.write(variableName, origin,
variableData);
writeableFile.write(latVar, lat_origin,
latData);
writeableFile.flush();
}
} finally {
if (writeableFile != null)
writeableFile.close(); // Exception is caught
here.
}
#################################
_______________________________________________
netcdf-java mailing list
netcdf-java@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
