I think I may have found a bug in 4.1, and I'd like to know if it's actually a
bug, and if there's
a workaround.
What I'm trying to do is write a NetCDF file by alternating define mode and
writing mode. It's
difficult in my code to get around doing that, because there are some
preliminary CF
metadata variables to set up before writing actual variable data, and those
code modules are
independent of each other.
What I decided to do was initially allocate 64k in the header right after
creating the file
object, so that it has enough space for the global metadata and then when I add
new
variables and attributes, it doesn't have to rewrite the whole file as the
documentation states
can be I/O intensive. But when I alternately define new variables and write
data to them, the
variable data in never written in some cases.
Here's a sample program that reproduces the problem:
----------------------
import ucar.nc2.*;
import ucar.ma2.*;
public class WriteTest {
public static void main (String args[]) throws Exception {
NetcdfFileWriteable ncFile = NetcdfFileWriteable.createNew (args[0], false);
ncFile.setExtraHeaderBytes (64*1024);
Dimension dim = ncFile.addDimension ("time", 100);
double[] jackData = new double[100];
for (int i = 0; i < 100; i++) jackData[i] = i;
double[] jillData = new double[100];
for (int i = 0; i < 100; i++) jillData[i] = i;
Dimension[] dims = new Dimension[] {dim};
ncFile.addVariable ("jack", DataType.DOUBLE, dims);
ncFile.addVariableAttribute ("jack", "where", "up the hill");
ncFile.create();
int[] start = new int[] {0};
int[] count = new int[] {100};
ncFile.write ("jack", start, Array.factory (double.class, count, jackData));
ncFile.setRedefineMode (true);
ncFile.addVariable ("jill", DataType.DOUBLE, dims);
ncFile.addVariableAttribute ("jill", "where", "up the hill");
ncFile.setRedefineMode (false);
ncFile.write ("jill", start, Array.factory (double.class, count, jillData));
ncFile.flush();
ncFile.close();
} // main
} // WriteTest class
----------------------
I compile and run it like this:
----------------------
phollema@Bean<cwf-3.2.4> javac -cp lib/toolsUI-4.1.jar:. WriteTest.java
phollema@Bean<cwf-3.2.4> java -cp lib/toolsUI-4.1.jar:. WriteTest mytest.nc
phollema@Bean<cwf-3.2.4> ncdump mytest.nc
netcdf mytest {
dimensions:
time = 100 ;
variables:
double jack(time) ;
jack:where = "up the hill" ;
double jill(time) ;
jill:where = "up the hill" ;
data:
jack = 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37,
38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55,
56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73,
74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91,
92, 93, 94, 95, 96, 97, 98, 99 ;
jill = 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0 ;
}
----------------------
As you can see, it never wrote the data for "jill". I found that if I remove
the call to
setExtraHeaderBytes(), it works correctly and writes all the data. But that
defeats my ability
to ensure that the entire file won't be re-written when I want to add a new
variable. I want to
be able to write files with hundreds of megabytes of data, and never have it
re-write the data
when I go into and out of define mode.
Any suggestions?
Peter
