I've encountered a problem writing values of type byte to a variable
with a single unlimited dimension to a netcdf-3 format file, in revision
4.2.18 (and probably earlier versions as well). It appears to
improperly pad single-byte values out to 4-byte words. If I don't make
the dimension unlimited, I don't have this problem.
Writing a simple array of consecutive byte values gives me this:
netcdf test-byte-unlimited {
dimensions:
X = 5 ;
D = UNLIMITED ; // (18 currently)
variables:
double X(X) ;
byte V(D) ;
data:
X = _, _, _, _, _ ;
V = 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0, 4, 0, 0, 0, 5, 0 ;
}
I'm expecting:
V = 1, 2, 3, 4, 5, 6, 7, 8, 9, -1, -2, -3, -4, -5, -6, -7, -8, -9 ;
Following is a patch that appears to solve the problem for me, followed
by my test-case code. It seems this issue has come up; there was a
commented-out fix for padding of byte values, but it only applied for
files with a single variable; I really need to have this working for
files with other fixed-dimension variables in them as well.
Note that the test case below does indeed pass; re-reading the written
variable using NetCDF-Java returns the entire array without the padding.
With the padding, though, the file size is larger (186 vs 240 bytes on
my system), and the C API shows incorrect values.
This is the first time I've looked at NetCDF library code of any kind -
it will probably be obvious that I'm not familiar with the file format
internals.
Here's the patch:
diff -rupN
ncSrc-4.2.18-base/cdm/src/main/java/ucar/nc2/iosp/netcdf3/N3header.java
ncSrc-4.2.18/cdm/src/main/java/ucar/nc2/iosp/netcdf3/N3header.java
---
ncSrc-4.2.18-base/cdm/src/main/java/ucar/nc2/iosp/netcdf3/N3header.java
2010-12-17 10:08:16.000000000 -0800
+++ ncSrc-4.2.18/cdm/src/main/java/ucar/nc2/iosp/netcdf3/N3header.java
2011-01-14 16:54:17.000000000 -0800
@@ -883,18 +883,10 @@ public class N3header {
raf.writeInt(n);
}
- // Note on padding: In the special case of only a single record
variable of character, byte, or short
- // type, no padding is used between data values.
- boolean usePadding = true;
- /* if (n == 1) {
- Variable var = vars.get(0);
- DataType dtype = var.getDataType();
- if ((dtype == DataType.CHAR) || (dtype == DataType.BYTE) ||
(dtype == DataType.SHORT))
- usePadding = false;
- } */
-
for (int i = 0; i < n; i++) {
Variable var = vars.get(i);
+ DataType dtype = var.getDataType();
+
writeString(var.getName());
// dimensions
@@ -908,7 +900,8 @@ public class N3header {
if (!dim.isUnlimited())
vsize *= dim.getLength();
}
- if (usePadding)
+ if (!var.isUnlimited() ||
+ ((dtype != DataType.CHAR) && (dtype != DataType.BYTE) &&
(dtype != DataType.SHORT)))
vsize += padding(vsize);
// variable attributes
------
Test case:
@Test
public void byteReadWriteUnlimitedDim() throws Exception {
File f = new File("/tmp/test-byte-unlimited.nc");
byte[] data = {1, 2, 3, 4, 5, 6, 7, 8, 9, -1, -2, -3, -4, -5,
-6, -7, -8, -9};
if (f.exists()) {
f.delete();
}
NetcdfFileWriteable outf =
NetcdfFileWriteable.createNew(f.getPath());
//Dimension d = outf.addDimension("D", data.length);
Dimension d0 = outf.addDimension("X", 5);
Dimension d = outf.addUnlimitedDimension("D");
Variable v0 = outf.addVariable("X", DataType.DOUBLE, new
Dimension[]{d0});
Variable v = outf.addVariable("V", DataType.BYTE, new
Dimension[]{d});
assertEquals(1, v.getElementSize());
outf.create();
Array arr = Array.factory(DataType.BYTE, new
int[]{data.length}, data);
outf.write(v.getName(), arr);
outf.close();
NetcdfFile ncf = NetcdfFile.open(f.getPath());
Variable inv = ncf.findVariable("V");
assertEquals(inv.getDataType(), DataType.BYTE);
int[] org = {0};
byte[] readdata = (byte[]) inv.read(org,
inv.getShape()).copyTo1DJavaArray();
assertEquals(1, inv.getElementSize());
assertArrayEquals(data, readdata);
// this test passes, but ncdump shows improper zero-padding
}
--------
--
Chris Chamberlin <chris.chamberlin@xxxxxxxx>
NOAA Center for Tsunami Research, NOAA PMEL/UW JISAO
7600 Sand Point Way NE Bldg 3, Seattle, WA 98115 USA
+1 206-526-6809 http://nctr.pmel.noaa.gov/
