Hi Chris,
Thanks for the detailed report. I'm able to replicate the problem here.
It only affects the case where there is only one variable that uses the
unlimited dimension and that variable is byte, char, or short. This case
has a special handling note in the netCDF-3 file format specification.
The netCDF-Java library wasn't handling it quite right.
I think I have a fix but I need to write a few tests before releasing.
Also, heading to a conference next week so that may delay this a bit.
We'll send word when a release with this fix is available.
Ethan
On 1/14/2011 6:20 PM, Chris Chamberlin wrote:
> I've encountered a problem writing values of type byte to a variable
> with a single unlimited dimension to a netcdf-3 format file, in revision
> 4.2.18 (and probably earlier versions as well). It appears to
> improperly pad single-byte values out to 4-byte words. If I don't make
> the dimension unlimited, I don't have this problem.
>
> Writing a simple array of consecutive byte values gives me this:
>
> netcdf test-byte-unlimited {
> dimensions:
> X = 5 ;
> D = UNLIMITED ; // (18 currently)
> variables:
> double X(X) ;
> byte V(D) ;
> data:
>
> X = _, _, _, _, _ ;
>
> V = 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0, 4, 0, 0, 0, 5, 0 ;
> }
>
> I'm expecting:
> V = 1, 2, 3, 4, 5, 6, 7, 8, 9, -1, -2, -3, -4, -5, -6, -7, -8, -9 ;
>
>
> Following is a patch that appears to solve the problem for me, followed
> by my test-case code. It seems this issue has come up; there was a
> commented-out fix for padding of byte values, but it only applied for
> files with a single variable; I really need to have this working for
> files with other fixed-dimension variables in them as well.
>
> Note that the test case below does indeed pass; re-reading the written
> variable using NetCDF-Java returns the entire array without the padding.
> With the padding, though, the file size is larger (186 vs 240 bytes on
> my system), and the C API shows incorrect values.
>
> This is the first time I've looked at NetCDF library code of any kind -
> it will probably be obvious that I'm not familiar with the file format
> internals.
>
> Here's the patch:
>
> diff -rupN
> ncSrc-4.2.18-base/cdm/src/main/java/ucar/nc2/iosp/netcdf3/N3header.java
> ncSrc-4.2.18/cdm/src/main/java/ucar/nc2/iosp/netcdf3/N3header.java
> ---
> ncSrc-4.2.18-base/cdm/src/main/java/ucar/nc2/iosp/netcdf3/N3header.java
> 2010-12-17 10:08:16.000000000 -0800
> +++ ncSrc-4.2.18/cdm/src/main/java/ucar/nc2/iosp/netcdf3/N3header.java
> 2011-01-14 16:54:17.000000000 -0800
> @@ -883,18 +883,10 @@ public class N3header {
> raf.writeInt(n);
> }
>
> - // Note on padding: In the special case of only a single record
> variable of character, byte, or short
> - // type, no padding is used between data values.
> - boolean usePadding = true;
> - /* if (n == 1) {
> - Variable var = vars.get(0);
> - DataType dtype = var.getDataType();
> - if ((dtype == DataType.CHAR) || (dtype == DataType.BYTE) ||
> (dtype == DataType.SHORT))
> - usePadding = false;
> - } */
> -
> for (int i = 0; i < n; i++) {
> Variable var = vars.get(i);
> + DataType dtype = var.getDataType();
> +
> writeString(var.getName());
>
> // dimensions
> @@ -908,7 +900,8 @@ public class N3header {
> if (!dim.isUnlimited())
> vsize *= dim.getLength();
> }
> - if (usePadding)
> + if (!var.isUnlimited() ||
> + ((dtype != DataType.CHAR) && (dtype != DataType.BYTE) &&
> (dtype != DataType.SHORT)))
> vsize += padding(vsize);
>
> // variable attributes
>
>
>
> ------
> Test case:
>
> @Test
> public void byteReadWriteUnlimitedDim() throws Exception {
> File f = new File("/tmp/test-byte-unlimited.nc");
> byte[] data = {1, 2, 3, 4, 5, 6, 7, 8, 9, -1, -2, -3, -4, -5,
> -6, -7, -8, -9};
> if (f.exists()) {
> f.delete();
> }
> NetcdfFileWriteable outf =
> NetcdfFileWriteable.createNew(f.getPath());
> //Dimension d = outf.addDimension("D", data.length);
> Dimension d0 = outf.addDimension("X", 5);
> Dimension d = outf.addUnlimitedDimension("D");
> Variable v0 = outf.addVariable("X", DataType.DOUBLE, new
> Dimension[]{d0});
> Variable v = outf.addVariable("V", DataType.BYTE, new
> Dimension[]{d});
> assertEquals(1, v.getElementSize());
> outf.create();
>
> Array arr = Array.factory(DataType.BYTE, new int[]{data.length},
> data);
> outf.write(v.getName(), arr);
> outf.close();
>
> NetcdfFile ncf = NetcdfFile.open(f.getPath());
> Variable inv = ncf.findVariable("V");
> assertEquals(inv.getDataType(), DataType.BYTE);
> int[] org = {0};
> byte[] readdata = (byte[]) inv.read(org,
> inv.getShape()).copyTo1DJavaArray();
>
> assertEquals(1, inv.getElementSize());
> assertArrayEquals(data, readdata);
> // this test passes, but ncdump shows improper zero-padding
> }
>
>
> --------
>
>
>
