Hi again Chris,
This bug is fixed in the netCDF-java 4.2.20 release.
This bug only applies to netCDF-3 classic or 64-bit datasets that meet
*all* of these conditions:
- the dataset has only one variable that uses the unlimited dimension
- the type of that variable is byte, char, or short
- the number of bytes per record for that variable is not divisible by
four
- the dataset has more than one record (i.e., the unlimited dimension
is greater than one)
- the dataset was written by netCDF-Java version 4.2.19 or earlier
Any such files would have been written with incorrect padding between
records and will give incorrect results beyond the first record when
reading values for that record variable. This is true whether reading
from the netCDF-C or netCDF-Java libraries.
Thanks,
Ethan
On 1/19/2011 11:35 PM, Ethan Davis wrote:
> Hi Chris,
>
> Thanks for the detailed report. I'm able to replicate the problem here.
>
> It only affects the case where there is only one variable that uses the
> unlimited dimension and that variable is byte, char, or short. This case
> has a special handling note in the netCDF-3 file format specification.
> The netCDF-Java library wasn't handling it quite right.
>
> I think I have a fix but I need to write a few tests before releasing.
> Also, heading to a conference next week so that may delay this a bit.
>
> We'll send word when a release with this fix is available.
>
> Ethan
>
> On 1/14/2011 6:20 PM, Chris Chamberlin wrote:
>> I've encountered a problem writing values of type byte to a variable
>> with a single unlimited dimension to a netcdf-3 format file, in revision
>> 4.2.18 (and probably earlier versions as well). It appears to
>> improperly pad single-byte values out to 4-byte words. If I don't make
>> the dimension unlimited, I don't have this problem.
>>
>> Writing a simple array of consecutive byte values gives me this:
>>
>> netcdf test-byte-unlimited {
>> dimensions:
>> X = 5 ;
>> D = UNLIMITED ; // (18 currently)
>> variables:
>> double X(X) ;
>> byte V(D) ;
>> data:
>>
>> X = _, _, _, _, _ ;
>>
>> V = 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0, 4, 0, 0, 0, 5, 0 ;
>> }
>>
>> I'm expecting:
>> V = 1, 2, 3, 4, 5, 6, 7, 8, 9, -1, -2, -3, -4, -5, -6, -7, -8, -9 ;
>>
>>
>> Following is a patch that appears to solve the problem for me, followed
>> by my test-case code. It seems this issue has come up; there was a
>> commented-out fix for padding of byte values, but it only applied for
>> files with a single variable; I really need to have this working for
>> files with other fixed-dimension variables in them as well.
>>
>> Note that the test case below does indeed pass; re-reading the written
>> variable using NetCDF-Java returns the entire array without the padding.
>> With the padding, though, the file size is larger (186 vs 240 bytes on
>> my system), and the C API shows incorrect values.
>>
>> This is the first time I've looked at NetCDF library code of any kind -
>> it will probably be obvious that I'm not familiar with the file format
>> internals.
>>
>> Here's the patch:
>>
>> diff -rupN
>> ncSrc-4.2.18-base/cdm/src/main/java/ucar/nc2/iosp/netcdf3/N3header.java
>> ncSrc-4.2.18/cdm/src/main/java/ucar/nc2/iosp/netcdf3/N3header.java
>> ---
>> ncSrc-4.2.18-base/cdm/src/main/java/ucar/nc2/iosp/netcdf3/N3header.java
>> 2010-12-17 10:08:16.000000000 -0800
>> +++ ncSrc-4.2.18/cdm/src/main/java/ucar/nc2/iosp/netcdf3/N3header.java
>> 2011-01-14 16:54:17.000000000 -0800
>> @@ -883,18 +883,10 @@ public class N3header {
>> raf.writeInt(n);
>> }
>>
>> - // Note on padding: In the special case of only a single record
>> variable of character, byte, or short
>> - // type, no padding is used between data values.
>> - boolean usePadding = true;
>> - /* if (n == 1) {
>> - Variable var = vars.get(0);
>> - DataType dtype = var.getDataType();
>> - if ((dtype == DataType.CHAR) || (dtype == DataType.BYTE) ||
>> (dtype == DataType.SHORT))
>> - usePadding = false;
>> - } */
>> -
>> for (int i = 0; i < n; i++) {
>> Variable var = vars.get(i);
>> + DataType dtype = var.getDataType();
>> +
>> writeString(var.getName());
>>
>> // dimensions
>> @@ -908,7 +900,8 @@ public class N3header {
>> if (!dim.isUnlimited())
>> vsize *= dim.getLength();
>> }
>> - if (usePadding)
>> + if (!var.isUnlimited() ||
>> + ((dtype != DataType.CHAR) && (dtype != DataType.BYTE) &&
>> (dtype != DataType.SHORT)))
>> vsize += padding(vsize);
>>
>> // variable attributes
>>
>>
>>
>> ------
>> Test case:
>>
>> @Test
>> public void byteReadWriteUnlimitedDim() throws Exception {
>> File f = new File("/tmp/test-byte-unlimited.nc");
>> byte[] data = {1, 2, 3, 4, 5, 6, 7, 8, 9, -1, -2, -3, -4, -5,
>> -6, -7, -8, -9};
>> if (f.exists()) {
>> f.delete();
>> }
>> NetcdfFileWriteable outf =
>> NetcdfFileWriteable.createNew(f.getPath());
>> //Dimension d = outf.addDimension("D", data.length);
>> Dimension d0 = outf.addDimension("X", 5);
>> Dimension d = outf.addUnlimitedDimension("D");
>> Variable v0 = outf.addVariable("X", DataType.DOUBLE, new
>> Dimension[]{d0});
>> Variable v = outf.addVariable("V", DataType.BYTE, new
>> Dimension[]{d});
>> assertEquals(1, v.getElementSize());
>> outf.create();
>>
>> Array arr = Array.factory(DataType.BYTE, new int[]{data.length},
>> data);
>> outf.write(v.getName(), arr);
>> outf.close();
>>
>> NetcdfFile ncf = NetcdfFile.open(f.getPath());
>> Variable inv = ncf.findVariable("V");
>> assertEquals(inv.getDataType(), DataType.BYTE);
>> int[] org = {0};
>> byte[] readdata = (byte[]) inv.read(org,
>> inv.getShape()).copyTo1DJavaArray();
>>
>> assertEquals(1, inv.getElementSize());
>> assertArrayEquals(data, readdata);
>> // this test passes, but ncdump shows improper zero-padding
>> }
>>
>>
>> --------
>>
>>
>>
