Re: [netcdf-java] no IOException for shortened file

  • To: Christian Ward-Garrison <cwardgar@xxxxxxxx>
  • Subject: Re: [netcdf-java] no IOException for shortened file
  • From: Bob Simons - NOAA Federal <bob.simons@xxxxxxxx>
  • Date: Wed, 2 Nov 2016 08:59:44 +0000
Thank you. That is very useful information.

I am sad to hear that the conscious default is for the library not to throw
an error when the file is truncated (but I understand),
but happy that I can use ucar.nc2.iosp.netcdf3.N3header.
disallowFileTruncation to change this behavior for my work.


On Wed, Nov 2, 2016 at 7:16 AM, Christian Ward-Garrison <cwardgar@xxxxxxxx>
wrote:

> Hi Bob,
>
> Sorry for the delay in responding. I would have expected an exception as
> well, but apparently this is working as intended. Specifically, we're
> trying to match the behavior of the netcdf C library, which allows
> truncated files [1]. I verified this with ncdump, which printed all data
> values without error. For truncated values, ncdump printed '0', which is
> the default for floats in C. NetCDF-Java, on the other hand, used whatever
> values were left in the RandomAccessFile buffer after the last successful
> read for truncated values. Maybe that's a problem.
>
> If you like, it's possible to disallow truncated files by setting
> ucar.nc2.iosp.netcdf3.N3header.disallowFileTruncation to true, in which
> case you'll get the exception you're expecting.
>
> Cheers,
> Christian
>
> [1] http://www.unidata.ucar.edu/software/netcdf/netcdf-4/
> newdocs/netcdf-c/Fill-Values.html#Fill-Values
>
> On Tue, Oct 18, 2016 at 11:56 AM, Bob Simons - NOAA Federal <
> bob.simons@xxxxxxxx> wrote:
>
>> In our system, and I suspect elsewhere, sometimes .nc files don't get
>> completely copied from one place to another. Normally, this is caught and
>> dealt with (e.g., by the script doing the copying), but sometimes it isn't
>> (e.g., a computer is restarted while a script is running), leaving a
>> shortened file in place.
>>
>> I wanted to understand how netcdf-java (v4.6.6) reacted to these shorted
>> files. So I made the attached file by copying only the first 1/6th of a
>> valid file. Then I tried to read it with this code
>>
>>             NetcdfFile ncFile = NetcdfFile.open("/git/erddapTe
>> st/nc/invalidShortened.nc");
>>             try {
>>                 Variable var = ncFile.findVariable("MWcdom");
>>  //size=[1,1,2321,4001]
>>                 Array array = var.read();
>>                 System.out.println("shape=" +
>> Arrays.toString(array.getShape()));
>>
>>                 array = var.read("0,0,0:2000:1000,0:4000:1000");
>>                 System.out.println("shape=" +
>> Arrays.toString(array.getShape()));
>>
>>                 //System.out.println(array.toString());
>>                 System.out.println("Shouldn't get here!");
>>                 ncFile.close();
>>             } catch (Throwable t) {
>>                 try {
>>                     ncFile.close();
>>                 } catch (Throwable t2) {
>>                 }
>>                 System.out.println("caught: " + t.toString());
>>             }
>>
>> Note that Variable.read() documentation says that the method throws an
>> IOException.
>>
>> To my surprise, the above code doesn't throw any exception.
>> netcdf-java says it can read the entire var and read the specified
>> section (which gets values from throughout the variable).
>>
>> Is that what you expected?
>> Am I doing something wrong in my test?
>> Doesn't netcdf-java check the validity of the file?
>> Why don't the read()s cause an IOException?
>>
>> Thank you for looking into this.
>>
>> --
>> Sincerely,
>>
>> Bob Simons
>> IT Specialist
>> Environmental Research Division
>> NOAA Southwest Fisheries Science Center
>> 99 Pacific St., Suite 255A      (New!)
>> Monterey, CA 93940               (New!)
>> Phone: (831)333-9878            (New!)
>> Fax:   (831)648-8440
>> Email: bob.simons@xxxxxxxx
>>
>> The contents of this message are mine personally and
>> do not necessarily reflect any position of the
>> Government or the National Oceanic and Atmospheric Administration.
>> <>< <>< <>< <>< <>< <>< <>< <>< <><
>>
>>
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web.  Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>>
>>
>> netcdf-java mailing list
>> netcdf-java@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>>
>
>


-- 
Sincerely,

Bob Simons
IT Specialist
Environmental Research Division
NOAA Southwest Fisheries Science Center
99 Pacific St., Suite 255A      (New!)
Monterey, CA 93940               (New!)
Phone: (831)333-9878            (New!)
Fax:   (831)648-8440
Email: bob.simons@xxxxxxxx

The contents of this message are mine personally and
do not necessarily reflect any position of the
Government or the National Oceanic and Atmospheric Administration.
<>< <>< <>< <>< <>< <>< <>< <>< <><
  • 2016 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: