Okay I did some more testing. I used R and the package "ncdf4" and it was able
to read the file. I also tried in python netCDF4 and xarray and they both
failed on the call to open the file. For python I am running 3.6.4, the conda
distribution, with the latest packages.
I believe that the R package ncdf4 is linked to an earlier version of the
netcdf library, which must be more forgiving of the bad offset. But as I said,
on my machine (MacBook pro, osx 10.13.5) two versions of ncdump fail on the
file (the one that comes with anaconda and the one that comes with fink), and
two python programs fail to open it.
This suggests the package that is creating the files is using an older version
of the library and also somehow creating corrupt files. The information sent
by Wei-keng is pretty definitive - the offset at the start of the file is
incorrect.
-Roy
> On Jun 23, 2018, at 11:03 AM, Roy Mendelssohn - NOAA Federal
> <roy.mendelssohn@xxxxxxxx> wrote:
>
> I don't know how the others get a dump on this. I tried every which way to
> download this and my ncdump still does not like it. For people who did
> download it can you mail it to me or see if netcdf4 or xarray can read it.
>
> IMO opinion, the real question is why is this file corrupt. Is it something
> to do with whatever software created it in the "minc format"
> (https://en.wikibooks.org/wiki/MINC/SoftwareDevelopment/MINC2.0_File_Format_Reference),
> something to do with the format itself, or something to do with the
> original file transfer. But I do not think you should be distributing a
> corrupt file, period.
>
> I looked at your code to read netcdf files and you are using code based on
> pupynere from our old buddy Roberto. pupynere operates at a low level on the
> file, not using the standard library calls, and therefore can probably pass
> over some problems in the file. But the fact that you can write code that
> can pass over problems in a file doesn't mean you should. If standard netcdf
> libraries/packages can't read the file, you should get a new file that isn't
> corrupt and that standard libraries can read.
>
> HTH,
>
> -Roy
>
>> On Jun 23, 2018, at 10:27 AM, Ankur Sinha <sanjay.ankur@xxxxxxxxx> wrote:
>>
>> Hi Roy, Wei-keng,
>>
>>
>> On Fri, Jun 22, 2018 19:18:42 -0500, Wei-keng Liao wrote:
>>> The file is corrupted. I ran it against ncvalidator and got the following
>>> errors.
>>
>> Thanks for looking into this. Someone had commented on the scipy issue[1]
>> about the offsets but I hadn't understood completely what they'd meant.
>> Thanks for clarifying it up.
>>
>> Would the suggested way of handling such files be to simply flag them as
>> corrupt and not process them at all then?
>>
>> While the scipy implementation crashes with this file, the netcdf
>> implementation that nibabel uses does not. I'm still trying to figure
>> out what they do differently.
>>
>> [1] https://github.com/scipy/scipy/issues/8153#issuecomment-364757198
>>
>> --
>> Thanks,
>> Regards,
>>
>> Ankur Sinha "FranciscoD"
>>
>> https://fedoraproject.org/wiki/User:Ankursinha
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web. Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>>
>>
>> netcdfgroup mailing list
>> netcdfgroup@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>
> **********************
> "The contents of this message do not reflect any position of the U.S.
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> ***Note new street address***
> 110 McAllister Way
> Santa Cruz, CA 95060
> Phone: (831)-420-3666
> Fax: (831) 420-3980
> e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
>
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected"
> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
>
**********************
"The contents of this message do not reflect any position of the U.S.
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new street address***
110 McAllister Way
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
