Interesting -- I've lost track a bit, but I thought that one of my
colleagues had gotten a similar error with NCO.
I think what may be going on here is that NCO made a copy by creating a new
file, and copying everything over -- I haven't tried that with Python (or
any way) yet, but maybe it doesn't check the _FillValue attribute if you
set it after creating the variable?
-CHB
On Wed, Oct 2, 2024 at 4:46 PM Dave Allured - NOAA Affiliate via
netcdfgroup <netcdfgroup@xxxxxxxxxxxxxxxx> wrote:
> Chris, thanks for filing the issue with netcdf-C.
>
> Now this just got weirder. I wondered if NCO Utilities might be able to
> copy that file, and somehow sanitize the offending _FillValue attributes.
> Either fail like nccopy did, or fix the attribute data type, or something .
> "ncks infile.nc outfile.nc." Well it replicated the input file with no
> complaints. However, IT PRESERVED THE _FILLVALUE TYPE MISMATCHES. This
> with modern NCO and library versions:
>
> ncks -r: ncks version 5.2.1
> ncdump -hs: :_NCProperties = "version=2,netcdf=4.9.2,hdf5=1.14.3"
> ;
> :_SuperblockVersion = 2 ;
>
> I thought this was no longer possible with the current netcdf-C. This
> blows away my assertion that an older netcdf-C version was responsible.
> How did NCO do that? Don't know. I am giving up for today. Perhaps an
> NCO or netcdf developer can shed more light.
>
> If this is pursued any deeper, someone will need to isolate netcdf-C
> library behavior from surrounding software, including the netcdf-python
> library, and/or NCO code.
>
>
> On Wed, Oct 2, 2024 at 5:14 PM Chris Barker via netcdfgroup <
> netcdfgroup@xxxxxxxxxxxxxxxx> wrote:
>
>> OK -- issue posted:
>>
>> https://github.com/Unidata/netcdf-c/issues/3038
>>
>> I only did the C lib, 'cause it's showing up in C and python -- is
>> Fortran a different code base? or does it use the same C lib under hood?
>>
>> As for how high priority it is -- that's not up to me -- not a high
>> enough priority for me to write a PR :-)
>>
>> Thanks,
>>
>> -CHB
>>
>>
>> On Wed, Oct 2, 2024 at 4:02 PM Roy Mendelssohn - NOAA Federal via
>> netcdfgroup <netcdfgroup@xxxxxxxxxxxxxxxx> wrote:
>>
>>> Actually now that I think of it, since you can read the file a better
>>> solution is to just create a new file with the proper attributes and then
>>> item by item copy the values ( i just checked that at least in Python you
>>> can read the values and they appear to be properly masked). A pain, but
>>> once written should work for any similar fils from cmem.
>>>
>>> -Roy
>>>
>>> > On Oct 2, 2024, at 3:52 PM, Roy Mendelssohn - NOAA Federal <
>>> roy.mendelssohn@xxxxxxxx> wrote:
>>> >
>>> > To edit right now I think a hex editor will work. Even in a good text
>>> editor FillValue shows up, it is just not clear how to reset it without
>>> seeing the hex.. But a hex editor should be able to get at it.
>>> >
>>> > -Roy
>>> >
>>> >
>>> >> On Oct 2, 2024, at 3:31 PM, Dave Allured - NOAA Affiliate via
>>> netcdfgroup <netcdfgroup@xxxxxxxxxxxxxxxx> wrote:
>>> >>
>>> >> Yes in my opinion, this is worth issues in both the netcdf-c and
>>> netcdf-python libraries, but at low priority. I think this is a relatively
>>> obscure case. The hope is that if fixed in the libraries, it will avoid
>>> more debugging efforts by future users.
>>> >>
>>> >>
>>> >> On Wed, Oct 2, 2024 at 3:10 PM Chris Barker <chris.barker@xxxxxxxx>
>>> wrote:
>>> >> On Wed, Oct 2, 2024 at 1:44 PM Dave Allured - NOAA Affiliate via
>>> netcdfgroup <netcdfgroup@xxxxxxxxxxxxxxxx> wrote:
>>> >> This file was probably written by a pre-2013 version of the netcdf-C
>>> library,
>>> >>
>>> >> Ahh, that may explain it.
>>> >>
>>> >> Modern libraries choke on the type mismatch, as seen here.
>>> >> ...
>>> >> However IMO, libraries should also support problem-free writes to
>>> such existing files.
>>> >>
>>> >> Is this worth an issue on the netCDF C lib?
>>> >>
>>> >> -CHB
>>> >>
>>> >>
>>> >> It may be possible to "fix" files like this, by replacing or deleting
>>> the offending _FillValue attributes with lower level or capable software,
>>> such as the HDF5 library.
>>> >>
>>> >>
>>> >> On Wed, Oct 2, 2024 at 1:58 PM Chris Barker via netcdfgroup <
>>> netcdfgroup@xxxxxxxxxxxxxxxx> wrote:
>>> >> I was wondering what the NaNf meant :-)
>>> >>
>>> >> But yeah, that's probably it -- now to track down whoever wrote this
>>> file --
>>> >> But I do wonder what software allowed this?
>>> >>
>>> >> -CHB
>>> >>
>>> >>
>>> >> On Wed, Oct 2, 2024 at 12:53 PM Benjamin Root <ben.v.root@xxxxxxxxx>
>>> wrote:
>>> >> I think the issue is that the _FillValue of "NaN" is probably a
>>> float64 while the variable's dtype is float32, which is probably what it
>>> means by a type mismatch. When I have a NaN FillValue for a float32
>>> variable, it prints "NaNf"
>>> >>
>>> >> ```
>>> >> float sum_tp_mm(time, latitude, longitude) ;
>>> >> sum_tp_mm:_FillValue = NaNf ;
>>> >> ```
>>> >>
>>> >> On Wed, Oct 2, 2024 at 12:05 PM Chris Barker via netcdfgroup <
>>> netcdfgroup@xxxxxxxxxxxxxxxx> wrote:
>>> >> Thanks Roy, at least it's not just us, and that error did give me a
>>> hint, look at this from ncdump-ing the header:
>>> >>
>>> >> float uo(time, depth, latitude, longitude) ;
>>> >> string uo:units = "m s-1" ;
>>> >> uo:_FillValue = NaN ;
>>> >> string uo:standard_name = "eastward_sea_water_velocity" ;
>>> >> string uo:long_name = "Eastward velocity" ;
>>> >>
>>> >> There's no type for the uo:_FillValue attribute.
>>> >>
>>> >> How in the world did that happen?
>>> >>
>>> >> At least now I have a specific thing to point the file creators to.
>>> >>
>>> >> -CHB
>>> >>
>>> >>
>>> >> On Wed, Oct 2, 2024 at 10:58 AM Roy Mendelssohn - NOAA Federal <
>>> roy.mendelssohn@xxxxxxxx> wrote:
>>> >> I can reproduce the error. I tried to copy the file using mccoy,
>>> got the following:
>>> >>
>>> >> nccopy cmems_mod_nws_phy_anfc_0.027deg-3D_P1D-m_1727700325546.nc
>>> test.nc
>>> >> NetCDF: Not a valid data type or _FillValue type mismatch
>>> >> Location: file ?; fcn ? line 1408
>>> >>
>>> >> hope this helps.
>>> >>
>>> >> -Roy
>>> >>
>>> >>
>>> >>> On Oct 2, 2024, at 10:46 AM, Chris Barker via netcdfgroup <
>>> netcdfgroup@xxxxxxxxxxxxxxxx> wrote:
>>> >>>
>>> >>> I have a user unable to use a netcdf file with our code.
>>> >>>
>>> >>> It turns out that the file works just fine when read, but if you try
>>> to open it for writing, it chokes:
>>> >>>
>>> >>> (this is the netCDF4 Python lib, version 1.6.5, which seems to be
>>> the latest release)
>>> >>>
>>> >>> I get:
>>> >>> In [3]: netCDF4.Dataset('
>>> cmems_mod_nws_phy_anfc_0.027deg-3D_P1D-m_1727700325546.nc', 'r+')
>>> >>>
>>> ---------------------------------------------------------------------------
>>> >>> OSError Traceback (most recent
>>> call last)
>>> >>> Cell In[3], line 1
>>> >>> ----> 1 netCDF4.Dataset('
>>> cmems_mod_nws_phy_anfc_0.027deg-3D_P1D-m_1727700325546.nc', 'r+')
>>> >>>
>>> >>> File src/netCDF4/_netCDF4.pyx:2469, in
>>> netCDF4._netCDF4.Dataset.__init__()
>>> >>>
>>> >>> File src/netCDF4/_netCDF4.pyx:2028, in
>>> netCDF4._netCDF4._ensure_nc_success()
>>> >>>
>>> >>> OSError: [Errno -103] NetCDF: Can't write file: '
>>> cmems_mod_nws_phy_anfc_0.027deg-3D_P1D-m_1727700325546.nc'
>>> >>>
>>> >>> Any idea how I can get more info as to what's wrong with it?
>>> >>>
>>> >>> Again, we have had no problems reading the file.
>>> >>>
>>> >>> Thanks,
>>> >>> - CHB
>>> >>>
>>> >>> File is 1.2MB, (not huge by today's standards?) so I've attached it
>>> here if anyone wants to take a look.
>>> >>> --
>>> >>> Christopher Barker, Ph.D.
>>> >>> Oceanographer
>>> >>>
>>> >>> Emergency Response Division
>>> >>> NOAA/NOS/OR&R (206) 526-6959 voice
>>> >>> 7600 Sand Point Way NE (206) 526-6329 fax
>>> >>> Seattle, WA 98115 (206) 526-6317 main reception
>>> >>>
>>> >>> Chris.Barker@xxxxxxxx
>>>
>>
>> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
>
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@xxxxxxxx
