Chris, here is a simple two-line workaround for your original question. I
found this by accident when a colleague suggested ncatted (NCO). I
shortened file names for clarity. As expected, the one-line version does
NOT work.
ncks orig.nc copy.nc
ncatted -a _FillValue,uo,o,f,NaN -a _FillValue,vo,o,f,NaN copy.nc
This successfully changes the NaN (double) _FillValue attributes into NaNf
(floats), while keeping the data intact. This tested okay with recent
software versions: NCO 5.2.1, netcdf 4.9.2, and hdf5 1.14.3.
This workaround is disturbing. Step 1 creates a new netcdf-4 file with a
_FillValue data type that is different than the variable data type. This
is supposed to be impossible, according to current netcdf-4 documentation.
Step 2 changes the _FillValue attribute of existing netcdf-4 variables with
data. This is also supposed to be impossible.
Therefore I am clueless as to why this works. Try and see if this works on
your system. Also see whether the resulting file is now readable and
writable by netcdf4-python. Also the behavior will probably depend on
software versions, and may change with future updates.
On Wed, Oct 2, 2024 at 6:01 PM Chris Barker <chris.barker@xxxxxxxx> wrote:
> Interesting -- I've lost track a bit, but I thought that one of my
> colleagues had gotten a similar error with NCO.
>
> I think what may be going on here is that NCO made a copy by creating a
> new file, and copying everything over -- I haven't tried that with Python
> (or any way) yet, but maybe it doesn't check the _FillValue attribute if
> you set it after creating the variable?
>
> -CHB
>
>
> On Wed, Oct 2, 2024 at 4:46 PM Dave Allured - NOAA Affiliate via
> netcdfgroup <netcdfgroup@xxxxxxxxxxxxxxxx> wrote:
>
>> Chris, thanks for filing the issue with netcdf-C.
>>
>> Now this just got weirder. I wondered if NCO Utilities might be able to
>> copy that file, and somehow sanitize the offending _FillValue attributes .
>> Either fail like nccopy did, or fix the attribute data type, or something.
>> "ncks infile.nc outfile.nc." Well it replicated the input file with no
>> complaints. However, IT PRESERVED THE _FILLVALUE TYPE MISMATCHES. This
>> with modern NCO and library versions:
>>
>> ncks -r: ncks version 5.2.1
>> ncdump -hs: :_NCProperties =
>> "version=2,netcdf=4.9.2,hdf5=1.14.3" ;
>> :_SuperblockVersion = 2 ;
>>
>> I thought this was no longer possible with the current netcdf-C. This
>> blows away my assertion that an older netcdf-C version was responsible.
>> How did NCO do that? Don't know. I am giving up for today. Perhaps an
>> NCO or netcdf developer can shed more light.
>>
>> If this is pursued any deeper, someone will need to isolate netcdf-C
>> library behavior from surrounding software, including the netcdf-python
>> library, and/or NCO code.
>>
>>
>> On Wed, Oct 2, 2024 at 5:14 PM Chris Barker via netcdfgroup <
>> netcdfgroup@xxxxxxxxxxxxxxxx> wrote:
>>
>>> OK -- issue posted:
>>>
>>> https://github.com/Unidata/netcdf-c/issues/3038
>>>
>>> I only did the C lib, 'cause it's showing up in C and python -- is
>>> Fortran a different code base? or does it use the same C lib under hood?
>>>
>>> As for how high priority it is -- that's not up to me -- not a high
>>> enough priority for me to write a PR :-)
>>>
>>> Thanks,
>>>
>>> -CHB
>>>
>>>
>>> On Wed, Oct 2, 2024 at 4:02 PM Roy Mendelssohn - NOAA Federal via
>>> netcdfgroup <netcdfgroup@xxxxxxxxxxxxxxxx> wrote:
>>>
>>>> Actually now that I think of it, since you can read the file a better
>>>> solution is to just create a new file with the proper attributes and then
>>>> item by item copy the values ( i just checked that at least in Python you
>>>> can read the values and they appear to be properly masked). A pain, but
>>>> once written should work for any similar fils from cmem.
>>>>
>>>> -Roy
>>>>
>>>> > On Oct 2, 2024, at 3:52 PM, Roy Mendelssohn - NOAA Federal <
>>>> roy.mendelssohn@xxxxxxxx> wrote:
>>>> >
>>>> > To edit right now I think a hex editor will work. Even in a good
>>>> text editor FillValue shows up, it is just not clear how to reset it
>>>> without seeing the hex.. But a hex editor should be able to get at it .
>>>> >
>>>> > -Roy
>>>> >
>>>> >
>>>> >> On Oct 2, 2024, at 3:31 PM, Dave Allured - NOAA Affiliate via
>>>> netcdfgroup <netcdfgroup@xxxxxxxxxxxxxxxx> wrote:
>>>> >>
>>>> >> Yes in my opinion, this is worth issues in both the netcdf-c and
>>>> netcdf-python libraries, but at low priority. I think this is a relatively
>>>> obscure case. The hope is that if fixed in the libraries, it will avoid
>>>> more debugging efforts by future users.
>>>> >>
>>>> >>
>>>> >> On Wed, Oct 2, 2024 at 3:10 PM Chris Barker <chris.barker@xxxxxxxx>
>>>> wrote:
>>>> >> On Wed, Oct 2, 2024 at 1:44 PM Dave Allured - NOAA Affiliate via
>>>> netcdfgroup <netcdfgroup@xxxxxxxxxxxxxxxx> wrote:
>>>> >> This file was probably written by a pre-2013 version of the netcdf-C
>>>> library,
>>>> >>
>>>> >> Ahh, that may explain it.
>>>> >>
>>>> >> Modern libraries choke on the type mismatch, as seen here.
>>>> >> ...
>>>> >> However IMO, libraries should also support problem-free writes to
>>>> such existing files.
>>>> >>
>>>> >> Is this worth an issue on the netCDF C lib?
>>>> >>
>>>> >> -CHB
>>>> >>
>>>> >>
>>>> >> It may be possible to "fix" files like this, by replacing or
>>>> deleting the offending _FillValue attributes with lower level or capable
>>>> software, such as the HDF5 library.
>>>> >>
>>>> >>
>>>> >> On Wed, Oct 2, 2024 at 1:58 PM Chris Barker via netcdfgroup <
>>>> netcdfgroup@xxxxxxxxxxxxxxxx> wrote:
>>>> >> I was wondering what the NaNf meant :-)
>>>> >>
>>>> >> But yeah, that's probably it -- now to track down whoever wrote this
>>>> file --
>>>> >> But I do wonder what software allowed this?
>>>> >>
>>>> >> -CHB
>>>> >>
>>>> >>
>>>> >> On Wed, Oct 2, 2024 at 12:53 PM Benjamin Root <ben.v.root@xxxxxxxxx>
>>>> wrote:
>>>> >> I think the issue is that the _FillValue of "NaN" is probably a
>>>> float64 while the variable's dtype is float32, which is probably what it
>>>> means by a type mismatch. When I have a NaN FillValue for a float32
>>>> variable, it prints "NaNf"
>>>> >>
>>>> >> ```
>>>> >> float sum_tp_mm(time, latitude, longitude) ;
>>>> >> sum_tp_mm:_FillValue = NaNf ;
>>>> >> ```
>>>> >>
>>>> >> On Wed, Oct 2, 2024 at 12:05 PM Chris Barker via netcdfgroup <
>>>> netcdfgroup@xxxxxxxxxxxxxxxx> wrote:
>>>> >> Thanks Roy, at least it's not just us, and that error did give me a
>>>> hint, look at this from ncdump-ing the header:
>>>> >>
>>>> >> float uo(time, depth, latitude, longitude) ;
>>>> >> string uo:units = "m s-1" ;
>>>> >> uo:_FillValue = NaN ;
>>>> >> string uo:standard_name = "eastward_sea_water_velocity" ;
>>>> >> string uo:long_name = "Eastward velocity" ;
>>>> >>
>>>> >> There's no type for the uo:_FillValue attribute.
>>>> >>
>>>> >> How in the world did that happen?
>>>> >>
>>>> >> At least now I have a specific thing to point the file creators to.
>>>> >>
>>>> >> -CHB
>>>> >>
>>>> >>
>>>> >> On Wed, Oct 2, 2024 at 10:58 AM Roy Mendelssohn - NOAA Federal <
>>>> roy.mendelssohn@xxxxxxxx> wrote:
>>>> >> I can reproduce the error. I tried to copy the file using mccoy,
>>>> got the following:
>>>> >>
>>>> >> nccopy cmems_mod_nws_phy_anfc_0.027deg-3D_P1D-m_1727700325546.nc
>>>> test.nc
>>>> >> NetCDF: Not a valid data type or _FillValue type mismatch
>>>> >> Location: file ?; fcn ? line 1408
>>>> >>
>>>> >> hope this helps.
>>>> >>
>>>> >> -Roy
>>>> >>
>>>> >>
>>>> >>> On Oct 2, 2024, at 10:46 AM, Chris Barker via netcdfgroup <
>>>> netcdfgroup@xxxxxxxxxxxxxxxx> wrote:
>>>> >>>
>>>> >>> I have a user unable to use a netcdf file with our code.
>>>> >>>
>>>> >>> It turns out that the file works just fine when read, but if you
>>>> try to open it for writing, it chokes:
>>>> >>>
>>>> >>> (this is the netCDF4 Python lib, version 1.6.5, which seems to be
>>>> the latest release)
>>>> >>>
>>>> >>> I get:
>>>> >>> In [3]: netCDF4.Dataset('
>>>> cmems_mod_nws_phy_anfc_0.027deg-3D_P1D-m_1727700325546.nc', 'r+')
>>>> >>>
>>>> ---------------------------------------------------------------------------
>>>> >>> OSError Traceback (most recent
>>>> call last)
>>>> >>> Cell In[3], line 1
>>>> >>> ----> 1 netCDF4.Dataset('
>>>> cmems_mod_nws_phy_anfc_0.027deg-3D_P1D-m_1727700325546.nc', 'r+')
>>>> >>>
>>>> >>> File src/netCDF4/_netCDF4.pyx:2469, in
>>>> netCDF4._netCDF4.Dataset.__init__()
>>>> >>>
>>>> >>> File src/netCDF4/_netCDF4.pyx:2028, in
>>>> netCDF4._netCDF4._ensure_nc_success()
>>>> >>>
>>>> >>> OSError: [Errno -103] NetCDF: Can't write file: '
>>>> cmems_mod_nws_phy_anfc_0.027deg-3D_P1D-m_1727700325546.nc'
>>>> >>>
>>>> >>> Any idea how I can get more info as to what's wrong with it?
>>>> >>>
>>>> >>> Again, we have had no problems reading the file.
>>>> >>>
>>>> >>> Thanks,
>>>> >>> - CHB
>>>> >>>
>>>> >>> File is 1.2MB, (not huge by today's standards?) so I've attached it
>>>> here if anyone wants to take a look.
>>>> >>> --
>>>> >>> Christopher Barker, Ph.D.
>>>> >>> Oceanographer
>>>> >>>
>>>> >>> Emergency Response Division
>>>> >>> NOAA/NOS/OR&R (206) 526-6959 voice
>>>> >>> 7600 Sand Point Way NE (206) 526-6329 fax
>>>> >>> Seattle, WA 98115 (206) 526-6317 main reception
>>>> >>>
>>>> >>> Chris.Barker@xxxxxxxx
>>>
>>>
