Hi Sylvain,
thanks very much for the thoughts! What is odd is that I agree with your
observation that the nc_get_var_double() is acting exactly as you say --
reading a float into a double, yielding incorrect results -- *but*
according to the documentation the function call is supposed to convert the
read-in data into double. E.g., from the netcdf man page that comes with
the source tarball:
int nc_get_var_double(int ncid, int varid, double in[]):
Reads an entire netCDF variable (i.e. all the values).
The
netCDF dataset must be open and in data mode. The data is
con-
verted from the external type of the specified variable, if
nec-
essary, to the type specified in the function name. If
conver-
sion is not possible, an NC_ERANGE error is returned.
So it seems to unambiguously say that the data should be converted to
double using this call, and that kind of error should not happen. Hence my
confusion.
Regards,
--Dave
On Tue, Sep 8, 2020 at 5:47 PM Sylvain Herlédan <
sylvain.herledan@xxxxxxxxxxxxxxxx> wrote:
> Hi David,
>
> I think you must use the nc_get_var_*() function matching the storage type
> of the variable, these methods are not meant to be used for type
> conversions.
>
> Your NetCDF file states that the data type for the phycocyanin variable is
> float, so you should use nc_get_var_float to read its values.
>
> On my system float values are stored on 32 bits whereas double values are
> stored on 64 bits.
>
> When you ask the library to read the content of the variable as double, it
> loads 64 bits from the file and interpret them as a double value. Since
> phycocyanin values are stored on 32 bits, it means that you actually read
> two float values from the file but interpret their binary representation as
> a single double value.
>
> It explains the strange values you get with nc_get_var_double:
> - the variable only contains fill values, i.e. -9999.f
> - on my machine the hexadecimal representation for -9999.f is 0xc61c3c00
> - if you read two adjacent floats with a -9999.f value, you get 0x
> c61c3c00c61c3c00.
> - if you interpret these 64 bits as a double value you get:
> -559239646634519513659653226496.000000
> In your test program if you print more values of ddata you will also see
> that you get the strange value for the 720 first items in the array, and
> then starting from the 721st you only get 0: since you actually need two
> phycocyanin values for each 64bits item of the array, you only have enough
> data to fill the first half of ddata.
>
> Cheers,
>
> Sylvain
>
> On 09/09/2020 00:30, David Pierce via netcdfgroup wrote:
>
> Hello netcdf-ers,
>
> I had a user of the R ncdf4 package alert me to an odd and perplexing
> apparent bug in the netcdf library. It is triggered by the netcdf file you
> can download here:
>
> http://cirrus.ucsd.edu/~pierce/tmp/mendota_buoy.2018-11-08.nc
>
> This file has the following variable (among others):
>
> float phycocyanin(time) ;
> phycocyanin:_FillValue = -9999.f ;
> phycocyanin:units = "RFU" ;
> phycocyanin:long_name = "Phycocyanin" ;
> phycocyanin:_Storage = "chunked" ;
> phycocyanin:_ChunkSizes = 1440 ;
> phycocyanin:_DeflateLevel = 4 ;
> phycocyanin:_Shuffle = "true" ;
> phycocyanin:_Endianness = "little" ;
>
> 'ncdump' reports all the data as missing:
>
> phycocyanin = _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _,
> _,
> _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _,
> (etc.)
>
> When I read the data into a floating point array using the C library using
> nc_get_var_float() then I get the expected values of -9999.0. However when
> I read it into a double precision array with nc_get_var_double() then I get
> strange values. Here is the output of the little test program that I've
> appended below:
>
> Sizeof float: 4 double: 8
> ======== DOUBLE ========
> -559239646634519513659653226496.000000
> -559239646634519513659653226496.000000
> -559239646634519513659653226496.000000
> -559239646634519513659653226496.000000
> -559239646634519513659653226496.000000
> -559239646634519513659653226496.000000
> -559239646634519513659653226496.000000
> -559239646634519513659653226496.000000
> -559239646634519513659653226496.000000
> -559239646634519513659653226496.000000
> ======== FLOAT ========
> -9999.000000 -9999.000000 -9999.000000 -9999.000000 -9999.000000
> -9999.000000 -9999.000000 -9999.000000 -9999.000000 -9999.000000
>
> This seems like a bug, unless I'm overlooking something obvious in the
> test code (always possible). The nc_get_var_double() call is supposed to
> convert the netcdf file's values into double precision and store them in
> the provided double precision array, right? That's my understanding anyway.
> I can't see why they are not -9999.00's in the resultant array.
>
> Any thoughts are appreciated,
>
> --Dave
>
> Test program:
>
> #include <stdio.h>
> #include <stdlib.h>
> #include "netcdf.h"
>
> void main( int argc, char *argv[] )
> {
> int err, ncid, varid;
> size_t nt;
> double *ddat;
> float *fdat;
>
> nt = 1440; /* just hardcode for test */
>
> printf( "Sizeof float: %ld double: %ld\n", sizeof(float),
> sizeof(double) );
>
> err = nc_open("mendota_buoy.2018-11-08.nc
> <https://urldefense.com/v3/__http://mendota_buoy.2018-11-08.nc__;!!Mih3wA!TzxceiDbm9bJ2PTB0rzFvI7Byrs9UxiWq2RbLDodthnzVgeMLwAghucDQsiNSho$>",
> 0, &ncid );
> if( err != 0 ) {
> printf( "err open = %d\n", err );
> exit(-1);
> }
>
> err = nc_inq_varid( ncid, "phycocyanin", &varid );
> if( err != 0 ) {
> printf( "err inq_varid = %d\n", err );
> exit(-1);
> }
>
> /* Make room for both double and floating dat */
> ddat = (double *)malloc( sizeof(double) * nt );
> fdat = (float *)malloc( sizeof(float ) * nt );
>
> /* Read into double array. Supposed to convert to double...? */
> err = nc_get_var_double( ncid, varid, ddat );
> if( err != 0 ) {
> printf( "err get_var_double = %d\n", err );
> exit(-1);
> }
>
> printf( "======== DOUBLE ========\n" );
> for( int ii=0; ii<10; ii++ )
> printf( "%lf ", ddat[ii] );
>
> printf( "\n" );
>
> /* === Do same thing, but for float === */
>
> err = nc_get_var_float( ncid, varid, fdat );
> if( err != 0 ) {
> printf( "err get_var_float = %d\n", err );
> exit(-1);
> }
>
> printf( "======== FLOAT ========\n" );
> for( int ii=0; ii<10; ii++ )
> printf( "%f ", fdat[ii] );
>
> printf( "\n" );
>
> }
>
>
> -------------------------------------------------------------------
> David W. Pierce
> Division of Climate, Atmospheric Science, and Physical Oceanography
> Scripps Institution of Oceanography
> (858) 534-8276 (voice) / (858) 534-8561 (fax) dpierce@xxxxxxxx
> -------------------------------------------------------------------
>
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> netcdfgroup mailing listnetcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
> <https://urldefense.com/v3/__https://www.unidata.ucar.edu/mailing_lists/__;!!Mih3wA!TzxceiDbm9bJ2PTB0rzFvI7Byrs9UxiWq2RbLDodthnzVgeMLwAghucD0oOpb3g$>
>
>
--
-------------------------------------------------------------------
David W. Pierce
Division of Climate, Atmospheric Science, and Physical Oceanography
Scripps Institution of Oceanography
(858) 534-8276 (voice) / (858) 534-8561 (fax) dpierce@xxxxxxxx
-------------------------------------------------------------------
