Re: [netcdfgroup] NetCDF-3 file format changes


Hi,

Dave Allured wrote:
If you don't mind me asking, have you worked with large Netcdf-3 files with variables > 4GB, types NC_FLOAT and NC_INT only (i.e. 32 bit types)?

Yes, I have several large files with > 4GB.

Is the current Netcdf 3.6.2 software reliable with these in your experience?

It is reliable if you stick to very strict constraints. I would actually
rather say 'It works, but the constraints are not easy to enforce'.
 - only one variable > 4GB per file, and this must be the last variable
   in the file.
 - no attributes stored for variables > 2GB.
 - (does not apply to you if you only want 32bit types): use my fix from
   the last mail then nc_byte and nc_short will work too.

(I do not recall precisely if the above limits are 2 or 4GB, have to look
it up tomorrow)

Cheers,

   Mario

It has seemed reliable for me, but I must confess my usage of files greater than 4GB has been very limited so far. Thank you for any feedback.

Dave Allured
CU/CIRES Climate Diagnostics Center (CDC)
http://cires.colorado.edu/science/centers/cdc/
NOAA/ESRL/PSD, Climate Analysis Branch (CAB)
http://www.cdc.noaa.gov/

Mario Emmenlauer wrote:
Hi,

Roy Mendelssohn wrote:
See http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#Large%20File%20Support0. It has been possible since netcdf3.6
with severe bugs, as posted by me to this mailing list on 09/12/2007
(i.e. [netcdfgroup] possible bug in 3.6.2 with variables > 4GB).
For variable types NC_BYTE, NC_CHAR and NC_SHORT it crashes badly.

I have actually never received an answer to my bug report, don't know
why nobody cared :-/

BTW: below is an attached copy of the old mail

Cheers,

    Mario

----
I hope this bug hasn't been filed before, I couldn't find it
on the list via a quick check.

This concerns writing large (>4GB) variables in netcdf with
large-file-support. In netcdf-3.6.2/libsrc/var.c around line
400 it reads:

if( varp->xsz <= X_UINT_MAX / product )
   /* if integer multiply will not overflow */
{
   varp->len = product * varp->xsz;
} else {
   /* OK for last var to be "too big", indicated by this special len */
   varp->len = X_UINT_MAX;
}
switch(varp->type) {
   case NC_BYTE :
   case NC_CHAR :
   case NC_SHORT :
     if( varp->len%4 != 0 )
     {
       varp->len += 4 - varp->len%4; /* round up */
       /* *dsp += 4 - *dsp%4; */
     }
   break;
   default:
   /* already aligned */
     break;
}

In the case of NC_BYTE, NC_CHAR and NC_SHORT, varp->len will end
up being X_UINT_MAX+1 instead of X_UINT_MAX. This in turn causes
an assertion when calling ncx_put_size_t later:
ncx.c:1812: ncx_put_size_t: Assertion `*ulp <= 4294967295U' failed.

I could not think about a useful fix despite qualifying the
rounding with the product-overflow (just moved the else-part
further down):

if( varp->xsz <= X_UINT_MAX / product )
   /* if integer multiply will not overflow */
{
   varp->len = product * varp->xsz;
   switch(varp->type) {
     case NC_BYTE :
     case NC_CHAR :
     case NC_SHORT :
       if( varp->len%4 != 0 )
       {
         varp->len += 4 - varp->len%4; /* round up */
         /* *dsp += 4 - *dsp%4; */
       }
     break;
     default:
     /* already aligned */
       break;
   }
} else {
   /* OK for last var to be "too big", indicated by this special len */
   varp->len = X_UINT_MAX;
}

I hope this bug report is useful. If you can send me a better
patch against netcdf-3.6.2 I would highly appreciate it.

Cheers,

   Mario Emmenlauer
----


On Feb 26, 2008, at 1:11 PM, Joe Sirott wrote:

Hi,

In the "classic" netCDF file format (netCDF-3), a variable without a record dimension cannot be larger than 2GB. This limitation has been giving me a lot of headaches lately. I know that netCDF-4 is supposed to solve this problem, but there are a number of reasons why netCDF-4 is not a good option for me (no Java write support, for one).

I could be missing something, but it seems like a very small change to the netCDF-3 file format would solve this problem. The only requirement would be changing the variable size info in the netCDF header from a 32 bit to a 64 bit int (and, of course, updating the version info in the header).

I'm guessing that I'm not the only netCDF user who has run into this problem and who is also reluctant to move to netCDF-4. Any possibility that Unidata could make these changes?

Cheers,

Joe S.


_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx <mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/
**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx <mailto:Roy.Mendelssohn@xxxxxxxx> (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."




------------------------------------------------------------------------

_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/

_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/


  • 2008 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: