Folks,
I am having trouble writing a Classic (actually 64-bit offset) file with
a single variable > 4GB. According to what I read on the netCDF Web
page this should be legal. This is from the FAQ section:
***************************************
Howevever, for the classic and 64-bit offset formats there are still
limits on sizes of netCDF objects. Each fixed-size variable (except the
last, when there are no record variables) and the data for one record's
worth of a single record variable (except the last) are limited in size
to a little less that 4 GiB, which is twice the size limit in versions
earlier than netCDF 3.6.
It is also possible to overcome the 4 GiB variable restriction for a
single fixed size variable, when there are no record variables, by
making it the last variable, as explained in the example in NetCDF
Classic Format Limitations.
***************************************
This is from the section on NetCDF Classic Format Limitations.
***************************************
If you don't use the unlimited dimension, only one variable can exceed 2
GiB in size, but it can be as large as the underlying file system
permits. It must be the last variable in the dataset, and the offset to
the beginning of this variable must be less than about 2 GiB.
The limit is really 2^31 - 4. If you were to specify a variable size of
2^31 -3, for example, it would be rounded up to the nearest multiple of
4 bytes, which would be 2^31, which is larger than the largest signed
integer, 2^31 - 1.
For example, the structure of the data might be something like:
netcdf bigfile1 {
dimensions:
x=2000;
y=5000;
z=10000;
variables:
double x(x); // coordinate variables
double y(y);
double z(z);
double var(x, y, z); // 800 Gbytes
}
***************************************
All of this implies that I should be able to create a netCDF file with a
single variable > 4GB as long as it is the last variable in the file.
However, when I try to create such a file, I get the following error in
my program:
baja:~>./netCDF_test1
Allocating memory ...
Setting array values ...
Creating netCDF file ...
netCDF_test1: ncx.c:1810: ncx_put_size_t: Assertion `*ulp <=
4294967295U' failed.
Abort (core dumped)
I have appended my test program after this message. The test program
works if count[2] is 2000, so the variable is just under 4GB. But if
count[2] is 3000 then I get the above error.
These tests were done with netCDF 3.6.2, because I am trying to solve a
similar problem in IDL, and that is the version that IDL 7.1.1 is built
with.
Am I doing something wrong, or is this a bug?
Thanks,
Mark Rivers
****************************
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <netcdf.h>
/* Handle errors by printing an error message and exiting with a
* non-zero status. */
#define ERR(e) {printf("error=%s\n", nc_strerror(e)); \
return(e);}
#define MAX_DIMENSIONS 3
int main(int argc, char **argv)
{
int ncId, dataId;
int dimIds[MAX_DIMENSIONS];
int ncType;
short *data;
size_t arraySize;
size_t i;
size_t start[MAX_DIMENSIONS], count[MAX_DIMENSIONS];
char *fileName = "netCDF_test1.nc";
int retval;
start[0] = start[1] = start[2] = 0;
count[0] = 1024;
count[1] = 1024;
count[2] = 3000;
arraySize = (long)count[0]*count[1]*count[2];
printf("Allocating memory ...\n");
data = (short *)calloc(arraySize, sizeof(short));
printf("Setting array values ...\n");
for (i=0; i<arraySize-1; i++) data[i] = i;
printf("Creating netCDF file ...\n");
/* Create the file. The NC_CLOBBER parameter tells netCDF to
* overwrite this file, if it already exists.*/
if ((retval = nc_create(fileName, NC_CLOBBER | NC_64BIT_OFFSET,
&ncId)))
ERR(retval);
/* Define the dimensions. NetCDF will hand back an ID for each. */
if ((retval = nc_def_dim(ncId, "Dim0", count[0], &dimIds[0])))
ERR(retval);
if ((retval = nc_def_dim(ncId, "Dim1", count[1], &dimIds[1])))
ERR(retval);
if ((retval = nc_def_dim(ncId, "Dim2", count[2], &dimIds[2])))
ERR(retval);
ncType = NC_SHORT;
/* Define the array data variable. */
if ((retval = nc_def_var(ncId, "array_data", ncType, MAX_DIMENSIONS,
dimIds, &dataId)))
ERR(retval);
/* End define mode. This tells netCDF we are done defining
* metadata. */
if ((retval = nc_enddef(ncId)))
ERR(retval);
printf("Writing data to disk ...\n");
if ((retval = nc_put_vara_short(ncId, dataId, start, count, data)))
ERR(retval);
printf("Closing netCDF file ...\n");
if ((retval = nc_close(ncId)))
ERR(retval);
return 0;
}
