[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
Re: UNLIMITED long long
- Subject: Re: UNLIMITED long long
- Date: Tue, 29 Jul 2014 21:24:00 -0600
Hi Jean,
For netcdf-3 64-bit-offset format, the number of records can be 2^32 - 1, which is 4,294,967,295 as the maximum size for the UNLIMITED dimension, the largest unsigned integer that can fit in the 32-bit slot for number of records in the classic or 64-bit-offset format.
I'm not sure why you're getting a segmentation fault. I've attached a C program that creates a file with 3,900,000,000 records. The program finishes and prints a line indicating success, which you can check using "ncdump -h" on the file it creates. If you run this program and it gets a segmentation fault, then maybe your file system is not configured for large files. See the answer to the FAQ
Why do I get an error message when I try to create a file larger than 2 GiB with the new library?
If you change the format to netCDF-4 classic model by changing "NC_64BIT_OFFSET" to "NC_NETCDF4 | NC_CLASSIC_MODEL" in the nc_create() call and also increase NUM_RECS, you can run the same program to create files with many more records (2^63 - 1 ?), probably enough to fill up your file system. This is because the HDF5 library uses a 64-bit type for unlimited-dimension sizes.
--Russ
/* Demonstrate writing a netCDF file with lots of records */
#include <stdlib.h>
#include <stdio.h>
#include <string.h> /* for memset() */
#include <netcdf.h>
/* This is the name of the data file we will create. */
#define FILE_NAME "large-unlim.nc"
#define NDIMS 2
#define FILL_SIZE 10000000
#define NUM_RECS 3900000000 /* 3,900,000,000 records */
/* Handle errors by printing an error message and exiting with a
* non-zero status. */
#define ERRCODE 2
#define ERR(e) {printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}
int
main()
{
size_t i;
int ncid, u_dimid, varid;
int dimids[NDIMS];
size_t start, count;
char *data;
int retval;
data = malloc(FILL_SIZE * sizeof(char));
memset(data,42,FILL_SIZE * sizeof(char));
/* Create the file. */
if ((retval = nc_create(FILE_NAME, NC_64BIT_OFFSET, &ncid)))
ERR(retval);
/* Define the UNLIMITED dimension and an associated variable. */
if ((retval = nc_def_dim(ncid, "unlim", NC_UNLIMITED, &u_dimid)))
ERR(retval);
dimids[0] = u_dimid;
if ((retval = nc_def_var(ncid, "bigvar", NC_BYTE, 1,
dimids, &varid)))
ERR(retval);
if ((retval = nc_enddef(ncid)))
ERR(retval);
/* Write data to the file in batches of FILL_SIZE records at a
time. */
for (i = 0; i < NUM_RECS; i += FILL_SIZE) {
start = i;
count = FILL_SIZE;
printf("writing %d records starting at %ld\n", FILL_SIZE, i);
if ((retval = nc_put_vara(ncid, varid, &start, &count, data)))
ERR(retval);
}
if ((retval = nc_close(ncid)))
ERR(retval);
printf("*** SUCCESS writing %ld records in example file %s!\n", i,
FILE_NAME);
return 0;
}
