Sergei,
Thanks a lot for this information, I was really running out of ideas here. I'll
try to install the latest (4.1.3) packages from Debian experimental and see how
it goes.
Regards,
Thorsten
-----Original Message-----
From: Shibaev, Sergei [mailto:Sergei.Shibaev@xxxxxxxxxx]
Sent: Montag, 6. Februar 2012 15:43
To: Schmidt Thorsten; netcdfgroup@xxxxxxxxxxxxxxxx
Subject: RE: [netcdfgroup] NetCDF uses ridiculous amounts of memory
Hi Thorsten,
NetCDF API version 4.1.1 has a bug which causes huge memory consumption.
It was corrected in version 4.1.2.
Please have a look at my benchmark on complex file writing (attached).
Regards,
Sergei
-----Original Message-----
From: netcdfgroup-bounces@xxxxxxxxxxxxxxxx
[mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Schmidt Thorsten
Sent: 06 February 2012 14:28
To: netcdfgroup@xxxxxxxxxxxxxxxx
Subject: [netcdfgroup] NetCDF uses ridiculous amounts of memory
Hello NetCDF-Group,
We are currently evaluating NetCDF as a format to store large amounts of
timeseries data. Since we need microsecond resolution for the time, I decided
to use the compound variable functionality of NetCDF4 to create the equivalent
of the standard Linux timeval data type. To get a feeling of the use and
performance of NetCDF, I wrote a small test program to create a sample nc file
with a selectable number of parameters and a selectable number of time points
(via the NUMVARS and DATALENGTH defines. When I run the program with very small
values like ten, there is no problem. The program completes quickly and the nc
file that is created can be viewed with ncdump. It contains what I would expect
it to contain (from the documentation). However, if I increase the number of
variables and/or data points to more realistic values (hundreds of variables,
thousands of data points), the program's memory footprint keeps growing and
growing until it uses up essentially all of the machine's memory (this is on a
16GiB machine!), resulting in very poor performance (with NUMVARS=100 and
DATLENGTH=100000, it grows to ca.
16GiB in the first three minutes, which is all the RAM I have. After that the
swapping starts and performance is really poor.). Looking at the memory
footprint with pmap, the large chunk of memory used is [ anon ], i.e.
dynamically allocated by malloc & co. I am using the netcdf/hdf packets from
the current Debian stable distribution (which are netCDF
4.1.1 and HDF5 1.8.4). I may be doing something really stupid here, but I am
running out of ideas what else to try. Any help is appreciated.
Regards, Thorsten
PS: Below is the test code I used. On my Debian box it compiles with "gcc
netcdftest.c -lnetcdf -Wall"
#include <stdio.h>
#include <netcdf.h>
#include <stdlib.h>
#include <math.h>
#include <time.h>
#include <string.h>
#include <unistd.h>
#include <sys/time.h>
#define ERRCODE 2
#define ERR(e) {printf ("Error: %s\n", nc_strerror(e)); exit (ERRCODE);}
#define NUMVARS 100
#define DATALENGTH 100000
struct nc_timeval {
int64_t tv_sec; /* seconds */
int64_t tv_usec; /* microseconds */
};
struct pilVariable {
char name[64];
char unit[64];
int id;
};
struct nc_timeval mknctime (struct timeval mytimeofday) {
struct nc_timeval mynctime;
mynctime.tv_sec = (int64_t) mytimeofday.tv_sec;
mynctime.tv_usec = (int64_t) mytimeofday.tv_usec;
return(mynctime);
}
int main (void) {
int retval;
int ncid;
int ncvarid;
int ncdimid;
int nctimeid;
int i,k;
struct timeval mytimeofday;
struct nc_timeval mynctime;
struct pilVariable myVars[NUMVARS];
static size_t mystart = 1;
double value;
/* Create netCDF dataset, enter define mode */
retval = nc_create
("myNetCDF.nc",NC_NOCLOBBER|NC_NETCDF4,&ncid);
if (retval != NC_NOERR) {
ERR(retval);
}
/* Define compound type(s) */
retval = nc_def_compound (ncid,sizeof(struct
nc_timeval),"timeval",&ncvarid);
if (retval != NC_NOERR) {
ERR(retval);
}
retval = nc_insert_compound (ncid,ncvarid,"tv_sec",0,NC_INT64);
if (retval != NC_NOERR) {
ERR(retval);
}
retval = nc_insert_compound
(ncid,ncvarid,"tv_usec",sizeof(int64_t),NC_INT64);
if (retval != NC_NOERR) {
ERR(retval);
}
/* define dimensions: from name and length */
retval = nc_def_dim (ncid,"Time",NC_UNLIMITED,&ncdimid);
if (retval != NC_NOERR) {
ERR(retval);
}
/* define variables: from name, type, ... */
/* Time is special, all others are double */
retval = nc_def_var (ncid,"Time",ncvarid,1,&ncdimid,&nctimeid);
if (retval != NC_NOERR) {
ERR(retval);
}
retval = nc_put_att_text (ncid,nctimeid,"Unit",strlen("seconds
since epoch, microseconds"),"seconds since epoch, microseconds");
if (retval != NC_NOERR) {
ERR(retval);
}
for (i=0;i<NUMVARS;i++) {
snprintf(myVars[i].name,64,"Variable_%d",i);
snprintf(myVars[i].unit,64,"Unit_%d",i);
/* Create Variable */
retval = nc_def_var
(ncid,myVars[i].name,NC_DOUBLE,1,&ncdimid,&myVars[i].id);
if (retval != NC_NOERR) {
ERR(retval);
}
/* Create Unit Attribute */
retval = nc_put_att_text
(ncid,myVars[i].id,"Unit",strlen(myVars[i].unit),myVars[i].unit);
if (retval != NC_NOERR) {
ERR(retval);
}
}
/* put attribute: assign attribute values */
/* end definitions: leave define mode */
retval = nc_enddef (ncid);
if (retval != NC_NOERR) {
ERR(retval);
}
/* provide values for variables */
/* Time */
for (i=0;i<DATALENGTH;i++) {
gettimeofday(&mytimeofday,0);
mynctime = mknctime(mytimeofday);
mystart = (size_t) i;
retval = nc_put_var1 (ncid,nctimeid,&mystart,&mynctime);
if (retval != NC_NOERR) {
ERR(retval);
}
for (k=0;k<NUMVARS;k++) {
value = (double)i;
retval = nc_put_var1_double
(ncid,myVars[k].id,&mystart,&value);
if (retval != NC_NOERR) {
ERR(retval);
}
}
// nc_sync(ncid);
}
/* close: save new netCDF dataset */
retval = nc_close (ncid);
if (retval != NC_NOERR) {
ERR(retval);
}
return(0);
}
__________________________________________________
Pilatus Aircraft Ltd.
Thorsten Schmidt
Flight Test Engineer
P.O. Box 992
6371 Stans, Switzerland
phone: +41 41 619 68 06
fax: +41 41 619 65 99
e-mail: thorsten.schmidt@xxxxxxxxxxxxxxxxxxxx
website: www.pilatus-aircraft.com
Please consider the environment before printing this email!
***DISCLAIMER***
The information and any attachments (herein referred to as 'document')
transmitted is intended only for the person or entity to which it is addressed.
It may contain confidential and/or privileged information. If you are not the
intended recipient of this document, you are hereby notified that any
dissemination, distribution, copying, other use of, or taking any action in
reliance upon this document by persons or entities other than the intended
recipient is illegal and prohibited. If you have received this in error, please
immediately notify the sender, permanently delete the original from any
computer and/or system, and destroy any printout of the 'document'. We thank
you for your co-operation in this matter. Pilatus Aircraft Ltd.
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
Click
https://www.mailcontrol.com/sr/GPTh4CNjOSbTndxI!oX7UtHdKBF8nr1DniOBWZWCo
mnO8xQEDcASAE5Mk0tjZefnf37CtQoX9FkNL52V0szYFQ== to report this email as spam.
Please consider the environment before printing this email!
***DISCLAIMER***
The information and any attachments (herein referred to as 'document')
transmitted is intended only for the person or entity to which it is addressed.
It may contain confidential and/or privileged information. If you are not the
intended recipient of this document, you are hereby notified that any
dissemination, distribution, copying, other use of, or taking any action in
reliance upon this document by persons or entities other than the intended
recipient is illegal and prohibited. If you have received this in error, please
immediately notify the sender, permanently delete the original from any
computer and/or system, and destroy any printout of the 'document'. We thank
you for your co-operation in this matter. Pilatus Aircraft Ltd.
