Hi Christoph,
Thanks very much, that indeed fixed the problem! When I first tried
your lines below I got compiler errors, but then I looked at your
version of config.h and saw that you also need to #include io.h and
process.h before the #defines. Thus my config.h now contains:
/* The size of `off_t', as computed by sizeof. */
#if defined(vxWorks)
#define SIZEOF_OFF_T 4
#elif defined(_WIN32)
#include<io.h>
#include<process.h>
#define SIZEOF_OFF_T 8
#define lseek _lseeki64
#define off_t __int64
#define _off_t __int64
#define _OFF_T_DEFINED
#else
#define SIZEOF_OFF_T 8
#endif
Thanks again for the help!
It's interesting that netCDF 3.6.3 contains a config.h in the win32/
directory, but it does not contain the above fixes, so it does not work
for creating files>2GB.
I also noted that your config.h file contains the following
/* I added the following to this config.h file by hand, after being
abducted by
aliens last week in Kansas. (All Hail Zorlock, Mighty Destroyer of
Worlds!) */
#include<io.h>
#include<process.h>
#define lseek _lseeki64
#define off_t __int64
#define _off_t __int64
#define _OFF_T_DEFINED
#define snprintf sprintf_s
#define strcasecmp _stricmp
So did the aliens teach you this trick, or where did you learn it?
Cheers,
Mark
-----Original Message-----
From: netcdfgroup-bounces@xxxxxxxxxxxxxxxx
[mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Christoph
Gohlke
Sent: Tuesday, July 05, 2011 5:46 PM
To: netcdfgroup@xxxxxxxxxxxxxxxx
Subject: Re: [netcdfgroup] Problem with netCDF "classic" files larger
than 2GB on Windows
Try to compile with 64 bit off_t and lseek. E.g. in config.h
#define SIZEOF_OFF_T 8
#define lseek _lseeki64
#define off_t __int64
#define _off_t __int64
#define _OFF_T_DEFINED
Your test_big_classic.c works for me. The config.h I'm using with msvc9
is attached.
Christoph
On 7/5/2011 3:13 PM, Mark Rivers wrote:
Hi Russ and Ed,
Thanks for your replies.
I have now upgraded from netCDF 3.6.3 to 4.1.3 as you suggested.
I have reproduced the problem in a simple test program, which I have
attached. The test program creates a netCDF file of the following
format
as revealed by ncdump:
corvette:areaDetector/ADApp/netCDFSrc>ncdump -k
/home/epics/scratch/test.nc
classic
corvette:areaDetector/ADApp/netCDFSrc>ncdump -h
/home/epics/scratch/test.nc
netcdf test {
dimensions:
numArrays = UNLIMITED ; // (4100 currently)
YSize = 1024 ;
XSize = 1024 ;
variables:
byte array_data(numArrays, YSize, XSize) ;
}
So it writes a 1024x1024 byte array to the file 4100 times, where 4100
is the unlimited dimension. This tests writing files that are larger
than both the 2GB and 4GB potential limits.
The test program runs fine on Linux and Cygwin.
However, when compiled with the Microsoft Visual Studio 2008 compiler,
it fails with exactly the same error I got in netCDF 3.6.3, except it
is
now in line 342 rather than 325 of posixio.c.
writing record 2047/4100
writing record 2048/4100
Assertion failed: pxp->bf_offset<= offset&& offset< pxp->bf_offset
+
(off_t) pxp->bf_extent, file
..\..\..\ADApp\netCDFSrc\libsrc\posixio.c,
line 342
By way of background on why I need to use the Microsoft compiler:
I run a fairly large NSF-funded user facility at the Advanced Photon
Source at Argonne National Laboratory. We collect data using a wide
variety of x-ray and other detectors under a distributed real-time
control system called EPICS. Some of these detectors can be controlled
from Linux and Cygwin, but many of them have vendor-supplied drivers
that can only be called from code compiled with the Microsoft
compiler.
I also need to build my application to run on the vxWorks and RTEMS
real-time operating systems, which we use to control specialized VME
hardware. These use the gcc compiler, but cross-compiling for the
real-time target on a Linux host.
I am building with gnumake, not the Visual Studio application
development environment on all platforms, including when building with
the Microsoft VC++ compiler on Windows.
I have also attached a copy of the config.h file I am using, which has
been modified from the version built by "configure" on Linux to do
things differently if _WIN32 or vxWorks are defined.
I had to make some minor changes to the 4.1.3 source code in order to
allow it to build with the Microsoft VS 2008 compiler, and to build on
vxWorks. I will send those changes in a separate e-mail.
I have also attached an IDL version of the same program. It should be
completely equivalent to the C program. It works fine on both Linux
and
on Windows. It appears to me that the idl_netcdf.dll file that ITT
provides with IDL on Windows is built with the Microsoft compiler
(judging from dumpbin /imports), but I am not certain of this. If so,
it
indicates that it is possible to write netCDF classic files when
compiling with Visual Studio, and I am probably just doing something
wrong.
Perhaps it is in my flags to the compiler. Here is the output when I
compile posixio.c as an example of the flags I am using:
J:\epics\devel\areaDetector\ADApp\netCDFSrc>make install.win32-x86
make -C O.win32-x86 -f ../Makefile TOP=../../.. T_A=win32-x86 install
make[1]: Entering directory
`J:/epics/devel/areaDetector/ADApp/netCDFSrc/O.win32-x86'
cl -c /nologo /D__STDC__=0 /D_CRT_SECURE_NO_DEPRECATE
/D_CRT_NONSTDC_NO_DEPRECATE /Ox /GL /W3 /w44355 /D_WIN32_WINNT=0x0503
-DHAVE_CONFIG_H /MT -DEPICS_DLL_NO - I. -I..\\O.Common - I. -I..
-I..\\..\\..\\ADApp\\netCDFSrc\\include
-I..\\..\\..\\ADApp\\netCDFSrc\\libsrc
-I..\\..\\..\\ADApp\\netCDFSrc\\libdispatch
-I..\\..\\..\\ADApp\\netCDFSrc\\liblib
-I..\\..\\..\\include\\os\\WIN32
-I..\\..\\..\\include -IJ:\\epics\\devel\\asyn-4-17\\include
-IJ:\\epics\\devel\\calc-2-8\\include
-IJ:\\epics\\devel\\busy-1-3\\include
-IJ:\\epics\\devel\\sscan-2-6-6\\include
-IJ:\\epics\\devel\\mca-6-12-4\\include
-IJ:\\epics\\devel\\autosave-4-7\\include\\os\\WIN32
-IJ:\\epics\\devel\\autosave-4-7\\include
-IJ:\\epics\\devel\\areaDetector-1-7beta1\\include\\os\\WIN32
-IJ:\\epics\\devel\\areaDetector-1-7beta1\\include
-IH:\\epics\\base-3.14.12.1\\include\\os\\WIN32
-IH:\\epics\\base-3.14.12.1\\include
-I..\\..\\..\\ADApp\\netCDFSrc\\include
..\\..\\..\\ADApp\\netCDFSrc\\libsrc\\posixio.c
posixio.c
..\..\..\ADApp\netCDFSrc\libsrc\posixio.c(433) : warning C4018: '<' :
signed/unsigned mismatch
..\..\..\ADApp\netCDFSrc\libsrc\posixio.c(441) : warning C4018: '<=' :
signed/unsigned mismatch
..\..\..\ADApp\netCDFSrc\libsrc\posixio.c(449) : warning C4018: '<=' :
signed/unsigned mismatch
..\..\..\ADApp\netCDFSrc\libsrc\posixio.c(454) : warning C4018: '>' :
signed/unsigned mismatch
Thanks,
Mark
-----Original Message-----
From: Russ Rew [mailto:russ@xxxxxxxxxxxxxxxx]
Sent: Wednesday, June 29, 2011 1:34 PM
To: Mark Rivers
Cc: netcdfgroup@xxxxxxxxxxxxxxxx
Subject: Re: [netcdfgroup] Problem with netCDF "classic" files larger
than 2GB on Windows
Hi Mark,
I am having a problem writing netCDF "classic" files larger than 2GB
when building with the Microsoft Visual Studio 2008 compiler. The
same
code works fine when build on Linux, and on Cygwin.
A bit of background. This is a project to build generic file writers
for cameras and detectors in the areaDetector package
(http://cars9.uchicago.edu/software/epics/areaDetectorDoc.html) for
the
EPICS (http://www.aps.anl.gov/epics/) real-time control system. This
is
a large project, with its own build system based on gnumake. I am
building the basic netCDF library from the same source code on all
supported platforms (Linux 32 and 64-bit, Windows 32 and 64-bit with
Microsoft compiler, Windows with Cygwin gcc compilet, vxWorks,
Darwin ,
Solaris, etc.). Because I have another file writer that handles
HDF5, I
am using netCDF 3.6.3, since I only want to netCDF to create
"classic"
files. I am using 3.6.3 because it is less complex than 4.x, not
requiring any HDF5 support, etc.
I recommend you use the latest netCDF-4 release, 4.1.3. If you don't
have and HDF5 library installed, it will be built without support for
netCDF-4, but with bug fixes since 3.6.3 was released 3 years ago. For
example, these bug fixes were mentioned in the RELEASE_NOTES for
4.1.2:
Fixed two large-file bugs with using classic format or
64-bit offset format and accessing multidimensional
variables with more than 2**32 values.
If the problem you're seeing still occurs in releases since 4.1.2, it
may be something new.
--Russ
The application is typically streaming uncompressed images, using
the
UNLIMITED dimension as the streaming dimension. Thus, each record is
small, only a few MB, and the file size limitations of the classic
format are not a problem. Here is an ncdump of a file header created
with this file writer on Linux:
corvette:ADApp/op/adl>ncdump -h /home/epics/scratch/netcdf_test_1.nc
netcdf netcdf_test_1 {
dimensions:
numArrays =3D UNLIMITED ; // (4100 currently)
dim0 =3D 1024 ;
dim1 =3D 1024 ;
attrStringSize =3D 256 ;
variables:
int uniqueId(numArrays) ;
double timeStamp(numArrays) ;
byte array_data(numArrays, dim0, dim1) ;
int Attr_ColorMode(numArrays) ;
double Attr_AcquireTime(numArrays) ;
double Attr_RingCurrent(numArrays) ;
char Attr_RingCurrent_EGU(numArrays, attrStringSize) ;
double Attr_ID_Energy(numArrays) ;
char Attr_ID_Energy_EGU(numArrays, attrStringSize) ;
int Attr_ImageCounter(numArrays) ;
int Attr_MaxSizeX(numArrays) ;
int Attr_MaxSizeY(numArrays) ;
char Attr_CameraModel(numArrays, attrStringSize) ;
char Attr_CameraManufacturer(numArrays, attrStringSize) ;
// global attributes:
:dataType =3D 1 ;
:NDNetCDFFileVersion =3D 3. ;
:numArrayDims =3D 2 ;
:dimSize =3D 1024, 1024 ;
...
So the only large array is called "array_data", and in this case it
is
[4100, 1024, 1024], where 4100 is the unlimited dimension. Thus,
this
file is over 4GB, and it can be written and read with no problems on
Linux and Cygwin. It also works fine when writing files on Windows
with
the Microsoft compiler, up to file sizes of 2GB.
However, when I try to write a file on Windows larger than 2GB using
the
program built with the Visual Studio compiler I get the following
error:
Assertion failed: pxp->bf_offset<=3D offset&& offset<
pxp->bf_offset =
+
(off_t) pxp->bf_extent, file ..\posixio.c, line 325
When I look at the code at line 325 in posixio.c, I see that offset
and
pdxp->bf_offset are of type off_t. I added a printf() in that code
to
print the sizeof(offset) and sizeof(off_t), and it comes up as 4,
not 8.
But when I look at the config.h file that comes in the win32/NET
directory in netCDF 3.6.3 it has the following:
corvette:areaDetector/ADApp/netCDFSrc>grep -C3 off_t
/usr/local/netcdf/netcdf-3.6.3/win32/NET/config.h
/* #undef HAVE_ST_BLKSIZE */
/* Define to `long' if<sys/types.h> doesn't define. */
/* #undef off_t */
/* Define to `unsigned' if<sys/types.h> doesn't define. */
/* #undef size_t */
--
/* The number of bytes in a size_t */
#define SIZEOF_SIZE_T 4
/* The number of bytes in a off_t */
#define SIZEOF_OFF_T 8
/* Define to `int' if system doesn't define. */
So it defines SIZEOF_OFF_T to be 8, not 4.
I have generated the netCDF config.h file on Linux, but then edited
it
to correctly (?) define things on other platforms, like _WIN32 and
vxWorks.
The compiler flags being used on Windows are illustrated in the
following output when I build:
cl -c /nologo /D__STDC__=3D0 /D_CRT_SECURE_NO_DEPRECATE
/D_CRT_NONSTDC_NO_DEPRECATE /Ox /GL /W3 /w44355
/D_WIN32_WINNT=3D0x0503 -D_FILE_OFFSET_BITS=3D64 /MT -DEPICS_DLL_NO
=
-I.
-I..\\O.Common - I. -I.. -I..\\..\\..\\include\\os\\WIN32
-I..\\..\\..\\include -IJ:\\epics\\devel\\asyn-4-17\\include
-IJ:\\epics\\devel\\calc-2-8\\include
-IJ:\\epics\\devel\\busy-1-3\\include
-IJ:\\epics\\devel\\sscan-2-6-6\\include
-IJ:\\epics\\devel\\mca-6-12-4\\include
-IJ:\\epics\\devel\\autosave-4-7\\include\\os\\WIN32
-IJ:\\epics\\devel\\autosave-4-7\\include
-IJ:\\epics\\devel\\areaDetector-1-7beta1\\include\\os\\WIN32
-IJ:\\epics\\devel\\areaDetector-1-7beta1\\include
-IH:\\epics\\base-3.14.12.1\\include\\os\\WIN32
-IH:\\epics\\base-3.14.12.1\\include ..\\var.c
There is something I don't understand here.
Has netCDF 3.6.3 been tested to correctly write classic files> 2GB
with
the Microsoft compiler? Why am I getting the assert error?
Thanks very much,
Mark Rivers
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
