Hi Russ and Ed,
Thanks for your replies.
I have now upgraded from netCDF 3.6.3 to 4.1.3 as you suggested.
I have reproduced the problem in a simple test program, which I have
attached. The test program creates a netCDF file of the following
format as revealed by ncdump:
corvette:areaDetector/ADApp/netCDFSrc>ncdump -k
/home/epics/scratch/test.nc
classic
corvette:areaDetector/ADApp/netCDFSrc>ncdump -h
/home/epics/scratch/test.nc
netcdf test {
dimensions:
numArrays = UNLIMITED ; // (4100 currently)
YSize = 1024 ;
XSize = 1024 ;
variables:
byte array_data(numArrays, YSize, XSize) ;
}
So it writes a 1024x1024 byte array to the file 4100 times, where 4100
is the unlimited dimension. This tests writing files that are larger
than both the 2GB and 4GB potential limits.
The test program runs fine on Linux and Cygwin.
However, when compiled with the Microsoft Visual Studio 2008 compiler,
it fails with exactly the same error I got in netCDF 3.6.3, except it is
now in line 342 rather than 325 of posixio.c.
writing record 2047/4100
writing record 2048/4100
Assertion failed: pxp->bf_offset <= offset && offset < pxp->bf_offset +
(off_t) pxp->bf_extent, file ..\..\..\ADApp\netCDFSrc\libsrc\posixio.c,
line 342
By way of background on why I need to use the Microsoft compiler:
I run a fairly large NSF-funded user facility at the Advanced Photon
Source at Argonne National Laboratory. We collect data using a wide
variety of x-ray and other detectors under a distributed real-time
control system called EPICS. Some of these detectors can be controlled
from Linux and Cygwin, but many of them have vendor-supplied drivers
that can only be called from code compiled with the Microsoft compiler.
I also need to build my application to run on the vxWorks and RTEMS
real-time operating systems, which we use to control specialized VME
hardware. These use the gcc compiler, but cross-compiling for the
real-time target on a Linux host.
I am building with gnumake, not the Visual Studio application
development environment on all platforms, including when building with
the Microsoft VC++ compiler on Windows.
I have also attached a copy of the config.h file I am using, which has
been modified from the version built by "configure" on Linux to do
things differently if _WIN32 or vxWorks are defined.
I had to make some minor changes to the 4.1.3 source code in order to
allow it to build with the Microsoft VS 2008 compiler, and to build on
vxWorks. I will send those changes in a separate e-mail.
I have also attached an IDL version of the same program. It should be
completely equivalent to the C program. It works fine on both Linux and
on Windows. It appears to me that the idl_netcdf.dll file that ITT
provides with IDL on Windows is built with the Microsoft compiler
(judging from dumpbin /imports), but I am not certain of this. If so,
it indicates that it is possible to write netCDF classic files when
compiling with Visual Studio, and I am probably just doing something
wrong.
Perhaps it is in my flags to the compiler. Here is the output when I
compile posixio.c as an example of the flags I am using:
J:\epics\devel\areaDetector\ADApp\netCDFSrc>make install.win32-x86
make -C O.win32-x86 -f ../Makefile TOP=../../.. T_A=win32-x86 install
make[1]: Entering directory
`J:/epics/devel/areaDetector/ADApp/netCDFSrc/O.win32-x86'
cl -c /nologo /D__STDC__=0 /D_CRT_SECURE_NO_DEPRECATE
/D_CRT_NONSTDC_NO_DEPRECATE /Ox /GL /W3 /w44355
/D_WIN32_WINNT=0x0503 -DHAVE_CONFIG_H /MT -DEPICS_DLL_NO -I.
-I..\\O.Common -I. -I.. -I..\\..\\..\\ADApp\\netCDFSrc\\include
-I..\\..\\..\\ADApp\\netCDFSrc\\libsrc
-I..\\..\\..\\ADApp\\netCDFSrc\\libdispatch
-I..\\..\\..\\ADApp\\netCDFSrc\\liblib -I..\\..\\..\\include\\os\\WIN32
-I..\\..\\..\\include -IJ:\\epics\\devel\\asyn-4-17\\include
-IJ:\\epics\\devel\\calc-2-8\\include
-IJ:\\epics\\devel\\busy-1-3\\include
-IJ:\\epics\\devel\\sscan-2-6-6\\include
-IJ:\\epics\\devel\\mca-6-12-4\\include
-IJ:\\epics\\devel\\autosave-4-7\\include\\os\\WIN32
-IJ:\\epics\\devel\\autosave-4-7\\include
-IJ:\\epics\\devel\\areaDetector-1-7beta1\\include\\os\\WIN32
-IJ:\\epics\\devel\\areaDetector-1-7beta1\\include
-IH:\\epics\\base-3.14.12.1\\include\\os\\WIN32
-IH:\\epics\\base-3.14.12.1\\include
-I..\\..\\..\\ADApp\\netCDFSrc\\include
..\\..\\..\\ADApp\\netCDFSrc\\libsrc\\posixio.c
posixio.c
..\..\..\ADApp\netCDFSrc\libsrc\posixio.c(433) : warning C4018: '<' :
signed/unsigned mismatch
..\..\..\ADApp\netCDFSrc\libsrc\posixio.c(441) : warning C4018: '<=' :
signed/unsigned mismatch
..\..\..\ADApp\netCDFSrc\libsrc\posixio.c(449) : warning C4018: '<=' :
signed/unsigned mismatch
..\..\..\ADApp\netCDFSrc\libsrc\posixio.c(454) : warning C4018: '>' :
signed/unsigned mismatch
Thanks,
Mark
-----Original Message-----
From: Russ Rew [mailto:russ@xxxxxxxxxxxxxxxx]
Sent: Wednesday, June 29, 2011 1:34 PM
To: Mark Rivers
Cc: netcdfgroup@xxxxxxxxxxxxxxxx
Subject: Re: [netcdfgroup] Problem with netCDF "classic" files larger
than 2GB on Windows
Hi Mark,
> I am having a problem writing netCDF "classic" files larger than 2GB
> when building with the Microsoft Visual Studio 2008 compiler. The
same
> code works fine when build on Linux, and on Cygwin.
>
> A bit of background. This is a project to build generic file writers
> for cameras and detectors in the areaDetector package
> (http://cars9.uchicago.edu/software/epics/areaDetectorDoc.html) for
the
> EPICS (http://www.aps.anl.gov/epics/) real-time control system. This
is
> a large project, with its own build system based on gnumake. I am
> building the basic netCDF library from the same source code on all
> supported platforms (Linux 32 and 64-bit, Windows 32 and 64-bit with
> Microsoft compiler, Windows with Cygwin gcc compilet, vxWorks, Darwin,
> Solaris, etc.). Because I have another file writer that handles HDF5,
I
> am using netCDF 3.6.3, since I only want to netCDF to create "classic"
> files. I am using 3.6.3 because it is less complex than 4.x, not
> requiring any HDF5 support, etc.
I recommend you use the latest netCDF-4 release, 4.1.3. If you don't
have and HDF5 library installed, it will be built without support for
netCDF-4, but with bug fixes since 3.6.3 was released 3 years ago. For
example, these bug fixes were mentioned in the RELEASE_NOTES for 4.1.2:
Fixed two large-file bugs with using classic format or
64-bit offset format and accessing multidimensional
variables with more than 2**32 values.
If the problem you're seeing still occurs in releases since 4.1.2, it
may be something new.
--Russ
> The application is typically streaming uncompressed images, using the
> UNLIMITED dimension as the streaming dimension. Thus, each record is
> small, only a few MB, and the file size limitations of the classic
> format are not a problem. Here is an ncdump of a file header created
> with this file writer on Linux:
>
> corvette:ADApp/op/adl>ncdump -h /home/epics/scratch/netcdf_test_1.nc
> netcdf netcdf_test_1 {
> dimensions:
> numArrays =3D UNLIMITED ; // (4100 currently)
> dim0 =3D 1024 ;
> dim1 =3D 1024 ;
> attrStringSize =3D 256 ;
> variables:
> int uniqueId(numArrays) ;
> double timeStamp(numArrays) ;
> byte array_data(numArrays, dim0, dim1) ;
> int Attr_ColorMode(numArrays) ;
> double Attr_AcquireTime(numArrays) ;
> double Attr_RingCurrent(numArrays) ;
> char Attr_RingCurrent_EGU(numArrays, attrStringSize) ;
> double Attr_ID_Energy(numArrays) ;
> char Attr_ID_Energy_EGU(numArrays, attrStringSize) ;
> int Attr_ImageCounter(numArrays) ;
> int Attr_MaxSizeX(numArrays) ;
> int Attr_MaxSizeY(numArrays) ;
> char Attr_CameraModel(numArrays, attrStringSize) ;
> char Attr_CameraManufacturer(numArrays, attrStringSize) ;
>
> // global attributes:
> :dataType =3D 1 ;
> :NDNetCDFFileVersion =3D 3. ;
> :numArrayDims =3D 2 ;
> :dimSize =3D 1024, 1024 ;
> ...
>
> So the only large array is called "array_data", and in this case it is
> [4100, 1024, 1024], where 4100 is the unlimited dimension. Thus, this
> file is over 4GB, and it can be written and read with no problems on
> Linux and Cygwin. It also works fine when writing files on Windows
with
> the Microsoft compiler, up to file sizes of 2GB.
>
> However, when I try to write a file on Windows larger than 2GB using
the
> program built with the Visual Studio compiler I get the following
error:
>
> Assertion failed: pxp->bf_offset <=3D offset && offset <
pxp->bf_offset =
> +
> (off_t) pxp->bf_extent, file ..\posixio.c, line 325
>
> When I look at the code at line 325 in posixio.c, I see that offset
and
> pdxp->bf_offset are of type off_t. I added a printf() in that code to
> print the sizeof(offset) and sizeof(off_t), and it comes up as 4, not
8.
>
>
> But when I look at the config.h file that comes in the win32/NET
> directory in netCDF 3.6.3 it has the following:
> corvette:areaDetector/ADApp/netCDFSrc>grep -C3 off_t
> /usr/local/netcdf/netcdf-3.6.3/win32/NET/config.h
> /* #undef HAVE_ST_BLKSIZE */
>
> /* Define to `long' if <sys/types.h> doesn't define. */
> /* #undef off_t */
>
> /* Define to `unsigned' if <sys/types.h> doesn't define. */
> /* #undef size_t */
> --
> /* The number of bytes in a size_t */
> #define SIZEOF_SIZE_T 4
>
> /* The number of bytes in a off_t */
> #define SIZEOF_OFF_T 8
>
> /* Define to `int' if system doesn't define. */
>
>
> So it defines SIZEOF_OFF_T to be 8, not 4.
>
> I have generated the netCDF config.h file on Linux, but then edited it
> to correctly (?) define things on other platforms, like _WIN32 and
> vxWorks.
>
> The compiler flags being used on Windows are illustrated in the
> following output when I build:
>
> cl -c /nologo /D__STDC__=3D0 /D_CRT_SECURE_NO_DEPRECATE
> /D_CRT_NONSTDC_NO_DEPRECATE /Ox /GL /W3 /w44355
> /D_WIN32_WINNT=3D0x0503 -D_FILE_OFFSET_BITS=3D64 /MT -DEPICS_DLL_NO
=
> -I.
> -I..\\O.Common -I. -I.. -I..\\..\\..\\include\\os\\WIN32
> -I..\\..\\..\\include -IJ:\\epics\\devel\\asyn-4-17\\include
> -IJ:\\epics\\devel\\calc-2-8\\include
> -IJ:\\epics\\devel\\busy-1-3\\include
> -IJ:\\epics\\devel\\sscan-2-6-6\\include
> -IJ:\\epics\\devel\\mca-6-12-4\\include
> -IJ:\\epics\\devel\\autosave-4-7\\include\\os\\WIN32
> -IJ:\\epics\\devel\\autosave-4-7\\include
> -IJ:\\epics\\devel\\areaDetector-1-7beta1\\include\\os\\WIN32
> -IJ:\\epics\\devel\\areaDetector-1-7beta1\\include
> -IH:\\epics\\base-3.14.12.1\\include\\os\\WIN32
> -IH:\\epics\\base-3.14.12.1\\include ..\\var.c
>
> There is something I don't understand here.
>
> Has netCDF 3.6.3 been tested to correctly write classic files > 2GB
with
> the Microsoft compiler? Why am I getting the assert error?
>
> Thanks very much,
> Mark Rivers
Attachment:
test_big_classic.c
Description: test_big_classic.c
Attachment:
test_big_classic.pro
Description: test_big_classic.pro
Attachment:
config.h
Description: config.h
netcdfgroup archives: