Hi Christoph,
Thanks very much, that indeed fixed the problem! When I first tried
your lines below I got compiler errors, but then I looked at your
version of config.h and saw that you also need to #include io.h and
process.h before the #defines. Thus my config.h now contains:
/* The size of `off_t', as computed by sizeof. */
#if defined(vxWorks)
#define SIZEOF_OFF_T 4
#elif defined(_WIN32)
#include <io.h>
#include <process.h>
#define SIZEOF_OFF_T 8
#define lseek _lseeki64
#define off_t __int64
#define _off_t __int64
#define _OFF_T_DEFINED
#else
#define SIZEOF_OFF_T 8
#endif
Thanks again for the help!
It's interesting that netCDF 3.6.3 contains a config.h in the win32/
directory, but it does not contain the above fixes, so it does not work
for creating files >2GB.
I also noted that your config.h file contains the following
/* I added the following to this config.h file by hand, after being
abducted by
aliens last week in Kansas. (All Hail Zorlock, Mighty Destroyer of
Worlds!) */
#include <io.h>
#include <process.h>
#define lseek _lseeki64
#define off_t __int64
#define _off_t __int64
#define _OFF_T_DEFINED
#define snprintf sprintf_s
#define strcasecmp _stricmp
So did the aliens teach you this trick, or where did you learn it?
Cheers,
Mark
-----Original Message-----
From: netcdfgroup-bounces@xxxxxxxxxxxxxxxx
[mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Christoph
Gohlke
Sent: Tuesday, July 05, 2011 5:46 PM
To: netcdfgroup@xxxxxxxxxxxxxxxx
Subject: Re: [netcdfgroup] Problem with netCDF "classic" files larger
than 2GB on Windows
Try to compile with 64 bit off_t and lseek. E.g. in config.h
#define SIZEOF_OFF_T 8
#define lseek _lseeki64
#define off_t __int64
#define _off_t __int64
#define _OFF_T_DEFINED
Your test_big_classic.c works for me. The config.h I'm using with msvc9
is attached.
Christoph
On 7/5/2011 3:13 PM, Mark Rivers wrote:
> Hi Russ and Ed,
>
> Thanks for your replies.
>
> I have now upgraded from netCDF 3.6.3 to 4.1.3 as you suggested.
>
> I have reproduced the problem in a simple test program, which I have
> attached. The test program creates a netCDF file of the following
format
> as revealed by ncdump:
>
> corvette:areaDetector/ADApp/netCDFSrc>ncdump -k
/home/epics/scratch/test.nc
>
> classic
>
> corvette:areaDetector/ADApp/netCDFSrc>ncdump -h
/home/epics/scratch/test.nc
>
> netcdf test {
>
> dimensions:
>
> numArrays = UNLIMITED ; // (4100 currently)
>
> YSize = 1024 ;
>
> XSize = 1024 ;
>
> variables:
>
> byte array_data(numArrays, YSize, XSize) ;
>
> }
>
> So it writes a 1024x1024 byte array to the file 4100 times, where 4100
> is the unlimited dimension. This tests writing files that are larger
> than both the 2GB and 4GB potential limits.
>
> The test program runs fine on Linux and Cygwin.
>
> However, when compiled with the Microsoft Visual Studio 2008 compiler,
> it fails with exactly the same error I got in netCDF 3.6.3, except it
is
> now in line 342 rather than 325 of posixio.c.
>
> writing record 2047/4100
>
> writing record 2048/4100
>
> Assertion failed: pxp->bf_offset <= offset && offset < pxp->bf_offset
+
> (off_t) pxp->bf_extent, file
..\..\..\ADApp\netCDFSrc\libsrc\posixio.c,
> line 342
>
> By way of background on why I need to use the Microsoft compiler:
>
> I run a fairly large NSF-funded user facility at the Advanced Photon
> Source at Argonne National Laboratory. We collect data using a wide
> variety of x-ray and other detectors under a distributed real-time
> control system called EPICS. Some of these detectors can be controlled
> from Linux and Cygwin, but many of them have vendor-supplied drivers
> that can only be called from code compiled with the Microsoft
compiler.
> I also need to build my application to run on the vxWorks and RTEMS
> real-time operating systems, which we use to control specialized VME
> hardware. These use the gcc compiler, but cross-compiling for the
> real-time target on a Linux host.
>
> I am building with gnumake, not the Visual Studio application
> development environment on all platforms, including when building with
> the Microsoft VC++ compiler on Windows.
>
> I have also attached a copy of the config.h file I am using, which has
> been modified from the version built by "configure" on Linux to do
> things differently if _WIN32 or vxWorks are defined.
>
> I had to make some minor changes to the 4.1.3 source code in order to
> allow it to build with the Microsoft VS 2008 compiler, and to build on
> vxWorks. I will send those changes in a separate e-mail.
>
> I have also attached an IDL version of the same program. It should be
> completely equivalent to the C program. It works fine on both Linux
and
> on Windows. It appears to me that the idl_netcdf.dll file that ITT
> provides with IDL on Windows is built with the Microsoft compiler
> (judging from dumpbin /imports), but I am not certain of this. If so,
it
> indicates that it is possible to write netCDF classic files when
> compiling with Visual Studio, and I am probably just doing something
wrong.
>
> Perhaps it is in my flags to the compiler. Here is the output when I
> compile posixio.c as an example of the flags I am using:
>
> J:\epics\devel\areaDetector\ADApp\netCDFSrc>make install.win32-x86
>
> make -C O.win32-x86 -f ../Makefile TOP=../../.. T_A=win32-x86 install
>
> make[1]: Entering directory
> `J:/epics/devel/areaDetector/ADApp/netCDFSrc/O.win32-x86'
>
> cl -c /nologo /D__STDC__=0 /D_CRT_SECURE_NO_DEPRECATE
> /D_CRT_NONSTDC_NO_DEPRECATE /Ox /GL /W3 /w44355 /D_WIN32_WINNT=0x0503
> -DHAVE_CONFIG_H /MT -DEPICS_DLL_NO - I. -I..\\O.Common - I. -I..
> -I..\\..\\..\\ADApp\\netCDFSrc\\include
> -I..\\..\\..\\ADApp\\netCDFSrc\\libsrc
> -I..\\..\\..\\ADApp\\netCDFSrc\\libdispatch
> -I..\\..\\..\\ADApp\\netCDFSrc\\liblib
-I..\\..\\..\\include\\os\\WIN32
> -I..\\..\\..\\include -IJ:\\epics\\devel\\asyn-4-17\\include
> -IJ:\\epics\\devel\\calc-2-8\\include
> -IJ:\\epics\\devel\\busy-1-3\\include
> -IJ:\\epics\\devel\\sscan-2-6-6\\include
> -IJ:\\epics\\devel\\mca-6-12-4\\include
> -IJ:\\epics\\devel\\autosave-4-7\\include\\os\\WIN32
> -IJ:\\epics\\devel\\autosave-4-7\\include
> -IJ:\\epics\\devel\\areaDetector-1-7beta1\\include\\os\\WIN32
> -IJ:\\epics\\devel\\areaDetector-1-7beta1\\include
> -IH:\\epics\\base-3.14.12.1\\include\\os\\WIN32
> -IH:\\epics\\base-3.14.12.1\\include
> -I..\\..\\..\\ADApp\\netCDFSrc\\include
> ..\\..\\..\\ADApp\\netCDFSrc\\libsrc\\posixio.c
>
> posixio.c
>
> ..\..\..\ADApp\netCDFSrc\libsrc\posixio.c(433) : warning C4018: '<' :
> signed/unsigned mismatch
>
> ..\..\..\ADApp\netCDFSrc\libsrc\posixio.c(441) : warning C4018: '<=' :
> signed/unsigned mismatch
>
> ..\..\..\ADApp\netCDFSrc\libsrc\posixio.c(449) : warning C4018: '<=' :
> signed/unsigned mismatch
>
> ..\..\..\ADApp\netCDFSrc\libsrc\posixio.c(454) : warning C4018: '>' :
> signed/unsigned mismatch
>
> Thanks,
>
> Mark
>
> -----Original Message-----
> From: Russ Rew [mailto:russ@xxxxxxxxxxxxxxxx]
> Sent: Wednesday, June 29, 2011 1:34 PM
> To: Mark Rivers
> Cc: netcdfgroup@xxxxxxxxxxxxxxxx
> Subject: Re: [netcdfgroup] Problem with netCDF "classic" files larger
> than 2GB on Windows
>
> Hi Mark,
>
>> I am having a problem writing netCDF "classic" files larger than 2GB
>
>> when building with the Microsoft Visual Studio 2008 compiler. The
same
>
>> code works fine when build on Linux, and on Cygwin.
>
>>
>
>> A bit of background. This is a project to build generic file writers
>
>> for cameras and detectors in the areaDetector package
>
>> (http://cars9.uchicago.edu/software/epics/areaDetectorDoc.html) for
the
>
>> EPICS (http://www.aps.anl.gov/epics/) real-time control system. This
is
>
>> a large project, with its own build system based on gnumake. I am
>
>> building the basic netCDF library from the same source code on all
>
>> supported platforms (Linux 32 and 64-bit, Windows 32 and 64-bit with
>
>> Microsoft compiler, Windows with Cygwin gcc compilet, vxWorks,
Darwin ,
>
>> Solaris, etc.). Because I have another file writer that handles
HDF5, I
>
>> am using netCDF 3.6.3, since I only want to netCDF to create
"classic"
>
>> files. I am using 3.6.3 because it is less complex than 4.x, not
>
>> requiring any HDF5 support, etc.
>
> I recommend you use the latest netCDF-4 release, 4.1.3. If you don't
>
> have and HDF5 library installed, it will be built without support for
>
> netCDF-4, but with bug fixes since 3.6.3 was released 3 years ago. For
>
> example, these bug fixes were mentioned in the RELEASE_NOTES for
4.1.2:
>
> Fixed two large-file bugs with using classic format or
>
> 64-bit offset format and accessing multidimensional
>
> variables with more than 2**32 values.
>
> If the problem you're seeing still occurs in releases since 4.1.2, it
>
> may be something new.
>
> --Russ
>
>> The application is typically streaming uncompressed images, using
the
>
>> UNLIMITED dimension as the streaming dimension. Thus, each record is
>
>> small, only a few MB, and the file size limitations of the classic
>
>> format are not a problem. Here is an ncdump of a file header created
>
>> with this file writer on Linux:
>
>>
>
>> corvette:ADApp/op/adl>ncdump -h /home/epics/scratch/netcdf_test_1.nc
>
>> netcdf netcdf_test_1 {
>
>> dimensions:
>
>> numArrays =3D UNLIMITED ; // (4100 currently)
>
>> dim0 =3D 1024 ;
>
>> dim1 =3D 1024 ;
>
>> attrStringSize =3D 256 ;
>
>> variables:
>
>> int uniqueId(numArrays) ;
>
>> double timeStamp(numArrays) ;
>
>> byte array_data(numArrays, dim0, dim1) ;
>
>> int Attr_ColorMode(numArrays) ;
>
>> double Attr_AcquireTime(numArrays) ;
>
>> double Attr_RingCurrent(numArrays) ;
>
>> char Attr_RingCurrent_EGU(numArrays, attrStringSize) ;
>
>> double Attr_ID_Energy(numArrays) ;
>
>> char Attr_ID_Energy_EGU(numArrays, attrStringSize) ;
>
>> int Attr_ImageCounter(numArrays) ;
>
>> int Attr_MaxSizeX(numArrays) ;
>
>> int Attr_MaxSizeY(numArrays) ;
>
>> char Attr_CameraModel(numArrays, attrStringSize) ;
>
>> char Attr_CameraManufacturer(numArrays, attrStringSize) ;
>
>>
>
>> // global attributes:
>
>> :dataType =3D 1 ;
>
>> :NDNetCDFFileVersion =3D 3. ;
>
>> :numArrayDims =3D 2 ;
>
>> :dimSize =3D 1024, 1024 ;
>
>> ...
>
>>
>
>> So the only large array is called "array_data", and in this case it
is
>
>> [4100, 1024, 1024], where 4100 is the unlimited dimension. Thus,
this
>
>> file is over 4GB, and it can be written and read with no problems on
>
>> Linux and Cygwin. It also works fine when writing files on Windows
with
>
>> the Microsoft compiler, up to file sizes of 2GB.
>
>>
>
>> However, when I try to write a file on Windows larger than 2GB using
the
>
>> program built with the Visual Studio compiler I get the following
error:
>
>>
>
>> Assertion failed: pxp->bf_offset <=3D offset && offset <
pxp->bf_offset =
>
>> +
>
>> (off_t) pxp->bf_extent, file ..\posixio.c, line 325
>
>>
>
>> When I look at the code at line 325 in posixio.c, I see that offset
and
>
>> pdxp->bf_offset are of type off_t. I added a printf() in that code
to
>
>> print the sizeof(offset) and sizeof(off_t), and it comes up as 4,
not 8.
>
>>
>
>>
>
>> But when I look at the config.h file that comes in the win32/NET
>
>> directory in netCDF 3.6.3 it has the following:
>
>> corvette:areaDetector/ADApp/netCDFSrc>grep -C3 off_t
>
>> /usr/local/netcdf/netcdf-3.6.3/win32/NET/config.h
>
>> /* #undef HAVE_ST_BLKSIZE */
>
>>
>
>> /* Define to `long' if <sys/types.h> doesn't define. */
>
>> /* #undef off_t */
>
>>
>
>> /* Define to `unsigned' if <sys/types.h> doesn't define. */
>
>> /* #undef size_t */
>
>> --
>
>> /* The number of bytes in a size_t */
>
>> #define SIZEOF_SIZE_T 4
>
>>
>
>> /* The number of bytes in a off_t */
>
>> #define SIZEOF_OFF_T 8
>
>>
>
>> /* Define to `int' if system doesn't define. */
>
>>
>
>>
>
>> So it defines SIZEOF_OFF_T to be 8, not 4.
>
>>
>
>> I have generated the netCDF config.h file on Linux, but then edited
it
>
>> to correctly (?) define things on other platforms, like _WIN32 and
>
>> vxWorks.
>
>>
>
>> The compiler flags being used on Windows are illustrated in the
>
>> following output when I build:
>
>>
>
>> cl -c /nologo /D__STDC__=3D0 /D_CRT_SECURE_NO_DEPRECATE
>
>> /D_CRT_NONSTDC_NO_DEPRECATE /Ox /GL /W3 /w44355
>
>> /D_WIN32_WINNT=3D0x0503 -D_FILE_OFFSET_BITS=3D64 /MT -DEPICS_DLL_NO
=
>
>> -I.
>
>> -I..\\O.Common - I. -I.. -I..\\..\\..\\include\\os\\WIN32
>
>> -I..\\..\\..\\include -IJ:\\epics\\devel\\asyn-4-17\\include
>
>> -IJ:\\epics\\devel\\calc-2-8\\include
>
>> -IJ:\\epics\\devel\\busy-1-3\\include
>
>> -IJ:\\epics\\devel\\sscan-2-6-6\\include
>
>> -IJ:\\epics\\devel\\mca-6-12-4\\include
>
>> -IJ:\\epics\\devel\\autosave-4-7\\include\\os\\WIN32
>
>> -IJ:\\epics\\devel\\autosave-4-7\\include
>
>> -IJ:\\epics\\devel\\areaDetector-1-7beta1\\include\\os\\WIN32
>
>> -IJ:\\epics\\devel\\areaDetector-1-7beta1\\include
>
>> -IH:\\epics\\base-3.14.12.1\\include\\os\\WIN32
>
>> -IH:\\epics\\base-3.14.12.1\\include ..\\var.c
>
>>
>
>> There is something I don't understand here.
>
>>
>
>> Has netCDF 3.6.3 been tested to correctly write classic files > 2GB
with
>
>> the Microsoft compiler? Why am I getting the assert error?
>
>>
>
>> Thanks very much,
>
>> Mark Rivers
>
>
>
> _______________________________________________
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
