Re: [netcdfgroup] Problem with netCDF "classic" files larger than 2GB on Windows

Hi Russ,

 

I am puzzled on how to proceed with this.  You recommend that I run
4.1.3, but my problem is specifically on Windows with the Microsoft VS
compiler.  When I look on the netCDF Web site I find that for Windows
with VS my choices are this:

 

NOTE: User Viet Eitner has contributed a port of 4.1.1 to Visual Studio,
including an F90 port to Intel Fortran. Download source
(ftp://ftp.unidata.ucar.edu/pub/netcdf/contrib/win32/netcdf-4.1.1-win32-
src.zip)
<ftp://ftp.unidata.ucar.edu/pub/netcdf/contrib/win32/netcdf-4.1.1-win32-
src.zip>  or binary
(ftp://ftp.unidata.ucar.edu/pub/netcdf/contrib/win32/netcdf-4.1.1-win32-
bin.zip)
<ftp://ftp.unidata.ucar.edu/pub/netcdf/contrib/win32/netcdf-4.1.1-win32-
bin.zip>  versions. This port was done before the code was refactored in
4.1.2. 

 

Or this:

Two versions of the netCDF DLLs are available, for different Fortran
compilers: 

*       NetCDF for Intel and Portland Group Fortran compilers.
<ftp://ftp.unidata.ucar.edu/pub/netcdf/contrib/win32/win32_vs_PGI_dll_4.
0.1.zip>  
*       NetCDF for other Fortran compilers.
<ftp://ftp.unidata.ucar.edu/pub/netcdf/contrib/win32/win32_vs_f2c_dll_4.
0.1.zip>  

The former is 4.1.1, and the latter are 4.0.1.

 

What is your suggestion as to how to build with VS on 4.1.3?

 

I do build my application with Cygwin, but I also need to build with VS,
because there are many detector drivers that do not build correctly with
Cygwin, they require VS.  So I need to build my application with both
compilers.

 

Note that I am not using the VS development environment, I am using
gnumake.  But I am invoking the VS compiler and linker.

 

Can anyone answer the question as to whether netCDF 3.6.3 with Windows
VS can correctly write classic netCDF files with sizes >2GB, as Linux
and Cygwin can?

 

Thanks,

Mark

 

 

 

-----Original Message-----
From: Russ Rew [mailto:russ@xxxxxxxxxxxxxxxx] 
Sent: Wednesday, June 29, 2011 1:34 PM
To: Mark Rivers
Cc: netcdfgroup@xxxxxxxxxxxxxxxx
Subject: Re: [netcdfgroup] Problem with netCDF "classic" files larger
than 2GB on Windows

 

Hi Mark,

 

> I am having a problem writing netCDF "classic" files larger than 2GB

> when building with the Microsoft Visual Studio 2008 compiler.  The
same

> code works fine when build on Linux, and on Cygwin.

> 

> A bit of background.  This is a project to build generic file writers

> for cameras and detectors in the areaDetector package

> (http://cars9.uchicago.edu/software/epics/areaDetectorDoc.html) for
the

> EPICS (http://www.aps.anl.gov/epics/) real-time control system.  This
is

> a large project, with its own build system based on gnumake.  I am

> building the basic netCDF library from the same source code on all

> supported platforms (Linux 32 and 64-bit, Windows 32 and 64-bit with

> Microsoft compiler, Windows with Cygwin gcc compilet, vxWorks, Darwin,

> Solaris, etc.).  Because I have another file writer that handles HDF5,
I

> am using netCDF 3.6.3, since I only want to netCDF to create "classic"

> files.  I am using 3.6.3 because it is less complex than 4.x, not

> requiring any HDF5 support, etc.

 

I recommend you use the latest netCDF-4 release, 4.1.3.  If you don't

have and HDF5 library installed, it will be built without support for

netCDF-4, but with bug fixes since 3.6.3 was released 3 years ago.  For

example, these bug fixes were mentioned in the RELEASE_NOTES for 4.1.2:

 

            Fixed two large-file bugs with using classic format or

            64-bit offset format and accessing multidimensional

            variables with more than 2**32 values.

 

If the problem you're seeing still occurs in releases since 4.1.2, it

may be something new.

 

--Russ

 

> The application is typically streaming uncompressed images, using the

> UNLIMITED dimension as the streaming dimension. Thus, each record is

> small, only a few MB, and the file size limitations of the classic

> format are not a problem.  Here is an ncdump of a file header created

> with this file writer on Linux:

> 

> corvette:ADApp/op/adl>ncdump -h /home/epics/scratch/netcdf_test_1.nc

> netcdf netcdf_test_1 {

> dimensions:

>         numArrays =3D UNLIMITED ; // (4100 currently)

>         dim0 =3D 1024 ;

>         dim1 =3D 1024 ;

>         attrStringSize =3D 256 ;

> variables:

>         int uniqueId(numArrays) ;

>         double timeStamp(numArrays) ;

>         byte array_data(numArrays, dim0, dim1) ;

>         int Attr_ColorMode(numArrays) ;

>         double Attr_AcquireTime(numArrays) ;

>         double Attr_RingCurrent(numArrays) ;

>         char Attr_RingCurrent_EGU(numArrays, attrStringSize) ;

>         double Attr_ID_Energy(numArrays) ;

>         char Attr_ID_Energy_EGU(numArrays, attrStringSize) ;

>         int Attr_ImageCounter(numArrays) ;

>         int Attr_MaxSizeX(numArrays) ;

>         int Attr_MaxSizeY(numArrays) ;

>         char Attr_CameraModel(numArrays, attrStringSize) ;

>         char Attr_CameraManufacturer(numArrays, attrStringSize) ;

> 

> // global attributes:

>                 :dataType =3D 1 ;

>                 :NDNetCDFFileVersion =3D 3. ;

>                 :numArrayDims =3D 2 ;

>                 :dimSize =3D 1024, 1024 ;

> ...

> 

> So the only large array is called "array_data", and in this case it is

> [4100, 1024, 1024], where 4100 is the unlimited dimension.  Thus, this

> file is over 4GB, and it can be written and read with no problems on

> Linux and Cygwin.  It also works fine when writing files on Windows
with

> the Microsoft compiler, up to file sizes of 2GB.

> 

> However, when I try to write a file on Windows larger than 2GB using
the

> program built with the Visual Studio compiler I get the following
error:

> 

> Assertion failed: pxp->bf_offset <=3D offset && offset <
pxp->bf_offset =

> +

> (off_t) pxp->bf_extent, file ..\posixio.c, line 325

> 

> When I look at the code at line 325 in posixio.c, I see that offset
and

> pdxp->bf_offset are of type off_t.  I added a printf() in that code to

> print the sizeof(offset) and sizeof(off_t), and it comes up as 4, not
8.

> 

> 

> But when I look at the config.h file that comes in the win32/NET

> directory in netCDF 3.6.3 it has the following:

> corvette:areaDetector/ADApp/netCDFSrc>grep -C3 off_t

> /usr/local/netcdf/netcdf-3.6.3/win32/NET/config.h

> /* #undef HAVE_ST_BLKSIZE */

> 

> /* Define to `long' if <sys/types.h> doesn't define.  */

> /* #undef off_t */

> 

> /* Define to `unsigned' if <sys/types.h> doesn't define.  */

> /* #undef size_t */

> --

> /* The number of bytes in a size_t */

> #define SIZEOF_SIZE_T 4

> 

> /* The number of bytes in a off_t */

> #define SIZEOF_OFF_T 8

> 

> /* Define to `int' if system doesn't define.  */

> 

> 

> So it defines SIZEOF_OFF_T to be 8, not 4.

> 

> I have generated the netCDF config.h file on Linux, but then edited it

> to correctly (?) define things on other platforms, like _WIN32 and

> vxWorks.

> 

> The compiler flags being used on Windows are illustrated in the

> following output when I build:

> 

> cl -c               /nologo /D__STDC__=3D0 /D_CRT_SECURE_NO_DEPRECATE

> /D_CRT_NONSTDC_NO_DEPRECATE   /Ox /GL   /W3 /w44355

> /D_WIN32_WINNT=3D0x0503 -D_FILE_OFFSET_BITS=3D64   /MT -DEPICS_DLL_NO
=

> -I.

> -I..\\O.Common -I. -I.. -I..\\..\\..\\include\\os\\WIN32

> -I..\\..\\..\\include  -IJ:\\epics\\devel\\asyn-4-17\\include

> -IJ:\\epics\\devel\\calc-2-8\\include

> -IJ:\\epics\\devel\\busy-1-3\\include

> -IJ:\\epics\\devel\\sscan-2-6-6\\include

> -IJ:\\epics\\devel\\mca-6-12-4\\include

> -IJ:\\epics\\devel\\autosave-4-7\\include\\os\\WIN32

> -IJ:\\epics\\devel\\autosave-4-7\\include

> -IJ:\\epics\\devel\\areaDetector-1-7beta1\\include\\os\\WIN32

> -IJ:\\epics\\devel\\areaDetector-1-7beta1\\include

> -IH:\\epics\\base-3.14.12.1\\include\\os\\WIN32

> -IH:\\epics\\base-3.14.12.1\\include       ..\\var.c

> 

> There is something I don't understand here.

> 

> Has netCDF 3.6.3 been tested to correctly write classic files > 2GB
with

> the Microsoft compiler?  Why am I getting the assert error?

> 

> Thanks very much,

> Mark Rivers

  • 2011 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: