Re: [netcdfgroup] Predictable NetCDF: HDF error

  • To: Gustavo Correa <gus@xxxxxxxxxxxxxxxxx>
  • Subject: Re: [netcdfgroup] Predictable NetCDF: HDF error
  • From: "Ian Harris (ENV)" <I.Harris@xxxxxxxxx>
  • Date: Mon, 26 Jun 2017 10:44:01 +0000
  • Authentication-results: ldeo.columbia.edu; dkim=none (message not signed) header.d=none;ldeo.columbia.edu; dmarc=none action=none header.from=uea.ac.uk;
  • Spamdiagnosticmetadata: NSPM
  • Spamdiagnosticoutput: 1:99
Hi Gustavo,

That hasn’t been my experience. If I create a file in classic format, then 
using (eg) nf_put_vara_real() results in an error. Creating it as nf_netcdf4  
works. That’s the only change I need to make.

I understand that it *ought* to work :)

I’m also finding it nearly impossible to install valgrind on OSX 10.12.5 
('Sierra'); they have snapshot builds that get close, but I really don’t have 
the time (or probably the expertise) to help them debug it.

Cheers

Harry

On 25 Jun 2017, at 23:37, Gustavo Correa 
<gus@xxxxxxxxxxxxxxxxx<mailto:gus@xxxxxxxxxxxxxxxxx>> wrote:

Hi Ian

As far as I know the F77 netCDF interface of netCDF-3 and
netCDF-4 (the latter compiled *without* --enable-netcdf4 or with it)
support Fortran REAL variables (single precision floating point).
Functions specific to REAL variables,
such as nf_get_var1_real that appear in your program,
are supported and work fine.

Gus Correa

On Jun 24, 2017, at 8:58 AM, aitch wrote:

Thank you both for your speedy suggestions!

Unfortunately I need NetCDF-4 for this, because the F77 NetCDF interface won’t 
allow REAL otherwise. However, I will try what combinations I can manage on 
Monday (screen sharing over a VPN is not ideal for *nix tinkering).

Cheers

Harry

On 23 Jun 2017, at 21:38, dmh@xxxxxxxx<mailto:dmh@xxxxxxxx> wrote:

All good suggestions.
The goal is to try to isolate the problem
to see if the error is associated with HDF5, or mpi, or
the core netcdf-c library.

On 6/23/2017 2:17 PM, Gus Correa wrote:
Hi Ian, Dennis
May I suggest a workaround?
If your files are netCDF "Classic" or "64 bit offset" ('ncdump -k' will tell) 
they don't need
NetCDF-4 or HDF5 support to be processed.
Am I right?
If so, you could compile the netCDF 3.6.3 library and link your program to it 
instead.
Alternatively, you could compile netCDF 4.4.1 disabling netCDF-4 (run configure 
without --enable-netcdf4).
Either one doesn't require HDF5, and *may* not have these errors.
Other than this, perhaps lowering the compiler optimization level,
or turning off OpenMP, if it is on.
I hope this helps,
Gus Correa
On 06/23/2017 03:32 PM, dmh@xxxxxxxx<mailto:dmh@xxxxxxxx> wrote:
Is there any chance that you can run this using valgrind?
My speculation is that it is a memory leak of some kind.
=Dennis Heimbigner
Unidata


On 6/23/2017 7:38 AM, Ian Harris (ENV) wrote:
Hi,

I have NetCDF installed (netcdf library version 4.4.1 per ncdump) on a fairly 
powerful iMac (4GHz i7, 16GB). I use Fortran 77 (yes, I know, but it’s very 
fast).

I’m processing a lot of data, inputs and outputs all NetCDF, in loops, and it’s 
dying at predictable and reproducible points.

One example is a simple prog I knocked up; it opens a nc file, then continually 
reads the first time value in a loop until it dies.

The last successful loop is 16777187, which may sound familiar, as it’s been 
here before:

http://www.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/2014/msg00173.html

The error is usually 'NetCDF: HDF error', though occasionally 'NetCDF: Error 
initializing for parallel access'.

I should note that a much more complicated program, which reads in three files 
and writes one, also dies like this after a fixed number of runs through the 
loops (taking over an hour).

The simple test program is below.

Please can anyone hazard a guess as to what’s happening?

Cheers

Harry


Here’s the entire program:

    program netcdftestloop
c NETCDFTESTLOOP  Testing iteration counts before it dies
    implicit none
    include 'netcdf.inc'
    integer nits,st,ncid,tvarid
    real    rday

    nits = 0
    st = nf_open('Prec_daily_WFD_CRU_grid_198605.nc',0,ncid)
     if (st.ne.nf_noerr) call handle_err(st,'openff')
    st = nf_inq_varid(ncid,'time',tvarid)
     if (st.ne.nf_noerr) call handle_err(st,'inqtvi')
    do
      nits=nits+1
      if (mod(nits,100000).eq.0) write(*,'(i10)')nits                 ! keep 
track
      st = nf_get_var1_real(ncid,tvarid,1,rday)                       ! get day 
1 time value
       if (st.ne.nf_noerr) then
         write(*,'("N. iterations = ",i10)')nits
         call handle_err(st,'getday')
       endif
     enddo
     st = nf_close(ncid)                                               ! 
totally unnecessary close file
       if (st.ne.nf_noerr) call handle_err(st,'closef')
     stop
     end


c 
***********************************************************************************************
    subroutine handle_err(errcode,code)
    implicit none
    include 'netcdf.inc'
    integer errcode
    character code*6

    print *, 'Error: ', code, ': ', nf_strerror(errcode)
    stop
    end

c 
***********************************************************************************************

In a more complicated program, I get a different error, ('')


Ian "Harry” Harris
NCAS-Climate
Climatic Research Unit
School of Environmental Sciences
University of East Anglia
Norwich NR4 7TJ
United Kingdom

IMPORTANT NOTICE - This email is intended for the named recipient only. It may 
contain privileged and confidential information. If you are not the intended 
recipient, notify the sender immediately and destroy this email. You must not 
copy, distribute or take action in reliance upon it. Whilst all efforts are 
made to safeguard emails, The School of Environmental Sciences cannot guarantee 
that attachments are virus free or compatible with your systems and does not 
accept liability in respect of viruses or computer problems experienced.











_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx<mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/


_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx<mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx<mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/

_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx<mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/

_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx<mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/


_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx<mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/



Ian "Harry” Harris
NCAS-Climate
Climatic Research Unit
School of Environmental Sciences
University of East Anglia
Norwich NR4 7TJ
United Kingdom

IMPORTANT NOTICE - This email is intended for the named recipient only. It may 
contain privileged and confidential information. If you are not the intended 
recipient, notify the sender immediately and destroy this email. You must not 
copy, distribute or take action in reliance upon it. Whilst all efforts are 
made to safeguard emails, The School of Environmental Sciences cannot guarantee 
that attachments are virus free or compatible with your systems and does not 
accept liability in respect of viruses or computer problems experienced.









  • 2017 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: