[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #WFA-619667]: Request for CONDUIT feed

Hi Randy and Martha,

re:
> Yes, Excellent!  We will have to address the wgrib2 issue. I wonder if
> this means we need to upgrade to 64bit OS now?

Need?  No.  Since the LDM was built with large file support (default when
one runs 'configure'), it will be able to create files up to 4 GB in
size (the max for most 32-bit Linux systems).  Other routines that need
to use those files will also need to be built with large file support.
To get a handle on how this is done, I recommend reviewing the 'configure'
log file that should still exist in the ~ldm/ldm-6.7.0/src directory.
This will have the compilation flags needed for large file builds in C.

re:
> You are correct that when you run wgrib2 on the 2.5 gig file on noaapxcd
> it fails. However, when we copy the file over to a 64bit machine it runs
> but gives a different error. It actually reads some of the grib messages
> before it fails. I am wondering if you are seeing the same symptoms with
> your files.

I haven't tried this specifically on motherlode or yakov.  I don't expect
problems, however, since THREDDS on motherlode is routinely handling other
model GRIB files that exceed 2 GB in size.

> I can't use IDV yet to read them because of the known issues
> with ensembles.

OK.

> You also made a comment in an earlier email about building wgrib2 with
> large file size. I am not sure I follow you on that one.

See above.

re:
> According to what I can find, the file size limitation for 32 bit Linux
> ext3 file systems with a 4KB blocksize, which is how our /data file system on
> noaapxcd is set up, is 2TB, so since our GEFS files are only 2.5GB in size,
> the file size limitation should not be an issue.

Individual file sizes are likely to be limited to 4 GB.  At least, the LDM
queue is limited to 4 GB because of the maximum size of the 'off_t' type.

> However, wgrib2 documentation says that on 32-bit OS's, it's own file
> size limitation is 2GB.  I cannot see how to force wgrib2 to change this,
> though I will continue looking, but this does explain how we can read
> the GEFS grib2 files on our 64-bit Linux box with wgrib2.  But as Randy
> says, though the 64-bit version starts to read the GEFS file, it eventually
> dies saying as before "missing end section" in rd_grib2_msg.

I can not explain this.  To troubleshoot, I would do the following:

- continue writing the GRIB2 messages into a single output file for each
  model run

- at the same time also write out the individual GRIB2 messages to disk.
  The timestamps on the GRIB2 message files will be monotonically increasing
  so you should be able to match the wgrib2 listing one-for-one with the
  individual files on disk.  You should then be able to take a hard look
  at the GRIB2 message(s) that show an error when running wgrib2.  If all
  of the individual GRIB2 messages look OK, then I would create a second
  copy of the concatenated output file and run wgrib2 on it.  If wgrib2
  run on that copy shows an error, then the problem is in wgrib2.  If not,
  then I will be scratching my head again as it would tend to indicate
  something in the OS :-(

> And we are looking for other apps that read grib2.

OK.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: WFA-619667
Department: Support IDD
Priority: Normal
Status: Closed