Daryl,
I have several shell scripts that I'm running out of cron every 5 minutes.
Each shell script runs 10 gpmap_gf processes in sequence. I've tried running 1
- 6 scripts at a time. This typically works fine during the day with one of the
these scripts completing in about 2 minutes. As evening comes on they take
longer and longer to run and it seems like that take more and more memory. From
the "top" command the scripts often use 500-800 MB of memory but in the evening
this seems to mushroom to > 3GB per script. The load on the machine at night
from these scripts alone jumps to >30 and by morning the machine usually dies
with out of memory errors even though I'm automatically killing the scripts
when they run longer than 2 minutes.
Looking at /var/log/debug.log I'm seeing segfault errors:
Apr 26 17:07:25 sferic kernel: gpmap_gf[2164]: segfault at 0 ip
000000392692ff7f sp 00007fff8d981128 error 4 in libc-2.12.so[3926800000+186000]
Apr 26 17:07:25 sferic abrt[2179]: saved core dump of pid 2164
(/export/ldm/home/gempak/GEMPAK6.4.0/os/linux64/bin/gpmap_gf) to
/var/spool/abrt/ccpp-201
2-04-26-17:07:25-2164.new/coredump (827392 bytes)
Apr 26 17:07:25 sferic abrtd: Directory 'ccpp-2012-04-26-17:07:25-2164'
creation detected
Apr 26 17:07:25 sferic kernel: gpmap_gf[2239]: segfault at 0 ip
000000392692ff7f sp 00007ffffd173658 error 4 in libc-2.12.so[3926800000+186000]
Apr 26 17:07:25 sferic kernel: gpmap_gf[2242]: segfault at 0 ip
000000392692ff7f sp 00007fff8f4df6f8 error 4 in libc-2.12.so[3926800000+186000]
Apr 26 17:07:25 sferic kernel: gpmap_gf[2247]: segfault at 0 ip
000000392692ff7f sp 00007fff73574d18 error 4 in libc-2.12.so[3926800000+186000]
Apr 26 17:07:25 sferic kernel: gpmap_gf[2261]: segfault at 0 ip
000000392692ff7f sp 00007fff8bda1358 error 4 in libc-2.12.so[3926800000+186000]
Apr 26 17:07:25 sferic kernel: gpmap_gf[2245]: segfault at 0 ip
000000392692ff7f sp 00007fff71495a28 error 4 in libc-2.12.so[3926800000+186000]
Apr 26 17:07:25 sferic kernel: Pid 2245(gpmap_gf) over core_pipe_limit
Apr 26 17:07:25 sferic kernel: Skipping core dump
Apr 26 17:07:25 sferic abrt[2260]: not dumping repeating crash in
'/export/ldm/home/gempak/GEMPAK6.4.0/os/linux64/bin/gpmap_gf'
Apr 26 17:07:25 sferic abrt[2279]: not dumping repeating crash in
'/export/ldm/home/gempak/GEMPAK6.4.0/os/linux64/bin/gpmap_gf'
Apr 26 17:07:25 sferic kernel: gpmap_gf[2289]: segfault at 0 ip
000000392692ff7f sp 00007fffca7118a8 error 4 in libc-2.12.so[3926800000+186000]
Apr 26 17:07:25 sferic kernel: gpmap_gf[2286]: segfault at 0 ip
000000392692ff7f sp 00007fffef00ac98 error 4 in libc-2.12.so[3926800000+186000]
Apr 26 17:07:25 sferic kernel: gpmap_gf[2303]: segfault at 0 ip
000000392692ff7f sp 00007fff92019618 error 4 in libc-2.12.so[3926800000+186000]
Apr 26 17:07:25 sferic kernel: Pid 2303(gpmap_gf) over core_pipe_limit
Greg
On Apr 30, 2012, at 12:10 PM, daryl herzmann wrote:
> On Mon, 30 Apr 2012, Greg Stossmeister wrote:
>
>> Does anyone generate a lot of individual NEXRAD level III products with
>> gpmap_gf? I'm trying to generate real-time plots of NOQ Reflectivity and NOU
>> Velocity from 30 radars in the midwest and its crashing my server after a
>> few hours, even when I only run 3 plots at a time. I'm running GEMPAK6.4.0
>> on a RHEL 6 machine with 64 GB of memory. I'm wondering what I'm doing wrong
>> and if someone has a better way of doing this.
>
> crashing your server, how? Exhausting memory? kernic panic? Are the
> processes not going away once running them? How are you running them?
>
> daryl
>
> --
> /**
> * Daryl Herzmann
> * Assistant Scientist -- Iowa Environmental Mesonet
> * http://mesonet.agron.iastate.edu
> */
-----------------------------------------------------------
Greg Stossmeister
Head - Collaborative Tools and Metadata Group
NCAR/Earth Observing Laboratory (EOL)
Telephone: (303)497-8692 Facsimilie:(303)497-2044
e-mail: gstoss@xxxxxxxx
Mailing Address: Shipping Address:
NCAR/EOL UCAR Foothills Lab 1
P.O. Box 3000 3450 Mitchell Ln
Boulder, CO 80307 Boulder, CO 80301
------------------------------------------------------------
