[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: CONDUIT latencies

Fri, 16 Jul 1999, Robb Kambic wrote:

> David,
> 
> There are Linux machines here at UCAR receiving NMC2 feed, Nexrad, etc.
> We have found out that it's NOT possible to receive the above feeds on one
> box because of the mmap memory management problems. These machine have
> gigabyte of RAM and couldn't handle it. My suggestion is too put at least
> 750 megabytes of RAM on the box receiving the NMC2 feed. In my opinion,
> RAM is cheap compared to your time. Some day the Linux programmers will
> fix the problem, until then use more RAM.  
> 
> Robb...

Granted, Linux's mmap is broke....but until its fixed there can
be workarounds.

Adding more memory doesn't necessarily cure the problem, it only makes the
system more usable.  You can still be hit with the wait to flush everything
out on exit problem. 

I'm going to step out on a limb and say I've found a software
fix to this problem.   Defining _MAPRGNS when compiling seems
to change this troublesome behaviour completely.  pq.c is
a bit hard to follow so I'm not sure exactly what going on
but it seems that mapping regions at a time takes care of
this.  Whenever a region is unmapped its freed to be written
to disk.  This does not seem to seriously impact performance
nor do I know of other negative side effects.  I've been 
running it on both the NMC2 ingest machine and our main
ingest machine (FOS,MCIDAS,NEXRAD,NLDN, DIFAX) etc for
a few weeks and have not had trouble.  Now when you type
"ldmadmin stop" the processes exit instantly.


Here's the patch to the configure script:

*** configure   Fri Apr 16 20:39:56 1999
--- configure.new       Wed Jun  9 21:47:17 1999
***************
*** 1696,1701 ****
--- 1696,1702 ----
        fi
      elif test -w /dev/conslog; then
        cat >> confdefs.h <<\EOF
+ #define _MAPRGNS 1
  #define _DEV_CONSLOG 1
  EOF




--------------------------------------------------------
 David Wojtowicz, Research Programmer/Systems Manager
 Department of Atmospheric Sciences Computer Services
 University of Illinois at Urbana-Champaign
 email: address@hidden  phone: (217)333-8390
--------------------------------------------------------