[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: motherlode & queue size

On Mon, 31 Jul 2000, Tom McDermott wrote:

> 
> On Mon, 31 Jul 2000, Robb Kambic wrote:
> 
> > I not exactly sure why you are/were receiving the extra products.  The
> > allow lines on motherlode are UNIDATA|FLS2 for all top tier sites
> > including PSU. Maybe SUNY was getting some extra products, ie radars that 
> > were being relayed down the pipe. I would change my request line to only
> > be UNIDATA|FSL2 and then see if that helps.   Also, if you run notifyme
> > maybe you can determine what the extra products are?  You are not seeing
> > NMC2 products?
> 
> I'm not sure I am getting any extra products, which is why I asked about
> the data rates.  Last Thurs our ldm data disk was 82% full (8.44Gb) vs.
> 83% full (8.62Gb) now, and most of that increase is accounted for by the
> increase in the queue size itself.  So if there are extra products in the
> queue, they're not being selected by my 'pqact.conf'.
> 

Tom,

After reading your message about request lines, I don't believe you are
receiving extra products.



> Here are my request lines to Albany from ldmd.conf:
> 
> request       MCIDAS|FSL2     ".*"    redwood.atmos.albany.edu
> request       IDS|DDPLUS      ".*"    redwood.atmos.albany.edu
> request HDS
> "(^[B-FHJ-NQRTUW-Z])|(^A[A-FH-Z])|(^IUP)|(^S[A-EG-Z])|(^[YZ].[HIQR])"
> redwood.atmos.albany.edu
> 
> These are the same as I am using for Cornell, so even if Albany was
> receiving additional products (I know they get NMC2), I shouldn't be
> seeing them unless they're within the scope of my requests.  And I am
> filtering out stuff on HDS, so I'm not getting everything available.
> 
> > > BTW, our max. latencies for the times when the 12Z model runs were being
> > > transmitted were awesome today, <4 minutes, vs. the usual >30 minutes and
> > > possibly this may be related to the changeover, we'll have to see.
> > > 
> > That's great, thelma was slow because of the high disk I/O problem. You
> > could be correct that the data rate might of pushed more of the model data
> > into one hour but an 200MB queue should of been large enough.
> 
> However, we do get BUF NIDS plus 3 floaters plus MCIDAS, FSL2, NLDN and
> DIFAX, although 200mb may be sufficient for just WMO.
> 
> I have an additonal concern related to this.  Ever since the extended
> hours were added to the ETA and AVN back in April or whenever, Cornell's
> ldm logs fill up with the 'Deleting oldest' messages when the models runs
> are being received.  This indicates to me that their queue may not be
> large enough, at least for those peak hours.  When Cornell connects to
> motherlode (they are still connected to thelma I believe), if they
> experience the same phenomenon as we do, I expect this problem will be
> exacerbated. I don't know how much it slows things down for the ingestor
> process to have to delete entries from the queue.  Possibly this may also
> be a factor in the improved latencies, although as I said before, we'll
> have to see if it persists.
> 

You are correct that Cornell's queue is not large enough for model peak
times.  The latest LDM that's not released doesn't use pqexpire so this
should not be a problem in the near future. It uses a algorithmn to make
space as products  arrive. The LDM release should be out in about a month.

There is definitly a cost for receiving duplicate streams from different
source sites. The product duplication for large products >16k happens on
the upstream node, but the product duplication for <16k are done on the
downstream node. ie the product is transferred then thrown away.  It's
actually better to just receive the products from the one reliable node.


Robb....


> Tom
> ------------------------------------------------------------------------------
> Tom McDermott                         Email: address@hidden
> System Administrator                  Phone: (716) 395-5718
> Earth Sciences Dept.                  Fax: (716) 395-2416
> SUNY College at Brockport
> 
> 

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================