[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #KOV-575304]: NEXRAD2 data

Hi John,

re: queue size larger than physical RAM
> Ok, I dropped the queue size down to 6GB.  Is that an issue with paging?

Yes.

re:
> Just curious why it needs to be less than the physical RAM size of the system.

We have used an 8 GB queue on a machine with 6 GB of RAM, and we have
observed a bit of disk thrashing.  This will eventually cause/help
the disks to fail, so it is typically not a good thing.

Question:

- why do you want to have such a large queue on a machine receiving
  data?

  Larger queues provide a couple of things:

  - rejection of duplicate products received (this is a very good thing!)

  - greater residency time in the queue so that the local processing
    to be done has more time in which to do the processing

  - a bigger backing store of data that would allow downstreams being
    fed data to be taken offline for maintenance and then be able to
    get all of the data that they had REQUESTed

re:
> We're specing a new machine with 32 GB of RAM.  Is that a good target
> for keeping this thing running for the next 10 years?  Or do I need
> to look at more?

Saying what will be viable 10 years from now is WAY beyond my ken :-)

Part of an answer would depend on what your use for the machine will
evolve into.  I can tell you that the machines we use as the real server
backends for the top level IDD relay cluster we operate, idd.unidata.ucar.edu,
each have at least 64 GB of RAM.  The reason for this is that we want our
LDM queue to be large enough to hold 3 hours of data.  The problem is that
the volume of data being relayed through the IDD keeps climbing (through
there being more and higher resolution model output; more and higher
resolution satellite imagery; etc.).  Where this will be in 10 years
is anyone's guess.  I feel confident, however, in saying that in 10
years most folks will _not_ want to ingest all of the data that they
want to use in education and research... instead, they will want to
be able to access those data transparently, reliably and quickly from
"the net".  Creating and deploying server software is one of the things
we have been doing for at least a decade, and its impact is continually
growing.

re:
> Thanks!

No worries.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: KOV-575304
Department: Support IDD
Priority: Normal
Status: Closed