Karen,
There's a name I haven't seen for awhile, great to see your name again!1
My queues are typically for the processing of real-time data. I don’t
generally keep an hour of data in the queue because I have redundancy
and in general if data coming across is more than 20-30 minutes old I’m
going to ignore it anyway. I know that most data is archived
elsewhere and I’m not interested in keeping an archive— just wanting to
process what I get in real-time.
Here's wher it get interesting... or perhaps this is just an old man
that gets excited about silly things... :) I like both having the dish
for the speediness of the delivery, but also to backup friends down the
road.. When I was a professor it was backing up others way back in the
day that there were only a few of us out there.. like gerry at A&M had
the massive feeds going... Well now I'm not a professor anymore, but I
like backing up other weather peeps in the commercial sector too :)
So, if I have one of my noaaport relays well here.. let me elaborate:
for those that may not remember I take the feed from the dish to the
novra, then split that multicast feed from the novra to two (minimum)
noaaport relays.. the job of the noaaport relays is to balance the feed
to all of my production servers (where the actual work on the data is
done) ... well, I also have friends out there that depend on timely
data, and as an old unidata member, I like to share, and help others
too, who in turn help me :) So those that have a downstream backup
feed from me, may have an issue with their setup and need to reboot
their system, or start up another and need that data they missed... here
is where it gets critical...
If someone is making a product, and had a hard time getting wherever
they needed to go to boot up their ingest feed, and I am one of their
backups, or if it's a university setup that backs up from me that is
running research, or whatever.... how much data should be kept in the
queu e for them to fire up?
In the beginning years ago 500m was fine... but by a few years ago we
needed several gig to cover an hour, and we have since had goesR added
and more model products added...
what I do on each of my boxes is build the ramdisk to keep the queue in
because it's much much faster (less io and such) at xferring data from
the queue to be processed on the fly triggered from receipt, but also
because someone that might be downstream from me as a backup might need
to pull some data too, though I setup my backup feeds from the relays
and not the production servers...
anyway, I'm just one of the weird guys that thinks about these things,
and years ago more people would chiume in and chat about things I guess
lol :)
so that's my 2 cents, with your 2 cents, so does that mean that together
we make less sense? :)
thanks for chiming in karen.. good to see your name again :)
cheers,
--patrick
