Re: [ldm-users] Double level 2 feed again...

  • Subject: Re: [ldm-users] Double level 2 feed again...
  • From: Karen Cooper <Karen.Cooper@xxxxxxxx>
  • Date: Thu, 09 Jul 2009 10:54:05 -0500

I receive a lot of data, and typically my queues are not big enough to hold an hour's worth of data. Right now I only have about 10-15 minutes of data in my queues, but I have not received any duplicated or delayed data.



Alan.Hall wrote:
Generally a larger volume of data and too small of a queue. I'm assuming a sustained rate of 6-10Mbits for the NEXRAD II feed, that means to keep an hours worth of data in the queue would take a queue size of 2.7GBytes to 4.5GBytes. Of course you don't have to have an hours worth of data in the queue as long as the pqact can process the data in the queue before they drop out of the queue. By processing I mean to pass it down stream and pqact.
You can multi-stream the pqacts and gain some performance.
When Dual Pol comes around you'll need a 9 GByte queue and then the AVSET might add another 20% onto that.
Alan.


Gilbert Sebenste said the following on 7/9/2009 10:40 AM:
On Thu, 9 Jul 2009, Gilbert Sebenste wrote:

Started yesterday, central region for sure, I haven't checked the other regions yet. Here are my latest files for KLOT (Romeoville/Chicago)...

-rw-rw-r--   1 ldm users 2255367 Jul  9 08:58 KLOT_20090709_1322

Notice, for example, the 1322Z data file is twice the normal size and came in about 40 minutes late.

I also just saw something using pqmon. (Darryl, just saw your message).

I would have to increase my queue size to nearly 6 GB to stay ahead of this. No, no, and heck, no! I'm trying to run my queue out of /dev/shm, to which I have 1.3 GB to do so. It says I am only saving "558" seconds worth of data with 1.2 GB of queue. That's insane! Even if the products get rejected, I still have to reject them by getting them first, correct?

OK. So when I use idd.aos.wisc.edu, flood.atmos.uiuc.edu, idd.unidata.ucar.edu or bigbird.tamu.edu singularly, that is, as the primary feed with no backups, I don't get problems. But I just tried it now with

request NEXRAD2 ".*" notam.atmos.uiuc.edu
request NEXRAD2 ".*" bigbird.tamu.edu

In order...and I get the double feed. I tried it in reverse, same thing (with bigbird as primary). Tried it with other sites...same results. And when I do an ldmadmin watch, look what I see:

Jul 09 14:36:01 pqutil INFO: 38876 20090709135846.462 NEXRAD2 49042 L2-BZIP2/KJAX/20090709135626/49/42/I/V03/0 Jul 09 14:36:01 pqutil INFO: 30687 20090709133606.451 NEXRAD2 626047 L2-BZIP2/KOAX/20090709133327/626/47/I/V03/0 Jul 09 14:36:01 pqutil INFO: 15952 20090709135846.302 NEXRAD2 507050 L2-BZIP2/KVNX/20090709135452/507/50/I/V03/0 Jul 09 14:36:01 pqutil INFO: 53219 20090709135845.193 NEXRAD2 570006 L2-BZIP2/KGWX/20090709135641/570/6/I/V04/0 Jul 09 14:36:01 pqutil INFO: 23850 20090709133606.815 NEXRAD2 906043 L2-BZIP2/KPUX/20090709133401/906/43/I/V03/0 Jul 09 14:36:01 pqutil INFO: 51674 20090709143600.836 NEXRAD2 533008 L2-BZIP2/KILX/20090709143434/533/8/I/V03/0 Jul 09 14:36:01 pqutil INFO: 18361 20090709133606.928 NEXRAD2 803027 L2-BZIP2/KIWA/20090709133443/803/27/I/V03/0 Jul 09 14:36:01 pqutil INFO: 45660 20090709135845.333 NEXRAD2 57012 L2-BZIP2/KRLX/20090709135810/57/12/I/V03/0 Jul 09 14:36:01 pqutil INFO: 15858 20090709143601.012 NEXRAD2 233018 L2-BZIP2/KBLX/20090709143228/233/18/I/V03/0 Jul 09 14:36:01 pqutil INFO: 21409 20090709135846.469 NEXRAD2 384016 L2-BZIP2/KLVX/20090709135726/384/16/I/V03/0 Jul 09 14:36:01 pqutil INFO: 10574 20090709135846.597 NEXRAD2 87029 L2-BZIP2/KFSD/20090709135708/87/29/I/V03/0 Jul 09 14:36:01 pqutil INFO: 32569 20090709143559.869 NEXRAD2 786017 L2-BZIP2/KINX/20090709143517/786/17/I/V03/0 Jul 09 14:36:01 pqutil INFO: 42332 20090709135847.305 NEXRAD2 701014 L2-BZIP2/KDMX/20090709135804/701/14/I/V03/0 Jul 09 14:36:01 pqutil INFO: 20409 20090709143600.580 NEXRAD2 61033 L2-BZIP2/KRLX/20090709142705/61/33/I/V03/0 Jul 09 14:36:01 pqutil INFO: 38611 20090709135847.733 NEXRAD2 777042 L2-BZIP2/KINX/20090709135647/777/42/I/V03/0 Jul 09 14:36:01 pqutil INFO: 10000 20090709143601.246 NEXRAD2 512014 L2-BZIP2/KARX/20090709143452/512/14/I/V04/0 Jul 09 14:36:01 pqutil INFO: 29056 20090709143559.812 NEXRAD2 689019 L2-BZIP2/KLZK/20090709143510/689/19/I/V03/0 Jul 09 14:36:01 pqutil INFO: 67204 20090709133606.339 NEXRAD2 709013 L2-BZIP2/KIND/20090709133526/709/13/I/V03/0

Obviously delayed duplicates, and I JUST cleared/deleted and remade the queues as the above was happening, so my queues weren't full. This never used to happen before. What's going on?

******************************************************************************* Gilbert Sebenste ******** (My opinions only!) ****** Staff Meteorologist, Northern Illinois University **** E-mail: sebenste@xxxxxxxxxxxxxxxxxxxxx *** web: http://weather.admin.niu.edu ** *******************************************************************************

_______________________________________________
ldm-users mailing list
ldm-users@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
ldm-users mailing list
ldm-users@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/

_______________________________________________
ldm-users mailing list
ldm-users@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/

--
-------------------------------------------

There are 2 kinds of people in the world:

1) Those who can extrapolate from incomplete data.

-------------------------------------------
Karen.Cooper@xxxxxxxx

Phone:  405-325-6982
Cell:   405-834-8559
INDUS Corporation
National Severe Storms Laboratory



  • 2009 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the ldm-users archives: