On Sat, 10 May 2014, Blair Trosper wrote:
Specifically, we seem to be waiting on VERY the last chunk of each scan.
You should run a command like this and see when the LDM queue insert time
is for each chunch of the scan:
$ pqcat -v -l - -f NEXRD2 -p 'AMA/20140511172234' > /dev/null
May 11 17:52:40 pqcat NOTE: Starting Up (2051)
May 11 17:52:40 pqcat INFO: 8768 20140511172237.031 NEXRAD2 221001
L2-BZIP2/KAMA/20140511172234/221/1/S/V06/0
May 11 17:52:40 pqcat INFO: 114024 20140511172248.126 NEXRAD2 221002
L2-BZIP2/KAMA/20140511172234/221/2/I/V06/0
May 11 17:52:40 pqcat INFO: 120607 20140511172300.220 NEXRAD2 221003
L2-BZIP2/KAMA/20140511172234/221/3/I/V06/0
May 11 17:52:40 pqcat INFO: 125158 20140511172312.319 NEXRAD2 221004
L2-BZIP2/KAMA/20140511172234/221/4/I/V06/0
May 11 17:52:40 pqcat INFO: 112504 20140511172322.406 NEXRAD2 221005
L2-BZIP2/KAMA/20140511172234/221/5/I/V06/0
May 11 17:52:40 pqcat INFO: 137939 20140511172335.522 NEXRAD2 221006
L2-BZIP2/KAMA/20140511172234/221/6/I/V06/0
May 11 17:52:40 pqcat INFO: 118930 20140511172347.626 NEXRAD2 221007
L2-BZIP2/KAMA/20140511172234/221/7/I/V06/0
May 11 17:52:40 pqcat INFO: 48809 20140511172359.683 NEXRAD2 221008
L2-BZIP2/KAMA/20140511172234/221/8/I/V06/0
May 11 17:52:40 pqcat INFO: 49433 20140511172411.737 NEXRAD2 221009
L2-BZIP2/KAMA/20140511172234/221/9/I/V06/0
May 11 17:52:40 pqcat INFO: 54736 20140511172424.792 NEXRAD2 221010
L2-BZIP2/KAMA/20140511172234/221/10/I/V06/0
May 11 17:52:40 pqcat INFO: 49576 20140511172434.842 NEXRAD2 221011
L2-BZIP2/KAMA/20140511172234/221/11/I/V06/0
May 11 17:52:40 pqcat INFO: 57399 20140511172446.900 NEXRAD2 221012
L2-BZIP2/KAMA/20140511172234/221/12/I/V06/0
May 11 17:52:40 pqcat INFO: 49768 20140511172458.954 NEXRAD2 221013
L2-BZIP2/KAMA/20140511172234/221/13/I/V06/0
May 11 17:52:40 pqcat INFO: 65977 20140511172512.027 NEXRAD2 221014
L2-BZIP2/KAMA/20140511172234/221/14/I/V06/0
May 11 17:52:40 pqcat INFO: 76160 20140511172524.105 NEXRAD2 221015
L2-BZIP2/KAMA/20140511172234/221/15/I/V06/0
May 11 17:52:40 pqcat INFO: 82324 20140511172536.182 NEXRAD2 221016
L2-BZIP2/KAMA/20140511172234/221/16/I/V06/0
May 11 17:52:40 pqcat INFO: 79884 20140511172547.262 NEXRAD2 221017
L2-BZIP2/KAMA/20140511172234/221/17/I/V06/0
May 11 17:52:40 pqcat INFO: 81238 20140511172559.339 NEXRAD2 221018
L2-BZIP2/KAMA/20140511172234/221/18/I/V06/0
May 11 17:52:40 pqcat INFO: 68100 20140511172611.411 NEXRAD2 221019
L2-BZIP2/KAMA/20140511172234/221/19/I/V06/0
May 11 17:52:40 pqcat INFO: 26953 20140511172623.462 NEXRAD2 221020
L2-BZIP2/KAMA/20140511172234/221/20/I/V06/0
May 11 17:52:40 pqcat INFO: 31827 20140511172636.522 NEXRAD2 221021
L2-BZIP2/KAMA/20140511172234/221/21/I/V06/0
May 11 17:52:40 pqcat INFO: 34510 20140511172646.580 NEXRAD2 221022
L2-BZIP2/KAMA/20140511172234/221/22/I/V06/0
May 11 17:52:40 pqcat INFO: 33948 20140511172658.634 NEXRAD2 221023
L2-BZIP2/KAMA/20140511172234/221/23/I/V06/0
May 11 17:52:40 pqcat INFO: 33823 20140511172711.696 NEXRAD2 221024
L2-BZIP2/KAMA/20140511172234/221/24/I/V06/0
May 11 17:52:40 pqcat INFO: 27970 20140511172723.748 NEXRAD2 221025
L2-BZIP2/KAMA/20140511172234/221/25/I/V06/0
May 11 17:52:40 pqcat INFO: 49144 20140511172735.805 NEXRAD2 221026
L2-BZIP2/KAMA/20140511172234/221/26/I/V06/0
May 11 17:52:40 pqcat INFO: 56772 20140511172748.866 NEXRAD2 221027
L2-BZIP2/KAMA/20140511172234/221/27/I/V06/0
May 11 17:52:40 pqcat INFO: 60996 20140511172758.926 NEXRAD2 221028
L2-BZIP2/KAMA/20140511172234/221/28/I/V06/0
May 11 17:52:40 pqcat INFO: 60507 20140511172810.994 NEXRAD2 221029
L2-BZIP2/KAMA/20140511172234/221/29/I/V06/0
May 11 17:52:40 pqcat INFO: 56882 20140511172824.058 NEXRAD2 221030
L2-BZIP2/KAMA/20140511172234/221/30/I/V06/0
May 11 17:52:40 pqcat INFO: 50683 20140511172834.122 NEXRAD2 221031
L2-BZIP2/KAMA/20140511172234/221/31/I/V06/0
May 11 17:52:40 pqcat INFO: 20903 20140511172847.166 NEXRAD2 221032
L2-BZIP2/KAMA/20140511172234/221/32/I/V06/0
May 11 17:52:40 pqcat INFO: 24209 20140511172859.212 NEXRAD2 221033
L2-BZIP2/KAMA/20140511172234/221/33/I/V06/0
May 11 17:52:40 pqcat INFO: 25977 20140511172912.257 NEXRAD2 221034
L2-BZIP2/KAMA/20140511172234/221/34/I/V06/0
May 11 17:52:40 pqcat INFO: 26113 20140511172922.306 NEXRAD2 221035
L2-BZIP2/KAMA/20140511172234/221/35/I/V06/0
May 11 17:52:40 pqcat INFO: 24476 20140511172934.358 NEXRAD2 221036
L2-BZIP2/KAMA/20140511172234/221/36/I/V06/0
May 11 17:52:40 pqcat INFO: 21243 20140511172947.407 NEXRAD2 221037
L2-BZIP2/KAMA/20140511172234/221/37/I/V06/0
May 11 17:52:40 pqcat INFO: 55142 20140511173012.497 NEXRAD2 221038
L2-BZIP2/KAMA/20140511172234/221/38/I/V06/0
May 11 17:52:40 pqcat INFO: 58391 20140511173034.590 NEXRAD2 221039
L2-BZIP2/KAMA/20140511172234/221/39/I/V06/0
May 11 17:52:40 pqcat INFO: 55245 20140511173059.686 NEXRAD2 221040
L2-BZIP2/KAMA/20140511172234/221/40/I/V06/0
May 11 17:52:40 pqcat INFO: 48265 20140511173123.764 NEXRAD2 221041
L2-BZIP2/KAMA/20140511172234/221/41/I/V06/0
May 11 17:52:40 pqcat INFO: 48377 20140511173146.835 NEXRAD2 221042
L2-BZIP2/KAMA/20140511172234/221/42/I/V06/0
May 11 17:52:40 pqcat INFO: 48771 20140511173210.908 NEXRAD2 221043
L2-BZIP2/KAMA/20140511172234/221/43/E/V06/0
May 11 17:52:40 pqcat NOTE: Exiting
May 11 17:52:40 pqcat NOTE: Number of products 43
So that shows the last chunk of the 17:22:34 scan being inserted at
17:32:10
The next scan then comes ten seconds later 17:32:20
May 11 17:51:25 pqcat INFO: 8677 20140511173220.652 NEXRAD2 222001
L2-BZIP2/KAMA/20140511173217/222/1/S/V06/0
Are you able to show the above with a 10 minute lag for the last chunk?
Perhaps you are seeing some irregularity with how data is getting handed
off to your ingester.
daryl
--
/**
* Daryl Herzmann
* Assistant Scientist -- Iowa Environmental Mesonet
* http://mesonet.agron.iastate.edu
*/
