[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20091021: Nexrad Level II decoder



Hiya,

I been revising the makeRadarII script to make as many complete products as possible. I started a log on incomplete products, I discovered that eventually the complete product arrives, it could be as much as 30 minutes after the first piece arrives. So here's what I've been doing, if it's an incomplete product, I wait until the next product arrives and then make a product in the live directory from the pieces that have already arrived. Most of the time the pieces are contiguous up to some point. Then I keep checking everytime an end piece arrives if I can make a complete product for the incomplete pieces. When all the pieces have arrived, a complete product is made and all the pieces are removed. This makes partial products available soon after the first piece arrives but eventually the partial products are replaced with a complete product. I've been running the script all day and there are no incomplete products that are over 30 minutes old. I feel confident that the old script should be replace with this new script, what do you think?


Robb...

~ldm/logs/makeRadarII.log:

KFTG 20091030 1621 Partial product created
KFTG 20091030 1651 Partial product created
KFTG 20091030 1721 Partial product created
KFTG 20091030 1908 Partial product created
KFTG 20091030 2004 Partial product created
KFTG 20091030 2008 Partial product created
KFTG 20091030 2034 Partial product created
KFTG 20091030 2038 Partial product created
KFTG 20091030 2103 Partial product created
KFTG 20091030 2108 Partial product created


On Thu, 22 Oct 2009, Jeff Weber wrote:

Thanks for the detailed response :)

Heck, why meet, Robb...is this something you think you could handle?

If not, let's get together Thurs aft

Thanks,

Jeff
---

On Wed, 21 Oct 2009, Tom Yoksas wrote:


Hi Jeff,

re:
I would like to meet to discuss the writing of a decoder to insure proper
order and complete receipt of the full scan of level II radar files.

Anytime thurs or fri works for me.

I have an Africa Initiative meeting from 11:30 - ?? tomorrow (Thursday),
but any time after that or on Friday is OK with me.

It seems to me that the pieces of a Level II volume scan can be written
to disk and then assembled after all have been received.  The tricky
part is that one needs to receive the final piece to know how many
pieces there are/should be and then know how long to wait before
deciding that all of the pieces have not been received before exiting.

Here is a snippit of a 'notifyme' listing for KFTG today (it just so
happens that the sequence of pieces for this volume scan are in
order):

Oct 21 19:08:26 notifyme[3769] INFO:    12632 20091021190220.884 NEXRAD2 88001  
L2-BZIP2/KFTG/20091021190220/88/1/S/V03/0
Oct 21 19:08:27 notifyme[3769] INFO:    41835 20091021190222.965 NEXRAD2 88002  
L2-BZIP2/KFTG/20091021190220/88/2/I/V03/0
Oct 21 19:08:27 notifyme[3769] INFO:    60408 20091021190225.064 NEXRAD2 88003  
L2-BZIP2/KFTG/20091021190220/88/3/I/V03/0
Oct 21 19:08:27 notifyme[3769] INFO:    61916 20091021190229.154 NEXRAD2 88004  
L2-BZIP2/KFTG/20091021190220/88/4/I/V03/0
 ...
Oct 21 19:08:28 notifyme[3769] INFO:    18450 20091021190615.685 NEXRAD2 88067  
L2-BZIP2/KFTG/20091021190220/88/67/I/V03/0
Oct 21 19:08:28 notifyme[3769] INFO:    16791 20091021190622.752 NEXRAD2 88068  
L2-BZIP2/KFTG/20091021190220/88/68/I/V03/0
Oct 21 19:08:28 notifyme[3769] INFO:    16010 20091021190626.810 NEXRAD2 88069  
L2-BZIP2/KFTG/20091021190220/88/69/I/V03/0
Oct 21 19:08:28 notifyme[3769] INFO:    16232 20091021190636.732 NEXRAD2 88070  
L2-BZIP2/KFTG/20091021190220/88/70/E/V03/0

Notice that the last piece contains both the end of product indicator
and the exptected number of pieces:

.../KFTG/20091021190220/88/70/E/V03/0
                           ^  ^______ end piece indicator
                           |_________ piece number/exptected number of pieces

The product ID for the last piece shows that we should expect 70 pieces
for this particular product.  If the pieces were being written to
individual files, then it is easy enough to sort the files and cat them
into a single output file.

I suggest that the assembly process/"decoder" be kicked off upon
receipt of the end of product piece.  It would then sort the pieces it
has received and create an output file and delete individual pieces_if_
all of the pieces are accounted for.  If one or more piece is missing,
it could reschedule itself in 'n' minutes (1 minute?) to check again.
If a rescheduling approach is taken, then the assembler/"decoder"
should pass itself a count of how many times it has been rescheduled,
and give up after some reasonable number (like 5?) of attempts.  On the
final pass, it should create an output file that is as complete as it
is likely to be, and provide some indication that the file is not
complete (log file output, or a special suffix, etc.).

The above scheme is not foolproof: what happens if the missing piece(s)
are received after the assembly process has run the maximun number of
times?  I guess that the pieces could be resorted and then concatenated
into an output file that is then complete, but this would mean that one
would need to keep the pieces on disk for some undetermined amount of
time and kick off checking upon receipt of each piece.  It is my belief
that checking upon receipt of each piece would not work as the system
would get bogged down.  I say this because I tried logging the arrival
of each Nexrad Level III product, and the system rapidly became unusable
because of I/O load.

Cheers,

Tom
--
+----------------------------------------------------------------------------+
* Tom Yoksas                                            UCAR Unidata Program *
* (303) 497-8642 (last resort)                                 P.O. Box 3000 *
* address@hidden                                  Boulder, CO 80307 *
* Unidata WWW Service                            http://www.unidata.ucar.edu/*
+----------------------------------------------------------------------------+



===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================