[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

19990129: lzo1c_2, radar feed using LDM

------- Forwarded Message

>To: address@hidden
>cc: address@hidden
>From: Harry Edmon <address@hidden>
>Subject: Re: lzo1c_2
>Organization: .
>Keywords: 199901292229.PAA10576

I picked lzo1c_2 as a compromise between compression and speed - but if the CPU
is not too loaded, there is no reason to go to higher compression.  However,
this would have to be coordinated between the ingestor and compressor.

What we may want to do is change either a byte in the data feed, or the data
header to indicate what compression is used to allow the correct decompression
algorithm to be used.  I want to change the headers anyway to match some
suggestions from Unidata.  Let's try to coordinate this.  I will do some
experiments on my end next week and then we can talk about the changes.

Have a good weekend.

James Deaton <address@hidden> wrote:
> I've been working with the CRAFT group and a current issue with the
> radar data overflowing the 56kbps circuit we have setup for it.  I'm
> curious about the other compression algorithms within the 'lzo'
> framework.  Jason Levit told me I needed to talk with you.
> 
> I noticed you utilized the 'lzo1c_2' version.  I'm wondering if we could
> drop in one of the higher compression ratio versions such as a high-end
> 'lzo1x_9' or even 'lzo1x_999'.  I have no experience dealing with the
> 'lzo' package and until I noticed the additional algorithms, had started
> down the path looking at some alternative systems.  In a couple of small
> tests that I've done, I found that the lzo1x_9 compressed a 9.8MB file
> down to 2.3MB file when lzo1c_2 compressed the same file down to 2.7MB.
> This difference is critical considering the 350 second window between
> files requires ~63kbps with the 2.7MB file and ~53kbps for the 2.3MB
> file.
> 
> Are there any gotchas that we need to be concerned with by simply making
> the relevent changes in the compress and decompression calls?  I figure
> we'd only need to make the change at the ingest device and most likely
> add an additional routine to convert back to the lzo1c scheme for the
> stored, compressed files at the collection machine.
> 
> Thanks,
> 
>                James E Deaton   |  ONENET  |  address@hidden
>             800 NE 15th Suite 532 Oklahoma City, Oklahoma 73104
>             Phone: 888-566-3638 |  KC5PRQ  |  Fax: 405-522-0823
> 
> 


--
Dr. Harry Edmon                 E-MAIL: address@hidden
(206) 543-0547                  FAX:    (206) 543-0308
Dept of Atmospheric Sciences
University of Washington, Box 351640, Seattle, WA 98195-1640

From address@hidden  Fri Jan 29 15:21:56 1999
        by unidata.ucar.edu (8.8.8/8.8.8) with ESMTP id PAA10474;
        Fri, 29 Jan 1999 15:21:56 -0700 (MST)
        by damp.atmos.washington.edu (8.9.1/8.9.1) id OAA26226;
        Fri, 29 Jan 1999 14:21:55 -0800 (PST)
From: Harry Edmon <address@hidden>
Message-Id: <address@hidden>
Date: Fri, 29 Jan 1999 14:21:55 -0800 (PST)
To: address@hidden
Subject: Re[3]: 19990114: Feed type for Nexrad level 2 data COMINGSOON/BLKDATA
X-Mailer: Ishmail 1.3.3-980626-sol <http://www.ishmail.com>
MIME-Version: 1.0
Content-Type: text/plain

I never heard back from you on this.  I am coordinating a change in the ingestor
with the Oklahoma folks, and this would be a great time to make a change.  Also,
you had suggested the NMC2 feed type, but that is what Chiz is using for the
feed from NCEP via GSFC.  How about the NMC3 type?  And perhaps there should be
a product ID in front of the station id in case we end up with more than one
type of data.  Perhaps it should be:

L2/IIII/YYYYMMDDHHMMSS/nnn/mm[/E]

Harry Edmon <address@hidden> wrote:
> The reason I break the product up is two-fold:
> 
> 
> 1. A volume takes from 6-10 minutes to produce.  I send the data as soon as it
> is received.  The partial product could be useful.
> 
> 2. Also, I am compressing the radar volume as I receive it, and each
> individual
> piece has been seperately compressed.
> 
> Given the above, it would make sense to leave it the way I have it.  Do you
> have
> any suggestions as to naming, perhaps instead of
> 
>       YYYYMMDDHHMMSS IIII nnn mm [E]
> 
> what would be better is:
> 
> IIII/YYYYMMDDHHMMSS/nnn/mm[/E]
> 
> 
> 
> --
> Dr. Harry Edmon                       E-MAIL: address@hidden
> (206) 543-0547                        FAX:    (206) 543-0308
> Dept of Atmospheric Sciences
> University of Washington, Box 351640, Seattle, WA 98195-1640


--
Dr. Harry Edmon                 E-MAIL: address@hidden
(206) 543-0547                  FAX:    (206) 543-0308
Dept of Atmospheric Sciences
University of Washington, Box 351640, Seattle, WA 98195-1640


------- End of Forwarded Message