[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20021028: [Fwd: 20021026: Unidata lossless PNG compression of NOAAPORT image products (correction)]

>From:  "Barbara Hickman" <address@hidden>
>Organization:  NOAA
>Keywords:  200210281539.g9SFd5q15745

Hi Barb,

re: PNG compression techniques used at the UPC

>Do you have any similar information on the timing delays per product
>introduced with PNG?

No, but my feeling is that they can't be very long.  The system we use
for NOAAPORT ingest (one we built ourselves) has very little buffering
(4 MB), and that buffering is used by all 4 channels.  This means that
-- at a maximum --, 3.5 T1s of data is pouring into that 4 MB buffer,
and no products are being lost during ingest.  To me, this says that
that the on-the-fly PNG compression that we are running (we are reading
image products out of the ingest card buffer and compressing the image
bytes as they become available) does not add any significant time delay
to the injection of any NOAAPORT product into our IDD.\

I don't think that you can use our time delay experience to judge what
the impact would be on your end, however.  I can put together some
numbers of how long it takes to PNG compress full NOAAPORT images
stored in files if you like, but this will take a little time to pull
together.  I can more quickly put together some numbers on the time it
takes to PNG compress images stored in McIDAS AREA files.

My opinion is that our PNG compression will be significantly faster
than wavelet compression techniques since it is a lot less CPU
intensive.

>I'm also curious as to where/how you obtained the
>compression utility package (i.e. cost & source)

The PNG stuff is like Zlib (and uses Zlib): it is available
free-of-charge.  The PNG Home Page (which is linked off of the Zlib
Home Page) is:

http://www.libpng.org/pub/png/

The following entry in the PNG FAQ page sheds light on PNGs availability:

Q: I thought you said PNG was patent-free--what about Stac, PKWARE, Apple, etc.?

  A: The PNG image format was designed in 1995 specifically in response to
     the patent problems with the LZW algorithm used in GIF. To the best of
     the PNG group's knowledge, PNG was then--and still
     is--completely patent-free. 

     However, the fact that is possible to implement PNG without infringing
     any known patents certainly does not imply that it is impossible to
     implement it without infringing any patents. Patents are a minefield,
     and several are known to be closely related to PNG technologies: 

     o Stac, LZ77 sorted hash, deflate algorithm 
     o PKWARE, whatever, deflate algorithm 
     o Apple, alpha mask images 

>and how it is tuned/configured for your usage,

We use a standard build of PNG in our compression routines.  We have
not done any additional tuning (yet) even though this would likely
improve the compression ratios that we can get for the NOAAPORT
products.  Customization is an important point.  The PNG Home Page
gives several examples of how people have created custom filters that
greatly improve compression of specific types of data.  Some of the
results listed are spectacular.

Our compression application simply reads the input, writes out the
unaltered header and then compresses the image data and appends it to
the output.  I include a schematic of the procedure at the end of this
note.

>type of platform on which it is run,

Our PNG compression/uncompression routines run on all of the OSes we
support for our applications:

  Compaq Tru64 5.1
  FreeBSD 4.[57]
  HP-UX B 11.00
  IBM AIX 4.3, 5.1
  RedHat Linux 6.x, 7.x (we will look at 8.x soon)
  SGI IRIX, IRIX64 6.5.x
  Sun Solaris 5.[6789] (SPARC and x86)
  
I have not tried this on Windows, but I can't say for sure that it
would work there, but given that PNG compressed images are supported in
Microsoft IE, I have to imagine that it would.

>how the platform is equipped, etc.

The code runs on the wide variety of platforms in use at our sites.
This ranges from 200 Mhz Linux PCs with 64 MB of RAM to Sun Enterprise
E450s and Sun Blade 1000, etc.

>Your overall numbers look
>consistent with our generalized product findings (Vis - least, WV -
>most, product subsat, resolution, and coverages by channel also play
>into the rates achieved, etc.) although the mean ratios do look
>significantly better.

I am glad that our results are consistent.

>Right now, though, NWS is specifically working with zlib since they
>believe that it is more readily avaialable and will have less impact to
>both government AWIPS users and commercial NOAAport users.

PNG uses Zlib compression techniques.  Both packages are freely
available and run on a very wide range of OSes (lots more than we
support our applications on).  Before dismissing PNG in favor of a
Zlib-only approach, I would hope that you read through the PNG Home
Page information and consider that display of PNG compressed imagery is
supported in virtually all current web browsers.  As far as the last
comment goes, our sites can strip off the uncompressed header of the
PNG compressed files we send in our IDD and look at the images in their
web broswer.  This makes it extremely easy for them to take quick looks
at imagery without even uncompressing it.

Our PNG compression applications effectively do the following:

NOAAPORT GINI:


       +-----------------------+                +-------------------------+
       |        Header         |                |          Header         |
       +-----------------------+                +-------------------------+
       |                       |                |                         |
       |                       |      PNG       |                         |
       |     Uncompressed      |     =====>     |        Compressed       |
       |         Data          |                |           Data          |
       |                       |                |                         |
       |                       |                |                         |
       |                       |                +-------------------------+
       |                       |
       |                       |
       |                       |
       |                       |
       |                       |
       +-----------------------+


McIDAS AREA:

       +-----------------------+                +-------------------------+
       |        Header         |                |          Header         |
       +-----------------------+                +-------------------------+
       |         Nav           |                |            Nav          |
       +-----------------------+     PNG        +-------------------------+
       |         Cal           |    =====>      |            Cal          |
       +-----------------------+                +-------------------------+
       |         AUX           |                |            AUX          |
       +-----------------------+                +-------------------------+
       |                       |                |      Comment Cards      |
       |                       |                +-------------------------+
       |                       |                |                         |
       |                       |                |                         |
       |     Uncompressed      |                |        Compressed       |
       |         Data          |                |           Data          |
       |                       |                |                         |
       |                       |                |                         |
       |                       |                +-------------------------+
       |                       |
       |                       |
       |                       |
       |                       |
       |                       |
       +-----------------------+
       |    Comment Cards      |
       +-----------------------+


We keep both the GINI and AREA header information uncompressed so that
the header information can be used without uncompressing it first.  For
instance, a McIDAS ADDE server for either only needs the header
information to do things like IMGLISTs.  The data only has to be
uncompressed when it is needed by things like IMGDISP, IMGCOPY,
IMGREMAP, etc.

Tom

>
--
+-----------------------------------------------------------------------------+
* Tom Yoksas                                             UCAR Unidata Program *
* (303) 497-8642 (last resort)                                  P.O. Box 3000 *
* address@hidden                                   Boulder, CO 80307 *
* Unidata WWW Service                             http://www.unidata.ucar.edu/*
+-----------------------------------------------------------------------------+