[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20030813: access to NIMAGE datasetream



>From: =?ISO-8859-1?Q?Christian_Pag=E9?= <address@hidden>
>Organization: UQAM
>Keywords: 200308132035.h7DKZ1Ld004120 IDD NIMAGE

Hi Christian,

re: NOAAPORT remapped satellite imagery

>If I understand correctly, this NIMAGE datastream would be the same  
>remapping that I am accessing through ADDE?

Yes, exactly.  The ADDE servers you have been accessing are simply serving
the data in the NIMAGE stream.

>Getting it trough IDD is better than ADDE for what we want to do with  
>the data, since I can have ldmd exec scripts right when the images get  
>in, and being less dependent on the ADDE server I chose.

OK.  I setup atm.geo.nsf.gov to allow you to feed NIMAGE from it.
Given the volume in the stream, I would suggest that you carefully
monitor your disk usage after setting up the ingest/decode AND scouring.

Here is the email I sent to existing NIMAGE users relating to the new
ldm-mcidas zlibg2gini routine.  A lot of the information is stuff
you won't need as a new NIMAGE user, but it is a lot easier to send
the entire email than to edit it down to just what you need.

----- From address@hidden Wed Jul 30 13:37:14 2003 -----

>Keywords: 200307301619.h6UGJWLd013737 ldm-mcidas NIMAGE pngg2gini zlibg2gi

Users of the IDD NIMAGE datastream imagery:

The NWS is changing the format of the NOAAPORT GINI images:

  489
  NOXX10 KWBC 291730
  DATA MGT MESSAGE 07-03.08
  
  TO  AWIPS/NOAAPORT USERS.. FAMILY OF SERVICES/FOS/SUBSCRIBERS...
      EMERGENCY MANAGERS WEATHER INFORMATION NETWORK /EMWIN/
      NATIONAL WEATHER SERVICE /NWS/ CUSTOMERS... WAFS USERS...
      GTS USERS... NWS FAX CHART USERS
  
  FROM  RTH WASHINGTON DATA MANAGEMENT
  
  SUBJECT GOES COMPRESSION
  
  REFERENCES DATA MGT MESSAGE 04-03.01 - NOAAPORT EVOLUTION
             DATA MGT MESSAGE 07-03.01 - NOAAPORT EVOLUTION
                                         UPDATE
  
  THE GOES EAST AND WEST DATA FEEDS WILL BE COMPRESSED.
  DECOMPRESSION MAY BE ACHIEVED USING THE TECHNIQUE SHOWN
  IN THE TECHNICAL REPORT LISTED IN WHAT S NEW WITH NOAAPORT
  AT THE FOLLOWING URL ADDRESS
  
  HTTP/COLON///WWW.NWS.NOAA.GOV/NOAAPORT/HTML/WHATS NW.SHTML
  
  IT IS PLANNED TO COMPRESS THE GOES WEST DATA FEED BETWEEN 1000
  AND 1100 EDT ON 7-30-03.
  
  IT IS PLANNED TO COMPRESS THE GOES EAST DATA FEED BETWEEN 1000
  AND 1100 EDT ON 7-31-03.
  
  IF YOU HAVE ANY QUESTIONS REGARDING THIS NOTICE... PLEASE
  CONTACT
  
  PHIL CRAGG
  E-MAIL  PHIL.CRAGG/AT SYMBOL/NOAA.GOV
  PHONE NUMBER  /301/ 713-1570 X134
  
  DATA MANAGEMENT
  LOWER CASE
  
  THIS INFORMATION AS WELL AS THE REFERENCED NOTICES ARE POSTED
  ON THE NWS DATA MANAGEMENT NOTICES WEB PAGE. THE URL IS
  LISTED BELOW.
  
  HTTP /COLON/ //WWW.NWS.NOAA.GOV/DATAMGMT/INDEX.HTML
  
  DATA MANAGEMENT
  TELECOMMUNICATION OPERATIONS CENTER
  RTH WASHINGTON SENDS

The first of these changes occurred at approximately 16:10Z this morning.
The next set of changes, those to GOES-East imagery, will occur at
about the same time tomorrow, July 31.

A new, "early" release of the ldm-mcidas package of decoders has been
made available by anonymous FTP on the Unidata FTP server,
ftp.unidata.ucar.edu.  This ldm-mcidas release, version ldm-mcidas-2003,
contains a new decoder for the Zlib-compressed NOAAPORT NIMAGE images,
zlibg2gini.  The ldm-mcidas-2003 distribution is available in source
and select binary distributions:

machine: ftp.unidata.ucar.edu
user:    anonymous
pass:    your_full_email_address

directory/filename:

source:
-------
pub/ldm-mcidas/ldm-mcidas-2003.tar.Z

binaries:
---------
pub/binary/freebsd_4.7-i386/ldm-mcidas-2003.tar.Z
pub/binary/hpux_11.00-hp9000/ldm-mcidas-2003.tar.Z
pub/binary/irix_6.5-mips/ldm-mcidas-2003.tar.Z
pub/binary/linux8_2.4-i686/ldm-mcidas-2003.tar.Z
pub/binary/linux_2.4-i686/ldm-mcidas-2003.tar.Z
pub/binary/osf1_5.1-alpha/ldm-mcidas-2003.tar.Z
pub/binary/sunos_5.6-sparc/ldm-mcidas-2003.tar.Z
pub/binary/sunos_5.7-i86pc/ldm-mcidas-2003.tar.Z
pub/binary/sunos_5.8-i86pc/ldm-mcidas-2003.tar.Z
pub/binary/sunos_5.8-sparc/ldm-mcidas-2003.tar.Z
pub/binary/sunos_5.9-sparc/ldm-mcidas-2003.tar.Z

Those sites that have been using the ldm-mcidas-2002b PNG decoder,
pngg2gini, to decode NOAAPORT GINI imagery will want to download the
new ldm-mcidas release and switch to use of zlibg2gini.  Invocation
syntax of zlibg2gini is identical to pngg2gini.

The easiest way for you do upgrade your system is:

- logon to your system as the user running your LDM, typically 'ldm'

- download the binary distribution that best matches your Operating System
  NOTE: for most OSes, a binary built on an older OS version will run
  on newer versions

cd ~ldm
ftp ftp.unidata.ucar.edu
  <user> aonymous
  <pass> your_full_email_address
  cd pub/binary/<appropriate OS distribution>
  binary
  get ldm-mcidas-2003.tar.Z
  quit

- unpack ldm-mcidas-2003.tar.Z in the ~ldm directory:

zcat ldm-mcidas-2003.tar.Z | tar xvf -

- make sure that the new executables will run on your OS.  How this is
  done is OS dependent.  On Sun Solaris, Linux, and FreeBSD, one can use
  the 'ldd' utility:

cd ldm-mcidas-2003/bin
ldd pnga2area
ldd pngg2gini
ldd zlibg2gini
 etc.

- copy the ldm-mcidas decoders that you are currently using from the
  2003 distribution to the directory you are using in your 'ldm' account.
  Here is an example that for sites that put their decoders in
  ~ldm/decoders:

cd ~ldm/ldm-mcidas-2003/bin
cp area2png nldn2md pnga2area pngg2gini proftomd zlibg2gini ~ldm/decoders

- adjust your pqact.conf entry for NIMAGE decoding.  Here is one example:

change:

#
# PNG compressed NOAAPORT GINI images
NIMAGE  ^sat/ch[0-9]/.*/(.*)/([12][0-9])([0-9][0-9])([01][0-9])([0-3][0-9]) 
([0-2][0-9])([0-5][0-9])/(.*)/(.*km)/
        PIPE    -close
        decoders/pngg2gini -vl logs/ldm-mcidas.log
        data/pub/raw/gini/\8/\9/\1/\1_\2\3\4\5_\6\7

to:

#
# PNG compressed NOAAPORT GINI images
NIMAGE  ^sat/ch[0-9]/.*/(.*)/([12][0-9])([0-9][0-9])([01][0-9])([0-3][0-9]) 
([0-2][0-9])([0-5][0-9])/(.*)/(.*km)/
        PIPE    -close
        decoders/pngg2gini -vl logs/ldm-mcidas.log
        data/pub/raw/gini/\8/\9/\1/\1_\2\3\4\5_\6\7

#
# Zlib compressed NOAAOPRT GOES-East GINI images
NIMAGE  ^satz/ch[0-9]/.*/(.*)/([12][0-9])([0-9][0-9])([01][0-9])([0-3][0-9]) 
([0-2][0-9])([0-5][0-9])/(.*)/(.*km)/
        PIPE    -close
        decoders/zlibg2gini -vl logs/ldm-mcidas.log
        data/pub/raw/gini/\8/\9/\1/\1_\2\3\4\5_\6\7


  NOTES:

  1) the only changes in the zlibg2gini entry are:

    - sat -> satz
    - pngg2gini -> zlibg2gini

  2) it is likely that the directory structure you are using to save
     these images is different from the ones listed above.  Make changes
     where appropriate.

  3) we do not know if the NWS will be sending any Zlib compressed
     GOES-East imagery between now and tomorrow when the change is to
     be made

- check the integrity of the pqact.conf changes you make:

ldmadmin pqactcheck

- if you get no errors from the pqactcheck action above, then send a HUP
  to pqact telling it to reread your pqact.conf file:

ldmadmin pqactHUP
ldmadmin tail

- make sure that the new decoders are working:

cd ~ldm
less logs/ldm-mcidas.log              <- adjust for whatever log file you are
                                         using


New Unidata releases of GEMPAK and McIDAS that support direct use of the
Zlib-compressed images are being readied for release.  After their release
and your installation of these packages, you may want to change your
pqact.conf action to FILE the image produts.  The disk savings you should
see by saving the Zlib compressed images will run around 40% over the
uncompressed images.

An example pqact.conf FILE action based on the actions above would look 
like:

#
# Zlib compressed NOAAOPRT GOES-East GINI images
NIMAGE  ^satz/ch[0-9]/.*/(.*)/([12][0-9])([0-9][0-9])([01][0-9])([0-3][0-9]) 
([0-2][0-9])([0-5][0-9])/(.*)/(.*km)/
        FILE    -overwrite      -close
        data/pub/raw/gini/\8/\9/\1/\1_\2\3\4\5_\6\7


Please send all questions/comments to Unidata User Support

Tom Yoksas


Since you are a new NIMAGE user, the part of importance is the
pqact.conf entry for uncompressing the Zlib-compressed NIMAGE images
using zlibg2gini.  In the original message, I had users keep an entry
for pngg2gini since the NWS as sending imagery in both compressed and
uncompressed formats.  They have now switched to just the
Zlib-compressed format, so the pngg2gini entry in pqact.conf is no
longer needed.

As far as ingest goes, the ldmd.conf entry is simple:

request NIMAGE  ".*"    atm.geo.nsf.gov PRIMARY

There is not great redundancy in the NIMAGE feed tree, so I will not
be providing you with a failover site just yet.

The other thing that will be important to you is the scouring script
we provide to sites ingesting the GINI data:

cd ~ldm/decoders
ftp ftp.unidata.ucar.edu
  <user> aonymous
  <pass> your_full_email_address
  cd pub/ldm/scour
  get prune_gini.csh
  quit

chmod +x prune_gini.csh

You will need edit prune_gini.csh and setup the top level directory
under which you will be storing the image data, and the number of
each type of image you want to keep.

>ADDE is really  
>great though for our interactive uses (courses and synoptic  
>presentations) since in these cases we can access data from "where the  
>action is", which is not necessarily the locations and datatypes we  
>look at or archive everyday for our research and educational uses.

The reason that the NIMAGE stream has not been made widely available
is that the number of sites that could take advantage of the data is
a smallish subset of the sites participating in the IDD.

Please letus know if you have any problems setting up the ingest, decode,
or scour of the NIMAGE data.

Cheers,

Tom