LDM-McIDAS:
Unidata-Wisconsin Changes


Impending Unidata-Wisconsin datastream changes

As announced in the summer of 2003, the contents of the Unidata-Wisconsin datastream are scheduled to be changed. The full set of changes to be made is detailed in Proposed Unidata-Wisconsin Datastream Changes.

The first portion of the changes are scheduled to be made at 18 UTC on Wednesday, January 7. These changes will include:

A second change is scheduled to be made on Friday, January 9. This change will consist of:

A third change will occur in late January:

Things to consider to accommodate the Unidata-Wisconsin changes

Modify your LDM request for the Unidata-Wisconsin imagery

The datastream changes will result in an increase in the number of products in the Unidata-Wisconsin datastream and corresponding increase in the datastream volume and disk space needed to store the images.

You can choose to ingest all or a portion of the broadcast images by tailoring the data request line for the UNIWISC (also known as MCIDAS) feed in your LDM's ldmd.conf file. To help you understand how you can tailor the request to suit your needs, we offer the following:

Each UNIWISC IDD product has two fields that can be used to tailor data requests:

Qn - hour portion indicator (e.g., Q0, Q1, Q2, Q3, Q4)

Un - McIDAS product routing code (e.g., UB, U5, U9, etc.)

Possible values for the hour portion indicator are as follows:

Q0 image products or composites; the time associated with the product has hourly resolution
Q1 images whose time lies between H+00 and H+15
Q2 images whose time lies between H+15 and H+30
Q3 images whose time lies between H+30 and H+45
Q4 images whose time lies between H+45 and H+60

McIDAS product code values for UNIWISC imagery are as follows (new products are highlighted in bold print):

CACIMSS Cloud Top Pressure
CBCIMSS Precipitable Water
CCCIMSS Sea Surface Temperature
CDCIMSS Lifted Index
CECIMSS CAPE
CFCIMSS Ozone
CGCIMSS Wildfire ABBA Nth Hemishere
CHCIMSS Wildfire ABBA Sth Hemishere
U1Antarctic composite
U3Manually Digitized Radar
U5GOES-W 10.7 um Infrared
U7GOES-W 3.9 um Infrared
U8GOES-W 12.0 um Infrared
U9GOES-W 0.65 um Visible
UAEducational floater I
UBGOES-W 6.8 um Water Vapor
UCEducational floater II
UDGOES-E 3.9 um Infrared
UEGOES-E 13.3 um CO2
UIGOES-E 10.7 um Infrared
URResearch floater
UVGOES-E 0.65 um Visible
UWGOES-E 6.5 um Water Vapor
UXGlobal Mollweide Infrared Composite
UYGlobal Mollweide Water Vapor Composite

Some example UNIWISC headers:

pnga2area Q0 CA 1100 GOES-10_SND UNKBAND 14km 20040104 2300
pnga2area Q1 U1 191 GOES-9_IMG UNKBAND 4km 20040104 2100
pnga2area Q1 U3 205 GRAPHICS UNKBAND 5km 20040104 2359
pnga2area Q1 U9 120 GOES-10_IMG 0.65um 4km 20040105 0000
pnga2area Q1 U5 130 GOES-10_IMG 10.7um 4km 20040105 0000
pnga2area Q1 UB 170 GOES-10_IMG 6.8um 8km 20040105 0000
pnga2area Q0 CF 1150 GOES-10_SND UNKBAND 20km 20040104 2300
pnga2area Q1 UW 210 GOES-12_IMG 6.5um 8km 20040105 0015
pnga2area Q1 UV 140 GOES-12_IMG 0.65um 4km 20040105 0015
pnga2area Q1 UI 150 GOES-12_IMG 10.7um 4km 20040105 0015
pnga2area Q1 UA 160 GOES-12_IMG 10.7um 4km 20040105 0015
pnga2area Q1 UC 60 GOES-12_IMG 10.7um 4km 20040105 0015
pnga2area Q0 CB 1110 GOES-10_SND UNKBAND 14km 20040104 2300
pnga2area Q0 CD 1130 GOES-10_SND UNKBAND 14km 20040104 2300
pnga2area Q0 CE 1140 GOES-10_SND UNKBAND 14km 20040104 2300
pnga2area Q1 UX 100 GOES-8_IMG 10.7um 1km 20040105 0000
pnga2area Q1 UY 110 GOES-8_IMG 6.8um 1km 20040105 0000

Images in the datastream before the changes all have hour portion indicators of Q0 or Q1. When the image broadcast frequency increases to twice-per-hour, the new, half hourly images will all have an hour portion indicator of Q3. This coupled with the routing code values in the image headers is enough for to completely tailor what you ingest.

The following are four possible ldmd.conf request lines. In all cases, upstream.host.name is the name of the IDD host you receive UNIWISC images from.

NOTE: anytime you modify your ldmd.conf file, you must stop and restart your LDM for the changes to become active:

<login as ldm>
ldmadmin stop
ldmadmin start

Modify the decoding of Unidata-Wisconsin imagery

If you decide that you want to continue to ingest just the hourly imagery that is in the datastream before the changes (either of the first two options above), and if your are currently decoding that imagery with no problems, you need do nothing.

If you decide that you want the new wavelength band images, then you will most likely need to adjust the pnga2area decode entries in your LDM's pqact.conf file. There are two possibilities:

  1. you use the McIDAS routing table approach to decoding images.

    In this case, you will need to add new entries to your McIDAS routing table. This is done by the user mcidas:

    <login as mcidas>
    cd workdata
    ftp ftp.unidata.ucar.edu
      <user> anonymous
      <pass> your_full_email_address
      binary
      cd pub/mcidas/data
      get UPRTE.BAT
      quit
    batch.k UPRTE.BAT
    

    NOTE: This assumes that you have a McIDAS REDIRECTion for the copy of ROUTE.SYS that is located in the directory in which you decode LDM-MCIDAS imagery.

  2. you use pnga2area entries in your LDM's pqact.conf file that write files into a directory hierarchy that is required by GEMPAK or used by McIDAS.

    In this case, you need to:

    • incorporate new pnga2area entries from ldm-mcidas-pqact.conf.all into your pqact.conf file
    • check your modified pqact.conf file for integrity by running ldmadmin pqactcheck correcting any errors that may have been found
    • tell the LDM pqact application to reread pqact.conf using ldmadmin pqactHUP.

    NOTE: An FTPable copy of ldm-mcidas-pqact.conf.all can be found in the pub/ldm-mcidas/pqact directory of anonymous FTP on the Unidata FTP server, ftp.unidata.ucar.edu:

    <login as ldm>
    cd etc
    ftp ftp.unidata.ucar.edu
      <user> anonymous
      <pass> your_full_email_address
      binary
      cd pub/ldm-mcidas/pqact
      get ldm-mcidas-pqact.conf.all
      quit
    

Modify your procedures for scouring data

If you are only using the McIDAS routing table approach for decoding UNIWISC imagery, you should not need alter your image scouring procedures.

If you have been using the directory hierarchy decoding method offered by the pnga2area, you should not need to alter your image scouring procedures (assuming that your data scouring is currently working correctly).

If you switch from the McIDAS routing table to the directory hierarchy approach for decoding UNIWISC imagery (and this is what we recommend), you will need to establish new procedures for scouring imagery from your system. We have made an FTPable C-shell script that you can use to scour UNIWISC images ingested into a directory hierarchy. FTP this file to a directory in the PATH defined for your ldm user:

<login as ldm>
cd util             <- for example
ftp ftp.unidata.ucar.edu
  <user> anonymous
  <pass> your_full_email_address
  cd pub/ldm/scour
  binary
  get prune_images.csh
  quit
chmod +x prune_images.csh

You will need to edit prune_images.csh to set several items to match your LDM setup:

PATH - insure that the directory containing prune_images.csh is in PATH
KEEP - set the number of images you want to keep (same for all bands)
areadir - set to the directory under which all images/image subdirectories are found

After prune_images.csh is properly configured AND set to be executable, you will need to add a crontab entry to run the script at least once-per-hour. Here is an example entry that is used here at the Unidata Program Center:

15 * * * * util/prune_images.csh > /dev/null 2>&1

NOTE: this entry assumes that prune_images.csh was copied to the ~ldm/util directory.

Modify McIDAS ADDE serving of the data you ingest (McIDAS users only)

If you run McIDAS, you will want to update your ADDE setup to be able to serve the new images you ingest. How you do this depends on how you are decoding the UNIWISC images. In both cases, we have provided new ADDE data serving template BATCH files that you should FTP and use to update your ADDE setup.
  1. you use the McIDAS routing table approach to decoding images. In this case, download the file DSSERVEOLD.BAT to the ~mcidas/data directory:

    <login as mcidas>
    cd ~mcidas/data
    ftp ftp.unidata.ucar.edu
      <user> anonymous
      <pass> your_full_email_address
      cd pub/mcidas/data
      binary
      get DSSERVEOLD.BAT
      quit
    
  2. you use pnga2area entries in your LDM's pqact.conf file that write files in to a directory hierarchy that is needed by GEMPAK or used by McIDAS.

    In this case, download the file DSSERVE.BAT to the ~mcidas/data directory:

    <login as mcidas>
    cd ~mcidas/data
    ftp ftp.unidata.ucar.edu
      <user> anonymous
      <pass> your_full_email_address
      cd pub/mcidas/data
      binary
      get DSSERVE.BAT
      quit
    
In both cases, use the file you just downloaded to create a new LSSERVE.BAT adjusting entries to match how you are decoding the UNIWISC imagery:
cp LSSERVE.BAT LSSERVE.BAT.bak

cp DSSERVEOLD.BAT LSSERVE.BAT
  - or -
cp DSSERVE.BAT LSSERVE.BAT

<edit LSSERVE.BAT and set entries to match your decoding setup>

IMPORTANT: make sure that you include any changes you made in your previous version of LSSERVE.BAT into the new copy!

After making the changes to LSSERVE.BAT, make the changes active for McIDAS ADDE serving:

cd ~mcidas/workdata
batch.k LSSERVE.BAT

After running batch.k LSSERVE.BAT, make sure that you can access the new imagery:

dsinfo.k IMAGE CIMSS
dsinfo.k IMAGE RTIMAGES

imglist.k RTIMAGES/GE-39
imglist.k RTIMAGES/GE-CO2
imglist.k RTIMAGES/GW-39
imglist.k RTIMAGES/GW-12


Send questions/comments to <support@unidata.ucar.edu>.