[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20040719: CRAFT (GPNEXR2) and bzipped data (cont.)

>From: Gerry Creager N5JXS <address@hidden>
>Organization: Texas A&M University -- AATLT
>Keywords: 200407190307.i6J372aW022195 LDM CRAFT

Gerry,

I logged onto gemdata2 and believe that the problems you were seeing
were all caused from you copying the GEMPAK executables into ~ldm/bin.
This is a no-no -- the ~ldm/bin directory should _only_ have LDM
routines.  One likely result from copying the GEMPAK executables
to ~/bin was in use of 'ps'.  GEMPAK unfortunately has an application
named 'ps'.  Your copying it to ~ldm/bin, and the ordering of directories
in the PATH of 'ldm' was causing the GEMPAK version of 'ps' to be
used instead of the Linux 'ps'.

My "solution" was to delete everything in ~ldm/bin and do a reinstall
of the LDM binaries.  After that, I ran 'ldmadmin clean' and then
'ldmadmin mkqueue -f' with no problems.

Next, I edited ~ldm/etc/ldmd.conf and changed all data feeds to be
off of bigbird.

Finally, I started the LDM and see data being ingested and filed
as expected:

cd /data/ldm/gempak/nexrad/craft_all
ls 20040721/*

ls 20040721
KABX  KBRO  KEPZ  KFDR  KHGX  KLBB  KMAF  KSJT  KTLX
KAMA  KCRP  KEWX  KFWS  KINX  KLCH  KSHV  KSRX  KVNX

It is important to leave the GEMPAK executables under the GEMPAK
directory -- with the exception of the GEMPAK data decoders which
you copy to ~ldm/decoders.

As I close this email, things look like they are running well
(but, then again, I didn't look too hard).

Cheers,

Tom

From: Gerry Creager N5JXS <address@hidden>
Date: Wed, 21 Jul 2004 22:20:33 -0500

I've not checked incoming mail yet but since it's up, it must, as you 
surmised, have been relatively simple.

I upgraded gempak to 5.7p2 and the scripts now work.  I *thought* 
everything worked for the bzipped files with 5.7p1... guess I was wrong.

I'm upgrading to the pqact.conf to run scripts tomorrow morning.  I'll 
keep you posted.

Thanks, again!
gerry
-- 
Gerry Creager -- address@hidden
Texas Mesonet -- AATLT, Texas A&M University    
Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.847.8578
Page: 979.228.0173
Office: 903A Eller Bldg, TAMU, College Station, TX 77843

>Attached is the pqact.conf_craft for reference.  I *think* it's a copy 
>of the pertinent data from bigbird.
>
>More inline...
>
>Unidata Support wrote:
>>>From: Gerry Creager N5JXS <address@hidden>
>>>Organization: Texas A&M University -- AATLT
>>>Keywords: 200407190307.i6J372aW022195 LDM CRAFT
>> 
>> 
>> Hi Gerry,
>> 
>> 
>>>I've been ingesting craft data to gemdata2.tamu.edu from bigbird.  I've 
>>>finally gotten all the tabs expanded properly so I'm seeing files that 
>>>appear to be appropriate in the directories.
>> 
>> 
>> It sounds like you should want the pqact.conf action(s) on gemdata2 to
>> be identical to the ones on bigbird: ~ldm/etc/pqact.gempak_craft.tamu.
>> I would simply copy pqact.gempak_craft.tamu from bigbird to gemdata2
>> and make sure that you run the appropriate set of pqact actions in
>> ~ldm/etc/ldmd.conf:
>> 
>> exec    "pqact -f CRAFT -p BZIP2/K[A-D] etc/pqact.gempak_craft.tamu"
>> exec    "pqact -f CRAFT -p BZIP2/K[E-K] etc/pqact.gempak_craft.tamu"
>> exec    "pqact -f CRAFT -p BZIP2/K[L-R] etc/pqact.gempak_craft.tamu"
>> exec    "pqact -f CRAFT -p BZIP2/K[S-Z] etc/pqact.gempak_craft.tamu"
>
>I *appear* to be gettin the appropriate CRAFT data in for the files, but 
>I'll try to invocation per bigbird.  I don't think that's the issue but 
>it might be.
>
>Per our discussion, I tried notifyme to see if the pattern works and it 
>does.
>
>>>I'm running your utility 
>>>to create the symlink from 'craft_all/[date]' to 'craft', which appears 
>>>to be working.
>> 
>> 
>> OK.
>> 
>> 
>>>I'm *NOT* running dcnexr2 to decompress the data; we decided that wasn't 
>>>necessary.
>> 
>> 
>> If you are copying what is on bigbird, then the actions needed are
>> a FILE that writes the CRAFT pieces for each station to a hidden file
>> for that station/time followed by an EXEC action that renames the
>> hidden file to make it visible.
>> 
>> 
>>>I'm getting bad file format errors both when I run the script,
>> 
>> 
>> Which script?
>
>KABX.sh is attached for review... It's representative of the others 
>which are for each invocation per a different site.  I might be able to 
>make one massive script with a switch for the siteID but this was what I 
>could manage on the airplane coming home.
>
>>>which makes pqact complain,
>> 
>> 
>> I don't understand.  Are you talking about pqact.conf actions?
>
>A 127 error, apparently whenever the script tries to execute?
>
>>>and when I run gpnexr2 interactively I see the 
>>>following:
>>>
>>> GEMPAK-GPNEXR2>l
>>> MAP      = 1
>>> GAREA    = dset
>>> PROJ     = rad
>>> RADFIL   = NEXRII|KHGX
>>> LATLON   =
>>> PANEL    = 0
>>> TITLE    = 1
>>> TEXT     = 1
>>> CLEAR    = yes
>>> DEVICE   = xw
>>> LUTFIL   = default
>>> IMCBAR   =
>>> TILT     = .5
>>> RADPARM  = dz
>>> RADTIM   = last
>>> GEMPAK-GPNEXR2>r
>>>Creating process: xw for queue 1238106115
>>> GPMAP PARAMETERS:
>>>
>>> Device:              xw 
>>> Projection:          rad 
>>> Graphics area name:  dset 
>>> Map:                 1 
>>> Title:               1 
>>> Panel:               0 
>>> Latlon: 
>>> Clear:               YES
>>> Image file:          $RAD/craft/KHGX/KHGX_20040719_0257 
>>>
>>>Enter <cr> to accept parameters or type EXIT:
>>>Can't read station ID from file, using KHGX
>>>oops, sweep 0.500000 +/- 20.000000 is null
>>> [IM -3]  Image file $RAD/craft/KHGX/KHGX_20040719_0257 not a supported 
>>>format
>>> [IM -8]  Could not open LUT file ...
>>> [GEMPLT -15]  NIPROJ - Invalid projection specified.
>>> [GG -7]  No map drawn.
>>> [GEMPLT -15]  NIPROJ - Invalid projection specified.
>>> [GG -13]  Error drawing lat/lon grid.
>>> Parameters requested: MAP,GAREA,PROJ,RADFIL,LATLON,PANEL,TITLE,TEXT,CLEAR,
>>> DEVICE,LUTFIL,IMCBAR,TILT,RADPARM,RADTIM.
>>> GEMPAK-GPNEXR2>
>>>
>>>Any idea where I'm screwing up?  This ought to be simple...
>> 
>> 
>> I am assuming that you are saying that your pqact.conf action(s) on
>> gemdata2 are screwing up, and that results in the gempak routine failure
>> when reading a file on disk.  True?
>
>Apparently so.
>
>> If the answer is yes, then I recommend simply using the
>> pqact.gempak_craft.tamu file from bigbird and the CRAFT pqact exects
>> from bigbird's ldmd.conf file.  Since things work on bigbird, they
>> should work on another Linux machine doing essentially the same
>> ingest/file/processing.
>
>I'll try that after I chauffer the 13 yr old to a drivng range for a 
>bucket of balls.
>
>Thanks, gerry
>-- 
>Gerry Creager -- address@hidden
>Texas Mesonet -- AATLT, Texas A&M University   
>Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.847.8578
>Page: 979.228.0173
>Office: 903A Eller Bldg, TAMU, College Station, TX 77843
>
>--------------080108020103090200040303
>Content-Type: text/plain;
> name="pqact.conf_craft"
>Content-Transfer-Encoding: 7bit
>Content-Disposition: inline;
> filename="pqact.conf_craft"
>
># CRAFT stored as raw bz2 for GEMPAK
>#
># file the raw data to a temporary file beginning with "." so that autoupdate 
> GUIs don't
># get ugly partial volume plots
>CRAFT  ^L2-BZIP2/(KABX|KAMA|KBRO|KCRP|KDFX|KDYX|KEPZ|KEWX|KFDR|KFDX|KFWS|KGRK|
> KHDX|KHGX|KINX|KLBB|KLCH|KLKZ|KMAF|KPOE|KSHV|KSJT|KSRX|KTLX|KVNX)/([0-9][0-9]
> [0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9])([0-9][0-9])
>       FILE    data/gempak/nexrad/craft_all/\2/\1/.\1_\2_\3
>#
># Done to move file after last record is received "/E" to prevent
># autoupdate from seeing partially received files (dccraft_move is a shell scr
> ipt copied from $NAWIPS/bin/scripts)
>CRAFT  ^L2-BZIP2/(KABX|KAMA|KBRO|KCRP|KDFX|KDYX|KEPZ|KEWX|KFDR|KFDX|KFWS|KGRK|
> KHDX|KHGX|KINX|KLBB|KLCH|KLKZ|KMAF|KPOE|KSHV|KSJT|KSRX|KTLX|KVNX)/([0-9][0-9]
> [0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9])([0-9][0-9]).*/E$
>       EXEC    util/dccraft_move data/gempak/nexrad/craft_all/\2/\1/.\1_\2_\3 
> data/gempak/nexrad/craft_all/\2/\1/\1_\2_\3
>#
>#
># CRAFT stored uncompressed (not needed for GEMPAK 5.7.2p2 and later)
>#CRAFT ^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][
> 0-5][0-9])([0-9][0-9])
>#      PIPE    decoders/dcnexr2 -s \1 -d /dev/null data/gempak/nexrad/craft_un
> compressed/\1/\1_\2_\3
>
>--------------080108020103090200040303
>Content-Type: application/x-sh;
> name="KABX.sh"
>Content-Transfer-Encoding: 7bit
>Content-Disposition: inline;
> filename="KABX.sh"
>
>#!/bin/csh
>
>### Rich Otto
>###  stolen shamelessly by Gerry Creager, 20040404
>######################################################################
>### This script will create base reflectivity images for all the radar 
>### sites in the TX area
>###
>### Mod'd to use gepnexr2 instead of gpmap gc 20040404
>### Mod for single site for GEMPAK execution:  gc 20040714
>######################################################################
>       
>source /etc/Gemenviron
>cd /usr/local/ldm/images/craft
>
>#set AMAarea="33.16;-104.18;37.29;-99.10"
>#set HGXarea="27.40;-97.43;31.50;-92.67"
>#set FWSarea="30.50;-99.71;34.63;-94.77"
>#set EWXarea="27.64;-100.36;31.76;-95.57"
>#set LCHarea="28.06;-95.56;32.18;-90.74"
>set ABXarea="33.08;-109.29;37.21;-104.21"
>#set LBBarea="31.59;-104.26;35.71;-99.25"
>#set MAFarea="29.87;-104.58;34.00;-99.69"
>#set BROarea="23.85;-99.70;27.98;-95.07"
>#set SHVarea="30.38;-96.25;34.51;-91.31"
>#set EPZarea="29.80;-109.09;33.93;-104.20"
>#set SJTarea="29.31;-102.87;33.42;-98.00"
>#set CRParea="25.72;-99.82;29.84;-95.13"
>#set SRXarea="33.22;-96.84;37.35;-91.75"
>#set GRKarea="28.65;-99.74;32.78;-94.93"
>#set VNXarea="34.67;-100.66;38.80;-95.50"
>#set DYXarea="30.48;-101.67;34.60;-96.75"
>#set FDXarea="32.57;-106.11;36.69;-101.07"
>#set HDXarea="31.01;-108.53;35.14;-103.58"
>#set DFXarea="27.21;-102.61;31.33;-97.86"
>#set POEarea="29.10;-95.35;33.21;-90.47"
>#set FDRarea="32.29;-101.41;36.42;-96.41"
>#set TLXarea="33.26;-99.73;37.39;-94.69"
>
>### remove oldest file, move back remaining files (1 hr old becomes 2 hrs old)
>rm -f KABX/dz/24-05-KABX.gif
>mv KABX/dz/23-05-KABX.gif KABX/dz/24-05-KABX.gif
>mv KABX/dz/22-05-KABX.gif KABX/dz/23-05-KABX.gif
>mv KABX/dz/21-05-KABX.gif KABX/dz/22-05-KABX.gif
>mv KABX/dz/20-05-KABX.gif KABX/dz/21-05-KABX.gif
>mv KABX/dz/19-05-KABX.gif KABX/dz/20-05-KABX.gif
>mv KABX/dz/18-05-KABX.gif KABX/dz/19-05-KABX.gif
>mv KABX/dz/17-05-KABX.gif KABX/dz/18-05-KABX.gif
>mv KABX/dz/16-05-KABX.gif KABX/dz/17-05-KABX.gif
>mv KABX/dz/15-05-KABX.gif KABX/dz/16-05-KABX.gif
>mv KABX/dz/14-05-KABX.gif KABX/dz/15-05-KABX.gif
>mv KABX/dz/13-05-KABX.gif KABX/dz/14-05-KABX.gif
>mv KABX/dz/12-05-KABX.gif KABX/dz/13-05-KABX.gif
>mv KABX/dz/11-05-KABX.gif KABX/dz/12-05-KABX.gif
>mv KABX/dz/10-05-KABX.gif KABX/dz/11-05-KABX.gif
>mv KABX/dz/09-05-KABX.gif KABX/dz/10-05-KABX.gif
>mv KABX/dz/08-05-KABX.gif KABX/dz/09-05-KABX.gif
>mv KABX/dz/07-05-KABX.gif KABX/dz/08-05-KABX.gif
>mv KABX/dz/06-05-KABX.gif KABX/dz/07-05-KABX.gif
>mv KABX/dz/05-05-KABX.gif KABX/dz/06-05-KABX.gif
>mv KABX/dz/04-05-KABX.gif KABX/dz/05-05-KABX.gif
>mv KABX/dz/03-05-KABX.gif KABX/dz/04-05-KABX.gif
>mv KABX/dz/02-05-KABX.gif KABX/dz/03-05-KABX.gif
>cp KABX/dz/01-05-KABX.gif KABX/dz/02-05-KABX.gif
>
>set RADFIL = "NEXRII|KABX"
>set DEV = "gif|KABX/dz/01-05-KABX.gif|600;600"
>
>
>echo RADFIL = $RADFIL
>echo DEV = $DEV
>
>set AREA = $ABXarea
>
>       ### run gpnexr2 for each site
>
>gpnexr2 << endgpnexr2
>
> MAP      = 1
> GAREA    = $AREA
> PROJ     = RAD
> RADFIL   = $RADFIL
> LATLON   = 0
> PANEL    = 0
> TITLE    = 1
> TEXT     = 1/3/HW
> CLEAR    = YES
> DEVICE   = $DEV
> LUTFIL   = DEFAULT
> IMCBAR   = 31/V/LL/.005;.05/.85;.01|.7/1/1/hw
> TILT     = 0.5
> RADPARM  = dz
> RADTIM   = current
>
>
>run
>
>
>
>exit
>
>
>endgpnexr2
>
>
>
>#end   ### end of foreach loop
>
>
>gpend
>
>
>
>######################################################################
>### We're reusing the script for Radial Velocity
>### This script will create vr  images for all the radar 
>### sites in the TX area
>###
>### Mod'd to use gepnexr2 instead of gpmap gc 20040404
>### Mod'd to generate 'vr' gc 20040618
>######################################################################
>
>### remove oldest file, move back remaining files (1 hr old becomes 2 hrs old)
>rm -f KABX/vr/24-05-KABX.gif
>mv KABX/vr/23-05-KABX.gif KABX/vr/24-05-KABX.gif
>mv KABX/vr/22-05-KABX.gif KABX/vr/23-05-KABX.gif
>mv KABX/vr/21-05-KABX.gif KABX/vr/22-05-KABX.gif
>mv KABX/vr/20-05-KABX.gif KABX/vr/21-05-KABX.gif
>mv KABX/vr/19-05-KABX.gif KABX/vr/20-05-KABX.gif
>mv KABX/vr/18-05-KABX.gif KABX/vr/19-05-KABX.gif
>mv KABX/vr/17-05-KABX.gif KABX/vr/18-05-KABX.gif
>mv KABX/vr/16-05-KABX.gif KABX/vr/17-05-KABX.gif
>mv KABX/vr/15-05-KABX.gif KABX/vr/16-05-KABX.gif
>mv KABX/vr/14-05-KABX.gif KABX/vr/15-05-KABX.gif
>mv KABX/vr/13-05-KABX.gif KABX/vr/14-05-KABX.gif
>mv KABX/vr/12-05-KABX.gif KABX/vr/13-05-KABX.gif
>mv KABX/vr/11-05-KABX.gif KABX/vr/12-05-KABX.gif
>mv KABX/vr/10-05-KABX.gif KABX/vr/11-05-KABX.gif
>mv KABX/vr/09-05-KABX.gif KABX/vr/10-05-KABX.gif
>mv KABX/vr/08-05-KABX.gif KABX/vr/09-05-KABX.gif
>mv KABX/vr/07-05-KABX.gif KABX/vr/08-05-KABX.gif
>mv KABX/vr/06-05-KABX.gif KABX/vr/07-05-KABX.gif
>mv KABX/vr/05-05-KABX.gif KABX/vr/06-05-KABX.gif
>mv KABX/vr/04-05-KABX.gif KABX/vr/05-05-KABX.gif
>mv KABX/vr/03-05-KABX.gif KABX/vr/04-05-KABX.gif
>mv KABX/vr/02-05-KABX.gif KABX/vr/03-05-KABX.gif
>cp KABX/vr/01-05-KABX.gif KABX/vr/02-05-KABX.gif
>
>set RADFIL = "NEXRII|KABX"
>set DEV = "gif|KABX/vr/01-05-KABX.gif|600;600"
>
>
>echo RADFIL = $RADFIL
>echo DEV = $DEV
>
>
>       ### run gpnexr2 for each site
>
>gpnexr2 << end2gpnexr2
>
> MAP      = 1
> GAREA    = $AREA
> PROJ     = RAD
> RADFIL   = $RADFIL
> LATLON   = 0
> PANEL    = 0
> TITLE    = 1
> TEXT     = 1/3/HW
> CLEAR    = YES
> DEVICE   = $DEV
> LUTFIL   = DEFAULT
> IMCBAR   = 31/V/LL/.005;.05/.85;.01|.7/1/1/hw
> TILT     = 0.5
> RADPARM  = vr
> RADTIM   = current
>
>
>run
>
>
>
>exit
>
>
>
>end2gpnexr2
>
>
>
>gpend
>
>
>
>######################################################################
>### We're reusing the script for Spectrum Width
>### This script will create vr  images for all the radar 
>### sites in the TX area
>###
>### Mod'd to use gepnexr2 instead of gpmap gc 20040404
>### Mod'd to generate 'sw' gc 20040618
>######################################################################
>
>### remove oldest file, move back remaining files (1 hr old becomes 2 hrs old)
>rm -f KABX/sw/24-05-KABX.gif
>mv KABX/sw/23-05-KABX.gif KABX/sw/24-05-KABX.gif
>mv KABX/sw/22-05-KABX.gif KABX/sw/23-05-KABX.gif
>mv KABX/sw/21-05-KABX.gif KABX/sw/22-05-KABX.gif
>mv KABX/sw/20-05-KABX.gif KABX/sw/21-05-KABX.gif
>mv KABX/sw/19-05-KABX.gif KABX/sw/20-05-KABX.gif
>mv KABX/sw/18-05-KABX.gif KABX/sw/19-05-KABX.gif
>mv KABX/sw/17-05-KABX.gif KABX/sw/18-05-KABX.gif
>mv KABX/sw/16-05-KABX.gif KABX/sw/17-05-KABX.gif
>mv KABX/sw/15-05-KABX.gif KABX/sw/16-05-KABX.gif
>mv KABX/sw/14-05-KABX.gif KABX/sw/15-05-KABX.gif
>mv KABX/sw/13-05-KABX.gif KABX/sw/14-05-KABX.gif
>mv KABX/sw/12-05-KABX.gif KABX/sw/13-05-KABX.gif
>mv KABX/sw/11-05-KABX.gif KABX/sw/12-05-KABX.gif
>mv KABX/sw/10-05-KABX.gif KABX/sw/11-05-KABX.gif
>mv KABX/sw/09-05-KABX.gif KABX/sw/10-05-KABX.gif
>mv KABX/sw/08-05-KABX.gif KABX/sw/09-05-KABX.gif
>mv KABX/sw/07-05-KABX.gif KABX/sw/08-05-KABX.gif
>mv KABX/sw/06-05-KABX.gif KABX/sw/07-05-KABX.gif
>mv KABX/sw/05-05-KABX.gif KABX/sw/06-05-KABX.gif
>mv KABX/sw/04-05-KABX.gif KABX/sw/05-05-KABX.gif
>mv KABX/sw/03-05-KABX.gif KABX/sw/04-05-KABX.gif
>mv KABX/sw/02-05-KABX.gif KABX/sw/03-05-KABX.gif
>cp KABX/sw/01-05-KABX.gif KABX/sw/02-05-KABX.gif
>
>set RADFIL = "NEXRII|KABX"
>set DEV = "gif|KABX/sw/01-05-KABX.gif|600;600"
>
>
>echo RADFIL = $RADFIL
>echo DEV = $DEV
>
>
>###    run gpnexr2 for each site
>
>gpnexr2 << end2gpnexr2
>
> MAP      = 1
> GAREA    = $AREA
> PROJ     = RAD
> RADFIL   = $RADFIL
> LATLON   = 0
> PANEL    = 0
> TITLE    = 1
> TEXT     = 1/3/HW
> CLEAR    = YES
> DEVICE   = $DEV
> LUTFIL   = DEFAULT
> IMCBAR   = 31/V/LL/.005;.05/.85;.01|.7/1/1/hw
> TILT     = 0.5
> RADPARM  = sw
> RADTIM   = current
>
>
>run
>
>
>
>exit
>
>
>end2gpnexr2
>
>
>gpend
>
>
>
>
>
>
>--------------080108020103090200040303--
>
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publically available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.