Re: 20051117: NetCDF decoder pqact entries (fwd)

NOTE: The decoders mailing list is no longer active. The list archives are made available for historical reasons.

To: bit-bucket <bit-bucket@xxxxxxxxxxxxxxxx>
Subject: Re: 20051117: NetCDF decoder pqact entries (fwd)
From: Robb Kambic <rkambic@xxxxxxxxxxxxxxxx>
Date: Thu, 1 Dec 2005 13:14:54 -0700 (MST)



==============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
rkambic@xxxxxxxxxxxxxxxx                   WWW: http://www.unidata.ucar.edu/
==============================================================================

---------- Forwarded message ----------
Date: Wed, 23 Nov 2005 13:21:01 -0700
From: Mark Seefeldt <mark.seefeldt@xxxxxxxxxxxx>
To: Robb Kambic <rkambic@xxxxxxxxxxxxxxxx>
Subject: Re: 20051117: NetCDF decoder pqact entries

Robb,

The tip on searching for the process ID in the log file was excellent.
It does not show up on any recent log files, but it does show up on the
file from when the ldm was restarted:
ldmd.log.5:Nov 18 20:20:46 foehn pqact[7584] NOTE: Starting Up
ldmd.log.5:Nov 18 20:20:46 foehn pqact[7584] NOTE: Configuration-file
"etc/pqact.netcdf" has no entries. You should probably not start this
program instead.

I have attached the ldmd.conf and the pqact.netcdf files to provide some
help.  It appears to me that there are entries in pqact.netcdf, and that
it is properly called from ldmd.conf.  Obviously something is not write.

To verify that the spelling jives, I did a listing of the etc directory:
[ldm@foehn etc]$ ls -l
total 344
drwxr-xr-x  2 ldm met-apps  4096 Apr 25  2005 6.1.0
-rw-r--r--  1 ldm met-apps  3116 Apr 22  2005 ldmadmin-pl.conf
-rw-r--r--  1 ldm met-apps  3118 Apr 22  2005 ldmadmin-pl.conf.orig
-rw-r--r--  1 ldm met-apps  4570 Nov 17 17:47 ldmd.conf
-rw-r--r--  1 ldm met-apps  4361 May  9  2005 ldmd.conf.20050509
drwxr-xr-x  2 ldm met-apps  4096 Nov 17 17:55 netcdf
-rw-r--r--  1 ldm met-apps  1622 Nov 10 14:19 netcheck.conf
-rw-r--r--  1 ldm met-apps  1622 Nov 10 14:19 netcheck.conf.dist
-rw-r--r--  1 ldm met-apps  3970 Nov 10 15:50 pqact.conf
-rw-r--r--  1 ldm met-apps  3966 May  2  2005 pqact.conf1
-rw-r--r--  1 ldm met-apps  3965 May  2  2005 pqact.conf2
-rw-r--r--  1 ldm met-apps 86663 Nov 10 12:20 pqact.gempak
-rw-r--r--  1 ldm met-apps 82423 Apr 25  2005 pqact.gempak.5.8.1
-rw-r--r--  1 ldm met-apps 86663 Nov 10 12:20 pqact.gempak.5.8.3a
-rw-r--r--  1 ldm met-apps  4216 Nov 17 17:55 pqact.netcdf
-rw-r--r--  1 ldm met-apps  3771 Nov 17 16:33 pqact.netcdf.unidata
-rw-r--r--  1 ldm met-apps   668 Nov 10 16:11 scour.conf
-rw-r--r--  1 ldm met-apps   623 Nov 10 14:19 scour.conf.dist

Please let me know as to the other suggestions as to where I should look
for problems.

Thanks

Mark


Robb Kambic wrote:

On Mon, 21 Nov 2005, Mark Seefeldt wrote:

Robb,

Thank you for the NetCDF decoder pqact entries.  I am currently not
producing any of the output files.

Information:
-LDM 6.4.3 on a Red Hat Enterprise Linux v.4
-installed:
   netcdf-3.6.0-p1
   netcdf-perl-1.2.3
   udunits-1.12.4
   decoders-4.0.0
-I placed the pqact entries into a separate pqact file named
pqact.netcdf.  This file is attached.
-The pqact.netcdf file is then called from ldmd.conf with the following:
   exec    "pqact etc/pqact.netcdf"
-When I do a process listing I see:
   ldm       7584  7580  0 Nov18 ?        00:12:38 pqact etc/pqact.netcdf

Are there any log files or other places where I can start looking to
find out why none of the netcdf files are being produced?


Mark,

the first place to look is the ~ldm/logs/ldmd.log file, the pqact will
send it errors to it. one can track the error by looking at the PID of the
pqact etc/pqact.netcdf process.

also check the pqact.netcdf that there are tabs not spaces between entry
parts because spaces could be part of pattern action. ie.

DDS|IDS<tab here>^S(A....|P....|XUS8.) .... ([0-3][0-9])
<tab here>PIPE<tab here>decoders/metar2nc.new1
<tab here>etc/metar.cdl.new1
<tab here>data/pub/decoded/netcdf/surface/metar

make sure all the decoder table files and cdl files are in the ~ldm/etc
directory.  they are located in the ~/etc directory in the build location.

from the ldm home directory, run the decoder from command line, ie

% decoders/metar2nc.new1 etc/metar.cdl.new1
data/pub/decoded/netcdf/surface/metar < <raw data file>

above is on one line


robb...

Thanks

Mark Seefeldt

Robb Kambic wrote:

On Thu, 17 Nov 2005, Unidata Support wrote:

Robb,

See second paragraph for request of pqact entries for NC decoders 4.0.0.

------- Forwarded Message

To: Unidata Support <support@xxxxxxxxxxxxxxxx>
From: Mark Seefeldt <mark.seefeldt@xxxxxxxxxxxx>
Subject: NetCDF decoder pqact entries
Organization: UCAR/Unidata
Keywords: 200511172016.jAHKGG7s010277

I am currently running LDM 6.4.3.  I have also intalled GEMPAK 5.8.3a.
I used the gen_pqact.csh script to create a pqact file to create the
GEMPAK files.  All appears to be working very well with that.

I have recently installed the NetCDF Decoders 4.0.0 (along with the
necessary netCDF, UDUNITS, and Perl-netCDF packages.  I would now like
to create a pqact file/entries for the netCDF files.  I have discovered
the file gribtonc.entries which appears to have the necessary entries
for the GRIB files.  Is this correct?  Other than making changes for the
local filesystem, is there anything else that I need to do to implement
these pqact entries?  Are there any similar sample entries for the
non-GRIB products?


Mark,

the documentation was sent out with the release, maybe i should include it
some where on the web page.  i'll attach the list of pqact actions for
both the text bulletions and the model files.

robb...

If there is a document already present to answer these questions, feel
free to point me that direction.  I was not able to find any in my search.

Thanks

Mark

--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.

------- End of Forwarded Message

==============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
rkambic@xxxxxxxxxxxxxxxx                   WWW: http://www.unidata.ucar.edu/
==============================================================================


------------------------------------------------------------------------

#
# surface data in netCDF format
#
# metar decoder, creates NetCDF files formated Surface_METAR_yyyymmdd_hh00.nc
# All aviation reports including metar tests. old netcdf file format
DDS|IDS ^S(A....|P....|XUS8.) .... ([0-3][0-9])
        PIPE    decoders/metar2nc
        etc/metar.cdl
        data/pub/decoded/netcdf/surface/metarold
#
# metar decoder, creates NetCDF files with new CDL format on a daily basis
# formated data/decoded/yyyymmdd_metar.nc
DDS|IDS ^S(A....|P....|XUS8.) .... ([0-3][0-9])
        PIPE    decoders/metar2nc.new1
        etc/metar.cdl.new1
        data/pub/decoded/netcdf/surface/metar
#
# upper air perl decoder
DDS|IDS ^U[ABDEFGHIJKLMPQRSTXZ].... .... ([0-3][0-9])
        PIPE    decoders/ua2nc
        etc/raob.cdl
        data/pub/decoded/netcdf/upperair
#
# buoy reports in yymmddhh_buoy.nc decoded into NetCDF format
DDS|IDS ^S[NS].... .... ([0-3][0-9])
        PIPE    decoders/buoy2nc
        etc/buoy.cdl
        data/pub/decoded/netcdf/surface/buoy
#
# Synoptic reports in yymmddhh_syn.nc decoded into NetCDF format
DDS|IDS ^SM.... .... ([0-3][0-9])
        PIPE    decoders/syn2nc
        etc/syn.cdl
        data/pub/decoded/netcdf/surface/synoptic
#
# grid data in netCDF format
#
# GFS grid on Lambert conformal CONUS grid replacing avn-q
HRS     ^Y.Q... KWB. (..)(..).*/m(AVN|SSIAVN|GFS|SSIGFS)
        PIPE    decoders/gribtonc
        etc/gfs_211.cdl
        
data/pub/decoded/netcdf/grid/NCEP/GFS/CONUS_80km/GFS_CONUS_80km_(\1:yyyy)(\1:mm)\1_\200.nc
#
# GFS grid on thinned grids, interpolated to global 5.0 x 2.5 regular grid
HRS     ^H.[I-P]... KWB. (..)(..).*/m(AVN|SSIAVN|GFS|SSIGFS)
        PIPE    decoders/gribtonc
        -q "lin,dlat=2.5,dlon=5.0" etc/gfs_37-44.cdl
        
data/pub/decoded/netcdf/grid/NCEP/GFS/Global_5x2p5deg/GFS_Global_5x2p5deg_(\1:yyyy)(\1:mm)\1_\200.nc
#
# ECMWF grid on global 2.5 x 2.5 degree grid
HRS     ^H.[A-L]... ECMF (..)(..)
        PIPE    decoders/gribtonc
        etc/ecmwf_1-12.cdl
        
data/pub/decoded/netcdf/grid/ECMWF/Global_2p5/ECMWF_Global_2p5_(\1:yyyy)(\1:mm)\1_\200.nc
#
# ETA grid on Lambert conformal CONUS grid
HRS     ^Y.Q... KWB. (..)(..).*/m(ETA|NAM)
        PIPE    decoders/gribtonc
        etc/eta_211.cdl
        
data/pub/decoded/netcdf/grid/NCEP/NAM/CONUS_80km/NAM_CONUS_80km_(\1:yyyy)(\1:mm)\1_\200.nc
#
# GFS grid on global 5.0 x 5.0 degree grid
HRS     ^H.[EF]... KWB. (..)(..)
        PIPE    decoders/gribtonc
        etc/gfs_25-26.cdl
        
data/pub/decoded/netcdf/grid/NCEP/GFS/Extended_Global_5p0deg/GFS_Extended_Global_5p0deg_(\1:yyyy)(\1:mm)\1_\200.nc
#
# NGM grid on Lambert conformal CONUS grid
#HRS    ^Y.Q... KWB. (..)(..).*/mNGM
#       PIPE    decoders/gribtonc
#       etc/ngm_211.cdl
#       
data/pub/decoded/netcdf/grid/NCEP/NGM/CONUS_80km/NGM_CONUS_80km_(\1:yyyy)(\1:mm)\1_\200.nc
#
# NOW and PWAV grids on global 5.0 x 2.5 degree grids
HRS     ^H.[A-D]... .... (..)(..).*(/mNOW|/mPWAV)
        PIPE    decoders/gribtonc
        etc/ocean_21-24.cdl
        
data/pub/decoded/netcdf/grid/NCEP/OCEAN/Global_5x2p5deg/OCEAN_Global_5x2p5deg_(\1:yyyy)(\1:mm)\1_\200.nc
#
# RUC grid on Lambert conformal CONUS grid
HRS     ^[YZ].Q... KWB. (..)(..).*/mRUC
        PIPE    decoders/gribtonc
        etc/ruc_211.cdl
        
data/pub/decoded/netcdf/grid/NCEP/RUC/CONUS_80km/RUC_CONUS_80km_(\1:yyyy)(\1:mm)\1_\200.nc
#
# RUC2 grid on Lambert conformal CONUS grid
HRS     ^Y.W... KWB. (..)(..).*/mRUC2
        PIPE    decoders/gribtonc
        etc/ruc2_236.cdl
        
data/pub/decoded/netcdf/grid/NCEP/RUC2/CONUS_40km/RUC2_CONUS_40km_(\1:yyyy)(\1:mm)\1_\200.nc
#
# SST grid on global 5.0 x 2.5 degree grids
HRS     ^H.[A-D]... .... (..)(..).*/mSST
        PIPE    decoders/gribtonc
        etc/sst_21-24.cdl
        
data/pub/decoded/netcdf/grid/NCEP/SST/Global_5x2p5deg/SST_Global_5x2p5deg_(\1:yyyy)(\1:mm)\1_\200.nc
#
# SST grid on global 2.0 x 2.0 degree grid
HRS     ^H.[T-W]... .... (..)(..).*/mSST
        PIPE    decoders/gribtonc
        etc/sst_61-64.cdl
        
data/pub/decoded/netcdf/grid/NCEP/SST/Global_2x2deg/SST_Global_2x2deg_(\1:yyyy)(\1:mm)\1_\200.nc
#


==============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
rkambic@xxxxxxxxxxxxxxxx                   WWW: http://www.unidata.ucar.edu/
==============================================================================

#####
# $Id: ldmd.conf,v 1.18 2003/02/27 17:12:57 steve Exp $
# Sample ldmd.conf for ldm5
####
#
# This is the main configuration file for the LDM server. All lines that start
# with a "#" sign are comments.
#
# To debug an LDM that hangs on start up, run the following from LDM home:
# % bin/rpc.ldmd -vl - -q data/ldm.pq etc/ldmd.conf
#
# If the LDM still hangs, comment out all lines in this file except noted
# allow line entry below, try again.
#
###############################################################################
# WARNING: Pathological Product-ID Patterns
###############################################################################
#
# The REQUEST and ACCEPT entries both have product-ID patterns that are regular
# expressions (RE-s).  It is EXTREMELY IMPORTANT that these patterns not have a
# ".*" prefix because: 1) such a prefix accomplishes nothing (i.e., the set of
# possible matches is unaltered by the prefix); and 2) such a prefix causes the 
# RE to take MUCH, MUCH LONGER TO EXECUTE (by 3 orders of magnitude in many
# cases).  Therefore, DO NOT USE RE-s like
#
#     ".*UFPBA"
#
# Instead, simply use
#
#     "UFPBA"
#
# The regex(1) utility of this package can be used to examine the efficiency
# of an RE.
#
###############################################################################
# Exec Entries
###############################################################################
#
# Programs that can share a queue with rpc.ldmd server, these programs
# are started by the "exec" command and are in the same process group.
#
# Regarding pqbinstats, while pqbinstats will generate .stats files in
# your logs directory, it is necessary to also run 'ldmadmin dostats'
# once per hour via cron in order to email the stats to the UPC and
# remove the .stats files that are no longer needed.  See
# http://www.unidata.ucar.edu/packages/ldm/ldmConfigInstallList.html#s7
# for more information.
#exec   "pqbinstats"
exec    "pqact"
exec    "pqact etc/pqact.gempak"
exec    "pqact etc/pqact.netcdf"
#
# rtstats uses the LDM to send product latency statistics to the UPC.
# We are currently evolving from pqbinstats to rtstats, but in the
# mean time we hope sites will run both.  For more info on rtstats
# please see the man page.
exec    "rtstats -h rtstats.unidata.ucar.edu"
#exec   "pqsurf"
#
###############################################################################
# Request Entries
###############################################################################
#
# LDM5 servers request data from Data Sources
#
#       request <feedset> <pattern> <hostname pattern>
#
#request WMO ".*" uni0.unidata.ucar.edu
request WMO ".*" rainbow.al.noaa.gov
# observational data from the NOAAPORT streams
request    IDS|DDPLUS    ".*"    rainbow.al.noaa.gov
# ETA/NAM model
request    HDS    "/mNAM_84"    rainbow.al.noaa.gov
# RUC model
request    HDS    "/mRUC"    rainbow.al.noaa.gov
# NEXRAD - Level III data
request NEXRAD ".*" rainbow.al.noaa.gov
#
# IDD-Antarctic
# UWisconsin AWS and composites
request    EXP    ".*"      ice.ssec.wisc.edu PRIMARY
#
###############################################################################
# Allow Entries
###############################################################################
#
# Giving permission for a Data Sink to perform a request to your LDM
#
# allow <feedset> <hostname pattern>
#
# Giving permission to your own machine and Unidata
#
# Under no circumstances comment out the next allow entry to localhost
# The LDM will NOT start if the lines are commented out.
allow   ANY
    ^((localhost|loopback)|(127\.0\.0\.1\.?$)|([a-z].*\.unidata\.ucar\.edu\.?$))
allow   ANY ice.ssec.wisc.edu
#
###############################################################################
# Accept Entries
###############################################################################
# ACCEPT: Who can feed us, currently this action is ONLY needed for WSI data
#
# accept <feedset> <pattern> <hostname pattern>
#
# accept anything from yourself
#
#accept ANY ".*" ^((localhost|loopback)|(127\.0\.0\.1\.?$))
#
# accept from your upstream site
#
# Antarctic-IDD entries"
ACCEPT EXP
       ".*"
       198\.253\.*\.*
ACCEPT EXP
       "USAP*"
       wheel\.mmm\.ucar\.edu
ACCEPT EXP
       ".*"
       157\.132\.119\.111
#
# WSI is using ldm4 protocol so the accept is still required
#accept WSI
#    .*
#    ^[a-z].*\.uni\.wsicorp\.com$
#
###############################################################################
# End 
###############################################################################

#############################################################################
# netCDF specific actions
#
# Mark Seefeldt
# University of Colorado - Boulder
#
# original file from:
# Rob Kambic
# Unidata Program Center
#
# last update 11/17/2005
#
#############################################################################

#
# begin netCDF decoder section
#

#--------------------------------------------------------------------
# surface data in netCDF format
#
# metar decoder, creates NetCDF files formated Surface_METAR_yyyymmdd_hh00.nc
# All aviation reports including metar tests. old netcdf file format
DDS|IDS ^S(A....|P....|XUS8.) .... ([0-3][0-9])
        PIPE    decoders/metar2nc
        etc/netcdf/metar.cdl
        data/netcdf/surface/metarold
#
# metar decoder, creates NetCDF files with new CDL format on a daily basis
# formated data/decoded/yyyymmdd_metar.nc
DDS|IDS ^S(A....|P....|XUS8.) .... ([0-3][0-9])
        PIPE    decoders/metar2nc.new1
        etc/netcdf/metar.cdl.new1
        data/netcdf/surface/metar
#
# upper air perl decoder
DDS|IDS ^U[ABDEFGHIJKLMPQRSTXZ].... .... ([0-3][0-9])
        PIPE    decoders/ua2nc
        etc/netcdf/raob.cdl
        data/netcdf/upperair
#
# buoy reports in yymmddhh_buoy.nc decoded into NetCDF format
DDS|IDS ^S[NS].... .... ([0-3][0-9])
        PIPE    decoders/buoy2nc
        etc/netcdf/buoy.cdl
        data/netcdf/surface/buoy
#
# Synoptic reports in yymmddhh_syn.nc decoded into NetCDF format
DDS|IDS ^SM.... .... ([0-3][0-9])
        PIPE    decoders/syn2nc
        etc/netcdf/syn.cdl
        data/netcdf/surface/synoptic
#
#--------------------------------------------------------------------
# grid data in netCDF format
#
# GFS grid on Lambert conformal CONUS grid replacing avn-q
HRS     ^Y.Q... KWB. (..)(..).*/m(AVN|SSIAVN|GFS|SSIGFS)
        PIPE    decoders/gribtonc
        etc/netcdf/gfs_211.cdl
        
data/netcdf/grid/NCEP/GFS/CONUS_80km/GFS_CONUS_80km_(\1:yyyy)(\1:mm)\1_\200.nc
#
# GFS grid on thinned grids, interpolated to global 5.0 x 2.5 regular grid
HRS     ^H.[I-P]... KWB. (..)(..).*/m(AVN|SSIAVN|GFS|SSIGFS)
        PIPE    decoders/gribtonc
        -q "lin,dlat=2.5,dlon=5.0" etc/netcdf/gfs_37-44.cdl
        
data/netcdf/grid/NCEP/GFS/Global_5x2p5deg/GFS_Global_5x2p5deg_(\1:yyyy)(\1:mm)\1_\200.nc
#
# ECMWF grid on global 2.5 x 2.5 degree grid
HRS     ^H.[A-L]... ECMF (..)(..)
        PIPE    decoders/gribtonc
        etc/netcdf/ecmwf_1-12.cdl
        
data/netcdf/grid/ECMWF/Global_2p5/ECMWF_Global_2p5_(\1:yyyy)(\1:mm)\1_\200.nc
#
# ETA grid on Lambert conformal CONUS grid
HRS     ^Y.Q... KWB. (..)(..).*/m(ETA|NAM)
        PIPE    decoders/gribtonc
        etc/netcdf/eta_211.cdl
        
data/netcdf/grid/NCEP/NAM/CONUS_80km/NAM_CONUS_80km_(\1:yyyy)(\1:mm)\1_\200.nc
#
# GFS grid on global 5.0 x 5.0 degree grid
HRS     ^H.[EF]... KWB. (..)(..)
        PIPE    decoders/gribtonc
        etc/netcdf/gfs_25-26.cdl
        
data/netcdf/grid/NCEP/GFS/Extended_Global_5p0deg/GFS_Extended_Global_5p0deg_(\1:yyyy)(\1:mm)\1_\200.nc
#
# NGM grid on Lambert conformal CONUS grid
#HRS    ^Y.Q... KWB. (..)(..).*/mNGM
#       PIPE    decoders/gribtonc
#       etc/ngm_211.cdl
#       
data/pub/decoded/netcdf/grid/NCEP/NGM/CONUS_80km/NGM_CONUS_80km_(\1:yyyy)(\1:mm)\1_\200.nc
#
# NOW and PWAV grids on global 5.0 x 2.5 degree grids
HRS     ^H.[A-D]... .... (..)(..).*(/mNOW|/mPWAV)
        PIPE    decoders/gribtonc
        etc/netcdf/ocean_21-24.cdl
        
data/netcdf/grid/NCEP/OCEAN/Global_5x2p5deg/OCEAN_Global_5x2p5deg_(\1:yyyy)(\1:mm)\1_\200.nc
#
# RUC grid on Lambert conformal CONUS grid
HRS     ^[YZ].Q... KWB. (..)(..).*/mRUC
        PIPE    decoders/gribtonc
        etc/netcdf/ruc_211.cdl
        
data/netcdf/grid/NCEP/RUC/CONUS_80km/RUC_CONUS_80km_(\1:yyyy)(\1:mm)\1_\200.nc
#
# RUC2 grid on Lambert conformal CONUS grid
HRS     ^Y.W... KWB. (..)(..).*/mRUC2
        PIPE    decoders/gribtonc
        etc/netcdf/ruc2_236.cdl
        
data/netcdf/grid/NCEP/RUC2/CONUS_40km/RUC2_CONUS_40km_(\1:yyyy)(\1:mm)\1_\200.nc
#
# SST grid on global 5.0 x 2.5 degree grids
HRS     ^H.[A-D]... .... (..)(..).*/mSST
        PIPE    decoders/gribtonc
        etc/netcdf/sst_21-24.cdl
        
data/netcdf/grid/NCEP/SST/Global_5x2p5deg/SST_Global_5x2p5deg_(\1:yyyy)(\1:mm)\1_\200.nc
#
# SST grid on global 2.0 x 2.0 degree grid
HRS     ^H.[T-W]... .... (..)(..).*/mSST
        PIPE    decoders/gribtonc
        etc/netcdf/sst_61-64.cdl
        
data/netcdf/grid/NCEP/SST/Global_2x2deg/SST_Global_2x2deg_(\1:yyyy)(\1:mm)\1_\200.nc
#

2005 messages navigation, sorted by:
1. Thread
2. Subject
3. Author
4. Date
5. ↑ Table Of Contents
Search the decoders archives: