[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #WFA-619667]: Request for CONDUIT feed

Hi Martha,

re:
> It seems that we are now receiving the CONDUIT feed (I am attaching a
> portion of a notifyme log for your viewing).

Very good.

> What is our next step?

I would say that the first step is to review the data you are getting via
CONDUIT using the LDM utility 'notifyme'.  I would do this by capturing
the product IDs being received into a log file that you can review at your
leisure:

<as 'ldm'>
cd ~ldm
notifyme -vl logs/conduit.log -f CONDUIT

Let this run for a day and a half (to make sure you have a full day's worth
of data represented in the log file) and then take a look at the products
that you are receiving.  I would then fashion an extended regular expression
(ERE) that would identify just the products you want from the feed and
incorporate that ERE into the ~ldm/etc/ldmd.conf 'request' line you are
using for the CONDUIT data.

Comments:

1) If decide to ingest all of the CONDUIT data, you may need to split your
   CONDUIT feed request into multiple feed requests.  Here, for instance,
   is an example of splitting the CONDUIT requests 5-ways:

   in ~ldm/etc/ldmd.conf:

request CONDUIT "([09]$)" idd.cise-nsf.gov
request CONDUIT "([18]$)" idd.cise-nsf.gov
request CONDUIT "([27]$)" idd.cise-nsf.gov
request CONDUIT "([36]$)" idd.cise-nsf.gov
request CONDUIT "([45]$)" idd.cise-nsf.gov

2) Please remember that you will need to stop and restart your LDM for
   changes made in ldmd.conf to take effect!

3) Depending on how much data your LDM is trying to ingest/process, you may
   need to increase the size of your LDM product queue.  If you decide to
   ingest all of NOAAPort and all of CONDUIT, it would be a good idea for
   your LDM queue to be at least 2 GB in size and possibly larger.  Just how
   big your queue should be will be a strong function of:

   - how much RAM your system has
   - how much data you are ingesting
   - how much processing you are doing on ingested data

4) You will know if your processing is laboring if you start seeing messages
   in your ~ldm/logs/ldmd.log file that note that it took 'n' seconds to
   process products.  If this occurs on your system, you might either need
   to process less data or reorganize your processing.  We can talk about
   approaches to that if you run into the problem.

5) After you are satisfied with the set of data you are getting in CONDUIT,
   you should add their processing to the pattern-action file that you are
   using to process binary data for McIDAS.  This will look like:

HDS|NGRID|CONDUIT    .*    PIPE
    xcd_run DDS

   If you are still using the McIDAS XCD method to process Nexrad Level III data
   you will need to include the NNEXRAD feed in the action above.

   ** Please remember tat certain white space in pattern-action file actions are
   tabs, not spaces:

   - tab between HDS|NGRID|CONDUIT and '.*'
   - tab betweek '.*' and PIPE
   - tab before xcd_run

6) One of the very nice additions that was made to LDM-6.7+ was the automatic
   checking of all pattern-action files for syntax errors.  It would be a
   good idea, therefore, to upgrade your LDM installation to the latest
   LDM release, LDM-6.7.0.

   Upgrading the LDM should not take much time, and it does _not_ require that
   you redo your processing setup.  Here is the 1-2-3 for upgrading the LDM:

<as 'ldm'>
cd ~ldm

ftp ftp.unidata.ucar.edu
  <user> anonymous
  <pass> address@hidden
  cd pub/ldm
  binary
  get ldm-6.7.0.tar.gz
  quit

tar xvzf ldm-6.7.0.tar.gz
cd ldm-6.7.0/src
./configure
make
make install
sudo make install_setuids

cd
ldmadmin stop
rm runtime
ln -s ldm-6.7.0 runtime
ldmadmin start


> Thank you.

No worries.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: WFA-619667
Department: Support IDD
Priority: Normal
Status: Closed