[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #DDL-425820]: NLDN feed

Hi Takeshi,

re:
> We use LDM for receiving NLDN data every summer.   I set up our LDM
> server for this summer, and realized that we're receiving NLDN data
> every ~1 min instead of 6 min.  Output of `ldmadmin watch` is attached
> as "ldmadmin-watch.out".  The description of NLDN data feed on your website
> says that NLDN data is collected in a 6 min bin, and we received the
> data every 6 min in previous years.

There was a change made in the NLDN ingest and relay processing being done
at the University of Albany a couple of months ago.  This effort resurrected
the NLDN data that had been missing from since late August, 2010.

The data is no longer collected into 6-minute bins before being sent out.  The
result is that data comes in more timely now.  We will correct the NLDN data 
page
to remove the comment about the data being collected into 6-minute bins.  We
sincerely apologize for our oversight in overlooking this needed change
to the documentation!

re:
> Our problem is that now the data product we're receiving contains two
> headers as an attached file "201105032045.dat".  It makes difficult for
> us to interpret the data, because we store the binary data and, later on,
> convert it to ascii format with a Python script.  Our "pqact.conf" is
> attached as well.

You are naming the files you are saving using the system clock instead
of the date/time in the product IDs.  This means that any/all products
that match your pattern (which is (.*) or everything) and are received
in the same hour/minute/second will be written into a single output file.
This accounts for your output files having more than one header in them.

Your current pqact.conf action:

LIGHTNING   (.*)   FILE   /home/ldm/nldn/%Y%m%d%H%M.dat

We employ an action that uses the information in the Product IDs:

# (2006)(264)(23)(12)(17)
#  CCYY  JJJ  HH  MM  SS
LIGHTNING       
^([12][0-9][0-9][0-9]|[0-9][0-9])([0-3][0-9][0-9])([0-2][0-9])([0-5][0-9])([0-5][0-9])
        FILE    -close
        data/nldn/\1\2/\1\2.\3

You may want to adjust your pqact.conf action to use something more like ours.

re:
> Does LDM currently distribute NLDN data every ~1 min instead of 6 min?

It is actually distributed whenever it is available.  This is more like
every minute, but this interval is not guaranteed.

re:
> Is it normal that our receiving data contains two headers?

There are two different types of NLDN records that are being sent: CONUS
and "long range", and each of these types of records can be sent in
the same second (this matches your ldmadmin-watch.out output).  Even
though the records can be sent in the same second, they will have different
Product IDs.  By adopting a pattern-action approach more like ours, you
will avoid writing more than one received product into an output file.

re:
> Thanks for you help,

No worries.  Again, I apologize for not having correctly updated the
NLDN Lightning data page!


Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: DDL-425820
Department: Support LDM
Priority: Normal
Status: Closed