Welcome back to AWIPS Tips!
Today we’re going to talk about the Local Data Manager (LDM) and how it’s used within AWIPS. The LDM is also developed by Unidata, and is a powerful software tool used to acquire and distribute data between computers. LDM can be installed independently, however within AWIPS, it is included in the base EDEX installation along with EDEXbridge and many other software components. On EDEX machines LDM is installed to /awips2/ldm.
Basic overview of the LDM process within AWIPS:
- Data are requested from an upstream LDM
- Data are stored locally
- edexBridge (an AWIPS-specific component that gets installed with awips2-ldm) notifies QPID the data are available for EDEX to process/ingest
NOTE: Our data are not publicly available. In order to receive requested data, permission from upstream LDM must be granted. Specifically, Unidata’s Internet Data Distribution (IDD) is made available to the University (edu) domain.
In this process, there are several ways to customize and optimize the data ingested by EDEX. There are a few important configuration files that need to be set-up to ensure that you are receiving and ingesting the data you want.ldmd.conf
The ldmd.conf configuration file can be found at /awips2/ldm/etc/ldmd.conf and provides top level information with a few specific command entries. General documentation on this file can be found on the LDM documentation website. There are two important functions that we'll explain next, the EXEC and REQUEST functions.
The EXEC command executes the specified programs on start-up and can send arguments like running from a command line:
EXEC "pqact -e"
Result: The pqact program gets executed
NOTE: This example is for our current production version of AWIPS (v18). Our beta version (v20) uses different syntax.
EXEC "edexBridge -vxl /awips2/ldm/logs/edexBridge.log -s localhost"
Result: edexBridge gets executed, specifies where to write out the log information, and specifies the host as localhost - this is specific to AWIPS only
EXEC "rtstats -h rtstats.unidata.ucar.edu"
Result: rtstats gets executed and sends LDM stats to Unidata
The REQUEST entry specifies which products you are requesting from which upstream LDM. You can specify the feedtype and pattern you are requesting.
In the example below, it shows instances of pulling all data from a feed – where you see the ".*" entries – and instances of more specific requests – "^TI[A-W]... KNES", for example. This configuration file makes use of POSIX extended regular expressions.
NOTE: The idd.unidata.ucar.edu LDM feed is not public, but is made available to educational institutions.
REQUEST FNEXRAD ".*" idd.unidata.ucar.edu # MRMS - Unidata feed via NCEP
REQUEST NEXRAD3 ".*" idd.unidata.ucar.edu # Radar Level3
REQUEST WMO ".*" idd.unidata.ucar.edu # WMO Feedtype includes HDS|IDS|DDPLUS
REQUEST UNIWISC|NIMAGE ".*" idd.unidata.ucar.edu # AREA/GINI and GOES (NIMAGE feedtype includes DIFAX)
REQUEST NOTHER "^TI[A-W]... KNES" idd.unidata.ucar.edu # VIIRS and GOES CMI via SBN
REQUEST NOTHER "^IXT[WXY]01" idd.unidata.ucar.edu #Special SBN GOES Derived
REQUEST NGRID ".*" idd.unidata.ucar.edu
pqact.conf
The pqact.conf file can be found at /awips2/ldm/etc/pqact.conf and provides information on where to store the raw data products to be processed. General documentation on this file can be found on this LDM webpage. Below is an example entry:
NOTE: Syntax is very important! This file uses tabs (not spaces) between different actions.
# GOES 16/17 derived products + derived motion wind via SBN - (using)
HDS ^(IXT.[8-9]9) (KNES) (..)(..)(..)
FILE -close -edex /awips2/data_store/GOES/(\3:yyyy)(\3:mm)\3/\4/derived-SBN/\1_KNES_\2\3\4\5-(seq)
This entry is saying if the feedtype matches “HDS” and the filename or WMO header matches the productId’s Extended Regular Expression, then “FILE” (an action which writes the file to disk at the specified path), then close the file, and have edexBridge notify QPID the file is ready to be processed.
While we don’t currently make use of it, the pqact program also has the EXEC action to execute other programs or scripts when a product comes in. In the example below, when the Lifted Index product comes in, the L2Remap script is executed with the arguments of the filename and channel/band:
# Derived Stability Indices - Lifted Index - Band 8
NIMAGE (OR_ABI-L2-DSIC.*)
EXEC mcidas/bin/L2Remap \1 8
It is important to remember that pqactions will only be executed on the data you’re requesting, which is very dependent on the contents of your ldmd.conf configuration.
Ideally, you’ve refined the ldmd.conf to specifically pull just the data you’re interested in, and then the pqact.conf only has actions for that same data.
Managing LDM
Typically within the direct LDM install and National Weather Service AWIPS distribution, the proper way to start and stop the LDM process are:
ldmadmin start
ldmadmin stop
However, in the Unidata AWIPS distribution, the LDM process gets started when EDEX is started (sudo edex start).
To manually stop and start you can use (this is necessary if a change to the ldmd.conf has been made):
sudo service edex_ldm start
sudo service edex_ldm stop
If a small change has been made to the pqact.conf, you can update ldm without needing to restart it:
- run ldmadmin pqactcheck
- If there were no errors in the previous step, run ldmadmin pqactHUP
- If there were errors indicated, re-edit the pattern action file and repeat the above steps
Thanks for joining us and check back in two weeks for the next blog post.
To view archived blogs, visit the AWIPS Tips blog tag, and get notified of the latest updates from the AWIPS team by signing up for the AWIPS mailing list. Questions or suggestions for the team on future topics? Let us know at support-awips@unidata.ucar.edu
