Welcome back to AWIPS Tips!
Today we’re going to discuss an alternative method to ingest data into EDEX outside of edexBridge. The most standard way of inserting data into the EDEX processing queue is by using the -edex command in your pqact entry when receiving data through the LDM.
Tip: You can read more about the pqact.conf and other important LDM configurations in our previous AWIPS Tips blog post.
There are certain instances where you may want to ingest data when the data is not available on LDM, you don’t have access to the specific LDM feed, or you want to do some preprocessing of the data received from LDM before handing it off to EDEX for ingest, to name a few.
Now, if you’re simply testing a new data source and only have a few files to ingest, you can use this method, or the manual entry point (at /awips2/edex/data/manual/). However, if you want to set up a production process that will be continually ingesting data, we recommend using the method described below, because the manual entry point is not an efficient way to continually ingest data.
To set up an automated ingest process, you can use a python script that’s included with EDEX called qpidNotify.py. This file is located in /awips2/fxa/bin/src/qpidNotify/qpidNotify.py. The script takes one argument – the fully qualified file path and name. If this is running in a script, you have to make sure to call the script with python. For example:
/awips2/python/bin/python /awips2/fxa/bin/src/qpidNotify/qpidNotify.py /awips2/data_store/test/myNewFile.grib
Note: The command above would be executed all in one line, even if it shows up on multiple lines in the browser currently.
This call can now be incorporated into an automated, production-ready process to ingest your own data, outside of the standard LDM process with edexBridge. For example, we receive gridded model data via LDM, but instead of passing them directly to edexBridge, we just store and concatenate them by not using the -edex flag.
Then we have a cronjob running on EDEX which executes the /awips2/ldm/dev/checkFileTime.sh bash script. That script calls the qpidNotify.py script with the newly concatenated grib data and the data is inserted back into the Qpid queue for EDEX to decode and ingest.
Thanks for taking the time to learn a little more about working with your own EDEX server! Check back in two weeks for the next blog post.
To view archived blogs, visit the AWIPS Tips blog tag, and get notified of the latest updates from the AWIPS team by signing up for the AWIPS mailing list. Questions or suggestions for the team on future topics? Let us know at support-awips@unidata.ucar.edu
This blog was posted in reference to v20.3.2-2 of NSF Unidata AWIPS
