Welcome back to AWIPS Tips!
Today we’re here to talk about the distributed architecture of Unidata’s EDEX. Initially, the goal of Unidata’s AWIPS team was to simplify the EDEX installation and design so that it could be run on one machine. This has been accomplished and is possible – with limitations. Due to the massive volume of data that is broadcast over our Internet Data Distribution (IDD) service, using a single machine to run EDEX and process all this data may not be possible (or at least, affordable). Because of these large datasets, at Unidata, we have moved to a Distributed EDEX architecture which allows the use of a few (in our case, three) EDEX machines that work together to ingest, decode, and serve as much data as we can to the public.
Our Distributed EDEX architecture takes advantage of two “types” of EDEX machines: Main EDEX and Ancillary EDEX.
Main EDEX
In short, this EDEX is capable of, with regards to any data:
- Requesting
- Ingesting
- Decoding
- Storing
- Serving
- Purging
A main EDEX installation has all the same capabilities of a standalone EDEX, like we talked about earlier. This machine has its own instance of the Local Data Manager (LDM) running which receives all the data defined from its ldmd.conf, and qpidd notifies EDEX the data are ready for ingest and decoding as defined by the entries in the pqact.conf. For more information about these configuration files, please refer to this previous blog. The main EDEX (edex_camel) then stores all the processed data as HDF5 files via httpd-pypies and postgres database entries. The processed data storage is managed by the purge rules, which define how long each file will stay on the machine before being deleted. Finally, this EDEX machine has a public facing URL or IP address (in our case: edex-cloud.unidata.ucar.edu) where it handles all data requests from both CAVE and python-awips connections.
NOTE: Be sure to edit the pqact.conf and remove any datasets you will be ingesting and decoding on the ancillary EDEX.
Ancillary EDEX
In short, this EDEX is capable of, with regards to a defined set of data:
- Requesting
- Ingesting
- Decoding
- Sending to a Main EDEX for storage
An ancillary EDEX installation has capabilities associated with receiving and decoding raw data. This machine will also have its own instance of LDM running, and should be requesting some related subset of data (satellite, model, radar, etc). It will then have a specific ingest mode running to decode that data. These decoded data are then sent to the main EDEX for storage (both as HDF5 files and as postgres database entries) and are then available to the public. All connections to this machine are on the internal network with the main EDEX machine, and it is not visible to the public.
Unidata's Current Architecture
Here at Unidata we have a main EDEX machine and two ancillary machines used to process radar and satellite data separately. The main EDEX machine ingests the largest amount of data at roughly 1TB per day, while the satellite EDEX machine ingests around 26GB per day, and the radar EDEX machine ingests about 9GB a day.
It is important to note, that while overall data volumes are a contributing factor to implementing a distributed EDEX system, the number of files is also important. For comparison, our main EDEX machine receives around 250,000 products per hour, our satellite EDEX receives approximately 7650 products per hour, and our radar EDEX receives just over 90,000 products per hour. Because of this discrepancy of size and throughput of the data feeds, the three EDEX machines vary in size.
This is important, because even though the radar EDEX has the smallest total volume of data per day, it was the first ancillary machine we needed to implement because of the large number of total radar products. Because of limitations in the original design of EDEX, a large number of products can drastically slow down an EDEX and potentially “fall behind” the data flow its receiving. By branching a significant number of these products off onto an ancillary machine, the main EDEX can still manage to keep up with its own data ingestion and still handle all of the data requests in a timely manner.
We hope this breakdown of the Distributed EDEX Architecture was useful to you. If you are running your own EDEX server(s), please don’t hesitate to reach out to us with any questions or assistance you may need. Check back in two weeks for the next blog post.
To view archived blogs, visit the AWIPS Tips blog tag, and get notified of the latest updates from the AWIPS team by signing up for the AWIPS mailing list. Questions or suggestions for the team on future topics? Let us know at support-awips@unidata.ucar.edu
This blog was posted in reference to v20.3.2-1 of NSF Unidata AWIPS
