Hi folks,
I have tried installing the current EDEX server on a new virtual machine
running RHEL 7.9. The VM has 20 cores allocated, 32 GB RAM, and the /awips2
path is a file system running on a pair of NVMe sticks on the host motherboard.
This is the same configuration that had been running the old 18.x server (under
a RHEL 6.x OS) for a number of years.
I have tried installing the software multiple times, always assuming that I
have done something wrong, but the system load continues to rise over time. I
have tracked it down to what looks like a file permission problem, where the
data files are owned by root, but the data processing scripts that want to move
files are run by the awips user.
Here are some pieces of evidence.
[root@edex ~]# grep
/awips2/data_store/radar/20240316/08/VAX/VAX_N1X_20240316_0829
/awips2/ldm/dev/logs/purgeRadar.log
find: cannot delete
‘/awips2/data_store/radar/20240316/08/VAX/VAX_N1X_20240316_0829’: Permission
denied
find: cannot delete
‘/awips2/data_store/radar/20240316/08/VAX/VAX_N1X_20240316_0829’: Permission
denied
find: cannot delete
‘/awips2/data_store/radar/20240316/08/VAX/VAX_N1X_20240316_0829’: Permission
denied
find: cannot delete
‘/awips2/data_store/radar/20240316/08/VAX/VAX_N1X_20240316_0829’: Permission
denied
find: cannot delete
‘/awips2/data_store/radar/20240316/08/VAX/VAX_N1X_20240316_0829’: Permission
denied
find: cannot delete
‘/awips2/data_store/radar/20240316/08/VAX/VAX_N1X_20240316_0829’: Permission
denied
find: cannot delete
‘/awips2/data_store/radar/20240316/08/VAX/VAX_N1X_20240316_0829’: Permission
denied
find: cannot delete
‘/awips2/data_store/radar/20240316/08/VAX/VAX_N1X_20240316_0829’: Permission
denied
find: cannot delete
‘/awips2/data_store/radar/20240316/08/VAX/VAX_N1X_20240316_0829’: Permission
denied
find: cannot delete
‘/awips2/data_store/radar/20240316/08/VAX/VAX_N1X_20240316_0829’: Permission
denied
....
There are 549 instances of this error in the log.
[root@edex 20240316]# ll
/awips2/data_store/radar/20240316/08/VAX/VAX_N1X_20240316_0829
/awips2/ldm/dev/logs/purgeRadar.log
-rw-r--r-- 1 root root 74507 Mar 16 04:31
/awips2/data_store/radar/20240316/08/VAX/VAX_N1X_20240316_0829
-rw-rw-r-- 1 awips fxalpha 421414977804 Mar 18 17:56
/awips2/ldm/dev/logs/purgeRadar.log
(That's a 400 GB log file of errors.)
I suspect this is the underlying issue, as the load on the machine went to
>1500 (yes, 1500) when I had previously stopped all of the EDEX server process,
including the LDM the night before. With no new data the load went from zero to
that value overnight. I realized what the problem was by using the ps command
to see the many stacked-up Python processes. (Launched from the awips user's
crontab entries.)
So, should there be some setting that makes the files writeable to the awips
user? I looked in the install script, and I don't see any setuid or umask
efforts. What step(s) have I failed to make?
I have scrubbed prior installations. I had some issues with group install and
rpm where the install aborted at the sed edit for the ldm registry. I think
that things are "properly installed" now, but perhaps there was a post install
event that didn't execute properly, or some other magic I failed to notice. I
did run the regutil commands to modify the queue location and size, used the
ldmadmin command and notifyme command to query both the locally running ldm and
remote sources. That stuff does seem to be working as expected, perhaps.
Here are the installed packages.
[root@edex ~]# rpm -qa | grep awips2-edex| sort
awips2-edex-20.3.2-2.x86_64
awips2-edex-alertviz-20.3.2-2.x86_64
awips2-edex-archive-20.3.2-2.x86_64
awips2-edex-base-20.3.2-2.x86_64
awips2-edex-bufr-20.3.2-2.x86_64
awips2-edex-common-core-20.3.2-2.x86_64
awips2-edex-config-auto-20.3.2-2.x86_64
awips2-edex-configuration-20.3.2-2.x86_64
awips2-edex-core-20.3.2-2.x86_64
awips2-edex-dataplugins-20.3.2-2.x86_64
awips2-edex-foss-20.3.2-2.x86_64
awips2-edex-foss-com-beust-jcommander-20.3.2-2.x86_64
awips2-edex-foss-com-google-protobuf-20.3.2-2.x86_64
awips2-edex-foss-com-mchange-20.3.2-2.x86_64
awips2-edex-foss-com-sun-jna-20.3.2-2.x86_64
awips2-edex-foss-edu-mit-ll-netcdf-20.3.2-2.x86_64
awips2-edex-foss-javax-activation-20.3.2-2.x86_64
awips2-edex-foss-javax-mail-20.3.2-2.x86_64
awips2-edex-foss-javax-media-opengl-20.3.2-2.x86_64
awips2-edex-foss-javax-servlet-20.3.2-2.x86_64
awips2-edex-foss-net-dods-20.3.2-2.x86_64
awips2-edex-foss-net-opengis-20.3.2-2.x86_64
awips2-edex-foss-net-opengis_v2-20.3.2-2.x86_64
awips2-edex-foss-net-sf-ehcache-20.3.2-2.x86_64
awips2-edex-foss-ogc-tools-gml-20.3.2-2.x86_64
awips2-edex-foss-org-apache-activemq-20.3.2-2.x86_64
awips2-edex-foss-org-apache-camel-20.3.2-2.x86_64
awips2-edex-foss-org-apache-commons-cli-20.3.2-2.x86_64
awips2-edex-foss-org-apache-commons-compress-20.3.2-2.x86_64
awips2-edex-foss-org-apache-commons-management-20.3.2-2.x86_64
awips2-edex-foss-org-apache-commons-net-20.3.2-2.x86_64
awips2-edex-foss-org-apache-commons-ssl-20.3.2-2.x86_64
awips2-edex-foss-org-apache-commons-validator-20.3.2-2.x86_64
awips2-edex-foss-org-apache-shiro-20.3.2-2.x86_64
awips2-edex-foss-org-apache-ws-commons-schema-20.3.2-2.x86_64
awips2-edex-foss-org-apache-ws-security-20.3.2-2.x86_64
awips2-edex-foss-org-apache-xalan-20.3.2-2.x86_64
awips2-edex-foss-org-apache-xml-resolver-20.3.2-2.x86_64
awips2-edex-foss-org-apache-xml-security-20.3.2-2.x86_64
awips2-edex-foss-org-apache-xml-serializer-20.3.2-2.x86_64
awips2-edex-foss-org-dom4j-20.3.2-2.x86_64
awips2-edex-foss-org-eclipse-jetty-20.3.2-2.x86_64
awips2-edex-foss-org-itadaki-bzip2-20.3.2-2.x86_64
awips2-edex-foss-org-jasypt-20.3.2-2.x86_64
awips2-edex-foss-org-jfreechart-20.3.2-2.x86_64
awips2-edex-foss-org-joda-time-20.3.2-2.x86_64
awips2-edex-foss-org-objectweb-20.3.2-2.x86_64
awips2-edex-foss-org-opensaml-20.3.2-2.x86_64
awips2-edex-foss-org-w3-xmlschema-20.3.2-2.x86_64
awips2-edex-foss-ucar-nc2-20.3.2-2.x86_64
awips2-edex-gfe-20.3.2-2.x86_64
awips2-edex-goesr-20.3.2-2.x86_64
awips2-edex-grid-20.3.2-2.x86_64
awips2-edex-griddednucaps-20.3.2-2.x86_64
awips2-edex-ncep-20.3.2-2.x86_64
awips2-edex-npp-20.3.2-2.x86_64
awips2-edex-nswrc-radar-20.3.2-2.x86_64
awips2-edex-ost-20.3.2-2.x86_64
awips2-edex-probsevere-20.3.2-2.x86_64
awips2-edex-radar-20.3.2-2.x86_64
awips2-edex-registry-client-20.3.2-2.x86_64
awips2-edex-remote-script-20.3.2-2.x86_64
awips2-edex-satellite-20.3.2-2.x86_64
awips2-edex-shapefiles-20.3.2-2.noarch
awips2-edex-text-20.3.2-2.x86_64
awips2-edex-warning-20.3.2-2.x86_64
Any and all input is welcome. I only pretend to know what I'm doing. (-:
Thanks!
Chris
Christopher Herbster, Ph.D.
Associate Professor of Meteorology
Applied Aviation Science
Daytona Beach Campus
College of Aviation Rm. 338
1 Aerospace Boulevard
Daytona Beach, FL 32114
386.226.6444
christopher.herbster@xxxxxxxx<mailto:christopher.herbster@xxxxxxxx>
Embry-Riddle Aeronautical University
Florida | Arizona | Worldwide
[cid:67e2a38d-2c29-4d74-aece-f339b8120cb3]
386.226.6446 Weather Center
http://wx.erau.edu/
Schedule at: http://wx.erau.edu/faculty/herbster/Schedules/
