I'm having some issues with THREDDS and a very high number of
interrupts/context switches killing the server's performance. Increasing the
LOG level shows the high switching related to thredds.filesystem.ControllerOS.
As soon as the messages pasted below begin, the interrupt/context switch number
jumps from <100 to nearly 20,000. CPU and memory do not appear to be a limiting
factor, their loads remain low. The filesystem is mounted via NFS, but
read/write speeds (via dd) seem fine, greater than 150 megabytes/s. Those
speeds are of course without the massive context switching overhead.
Despite <update startup ="nocheck" /> being declared (see catalog below),
ControllerOS seems to be called in a loop somewhere and appears to be listing
every directory and file within and repeats itself once complete.
Is there a way to avoid such heavy context switching related to this listing of
files, possibly by consolidating the files to a single location, or disabling
the 'listing' being done by ControllerOS all together?
Thanks,
Brian McKenna
threddsServlet.log
-----------------------
2013-07-31T09:06:53.614 -0400 [ 11067][ ] DEBUG -
thredds.filesystem.ControllerOS - List Directory
/data/maracoos/ensemble/WRF_ensemble_small
2013-07-31T09:06:53.630 -0400 [ 11083][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble_small/2012.02.10.00
2013-07-31T09:06:53.633 -0400 [ 11086][ ] DEBUG -
thredds.filesystem.ControllerOS - List Directory
/data/maracoos/ensemble/WRF_ensemble
2013-07-31T09:06:53.634 -0400 [ 11087][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble_small/2012.06.03.00
2013-07-31T09:06:53.638 -0400 [ 11091][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble_small/2012.09.30.00
2013-07-31T09:06:53.641 -0400 [ 11094][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble_small/2012.03.05.00
2013-07-31T09:06:53.645 -0400 [ 11098][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble_small/2012.03.14.12
2013-07-31T09:06:53.648 -0400 [ 11101][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble_small/2012.10.05.12
2013-07-31T09:06:53.651 -0400 [ 11104][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble_small/2012.10.01.00
2013-07-31T09:06:53.653 -0400 [ 11106][ ] DEBUG -
thredds.filesystem.ControllerOS - List Directory /data/maracoos/ensemble/AKQ_WRF
2013-07-31T09:06:53.659 -0400 [ 11112][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble/2012.02.10.00
2013-07-31T09:06:53.660 -0400 [ 11113][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble_small/2012.06.01.00
2013-07-31T09:06:53.664 -0400 [ 11117][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble/2012.06.03.00
2013-07-31T09:06:53.664 -0400 [ 11117][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble_small/2012.11.16.00
2013-07-31T09:06:53.668 -0400 [ 11121][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble/2011.12.19.00
2013-07-31T09:06:53.668 -0400 [ 11121][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble_small/2012.05.04.12
featureCollection from catalog
---------------------
<featureCollection featureType="FMRC" name="Ensemble WRF (large grid) Forecast
Aggregation Collection" harvest="true" path="WRF-Large/Agg-Collection.nc">
<metadata inherited="true">
<serviceName>agg</serviceName>
<dataFormat>NetCDF</dataFormat>
</metadata>
<collection
spec="/data/maracoos/ensemble/WRF_ensemble/**/#yyyy.MM.dd.HH#.*\.nc$" name="
WRF_Large_Ensemble" olderThan="5 min" />
<update startup="nocheck" trigger="allow" />
<fmrcConfig regularize="true" datasetTypes="Best Files Runs" />
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
<variable name="lat" orgName="g0_lat_0">
<attribute name="standard_name" value="latitude" />
</variable>
<variable name="lon" orgName="g0_lon_1">
<attribute name="standard_name" value="longitude" />
</variable>
<variable name="u" orgName="U_GRD_GDS0_HTGL_10">
<attribute name="standard_name" value="eastward_wind_velocity" />
<attribute name="coordinates" value="time lat lon" />
</variable>
<variable name="v" orgName="V_GRD_GDS0_HTGL_10">
<attribute name="standard_name" value="northward_wind_velocity" />
<attribute name="coordinates" value="time lat lon" />
</variable>
<variable name="pressure" orgName="PRMSL_GDS0_SFC_10">
<attribute name="coordinates" value="time lat lon" />
</variable>
<variable name="temperature" orgName="TMP_GDS0_HTGL_10">
<attribute name="coordinates" value="time lat lon" />
</variable>
<variable name="humidity" orgName="R_H_GDS0_HTGL_10">
<attribute name="coordinates" value="time lat lon" />
</variable>
</netcdf>
</featureCollection>
