Xmx likely needs to be more in the range of 1200Mb in our experience.
-Roy
On Jan 12, 2009, at 12:27 PM, Ken Tanaka wrote:
John and Roy,
It looks like the current Xmx memory setting is 256 MB for the
tomcat process. I'll arrange to have the memory increased, but we
were concerned that we would still run out of memory--just at a
higher number of files. Earlier posts indicated that I should be
able to aggregate a large number of files (over 2000), so I wasn't
sure if I was configuring the server correctly. Should I delve into
caching or types of performance tuning? We would like to add a file
once a day and to an unlimited Time coordinate, our joinExisting
dimension, indefinitely.
We were thinking of moving a large number of Global Attributes
(metadata) out of the files and into an NcML file since most of it
is the same (constant) in all the files. We felt that it made sense
to place the aggregation and attributes into the NcML file rather
than clutter the catalog.xml in this case. Will this save memory to
abstract out the repeated Global Attributes and provide them in a
single NcML wrapper?
-Ken
John Caron wrote:
Hi Ken:
How much memory are you giving the JVM?
These kinds of aggregations should not consume memory, but there
are some bugs in there.
You configuration looks ok, although you could embed the ncml
directly into the catalog:
<dataset name="SST Aerosol Aggregation NOAA-18(N)" ID="SST-Aerosol-
NN-Agg"
urlPath="sst-aerosol-nn-aggregation">
<metadata inherited="true">
<serviceName>multiple</serviceName>
<dataType>Grid</dataType>
</metadata>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2
" >
<attribute name="myAttribute" value="NN-attribute" />
<aggregation dimName="Time" type="joinExisting">
<scan location="/data1/thredds/sst/Aerosol_NN_test/"
suffix=".nc" />
</aggregation>
</netcdf>
</dataset>
I dont think that should matter, but if its easy, why dont you try
that?
Meanwhilie ill try to reproduce the problem.
Ken Tanaka wrote:
Hi,
I'm encountering "java.lang.OutOfMemoryError: Java heap space"
errors in
aggregating netcdf files. The messages are appearing in the
tomcat5/content/thredds/logs/threddsServlet.log and the web page
will
return a web page showing
Error { code = -1; message = "Server Error on dataset sst-aerosol-
nl-aggregation"; };
instead of the normal OPENDAP access page (
http://www.ngdc.noaa.gov/thredds/dodsC/sst-aerosol-nl-aggregation.html
). This page usually returns normally for a few requests, but once
the
OutOfMemoryError occurs the service is down until I restart
tomcat. I
did try an upgrade to the latest THREDDS warfile, 3.16.48.0, but
still
see this error. The TDS is running on Red Hat Enterprise Linux WS
release 3 (Taroon Update 9). The java.vm.version = 1.5.0_06-b05, and
tomcat version is 5.5.17.
The files we are serving up are SST Aerosol products, with a Time
coordinate, with a single time slice per file. The typical
filesize is
3.6MB. We have quite a bit of metadata as Global Attributes in
each, but
I think we can abstract that out with an NcML wrapper file. Here
is the
wrapper file, which is in the same directory with the data files:
<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2
" >
<attribute name="myAttribute" value="NN-attribute" />
<aggregation dimName="Time" type="joinExisting">
<scan location="/data1/thredds/sst/Aerosol_NN_test/"
suffix=".nc" />
</aggregation>
</netcdf>
And here is the portion of the catalog.xml that uses this
aggregation:
<!-- Aerosol NN, joinExisting with ncml wrapper
-->
<dataset name="SST Aerosol Aggregation NOAA-18(N)" ID="SST-
Aerosol-NN-Agg"
urlPath="sst-aerosol-nn-aggregation">
<metadata inherited="true">
<serviceName>multiple</serviceName>
<dataType>Grid</dataType>
</metadata>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2
"
location="/data1/thredds/sst/Aerosol_NN_test/
Aerosol_NN_test.ncml" >
</netcdf>
</dataset>
Do you see any troubles with my configuration? Earlier posts to this
list indicated that there should not be a problem with aggregating a
large number of files. This particular aggregation is 245 files. I
definitely get errors right away if I also set up separate but
similar
aggregation with 135 files in the same catalog.
Thanks for any suggestions,
Ken
_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
--
= Enterprise Data Services Division ===============
| CIRES, National Geophysical Data Center / NOAA |
| 303-497-6221 |
= Ken.Tanaka@xxxxxxxx =============================
_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
**********************
"The contents of this message do not reflect any position of the U.S.
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097
e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/
"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
