You can run THREDDS indirectly through Apache. Apache does have
mechanisms to throttle requests. It is something we are looking at
presently - perhaps others have implemented it and can give some advice.
-Roy
On Dec 1, 2008, at 10:57 PM, Heiko Klein wrote:
Thanks all for your replies,
it is a pity that thredds/opendap cannot be configured to check
available memory before serving a request. We have enabled the
http-server, but this won't solve the problem of people pressing the
'Get Ascii' button and with that causing a DoS attack, accidentally
or not.
On the other hands, this answers my former question concerning
thredds-administration. I will start looking for a 64bit machine full
with memory.
Best regards,
Heiko
Ethan Davis wrote:
Hi Heiko,
Roy is right, the TDS requires a good amount of memory to handle
large
OPeNDAP data requests. (And the OPeNDAP ASCII requests, in
particular,
are very big memory hogs.) For downloads of entire files, you might
consider enabling straight HTTP access for your data files (more on
that
below).
On a 32-bit machine, there is 2GB of addressable memory. The JVM heap
space is limited at closer to 1.5 or 1.6 GB. A 64-bit machine/OS/JVM
will definitely give you more room to play.
Straight HTTP downloads ...
If you don't already have the TDS setup to serve data over HTTP, you
will need to modify the configuration catalogs to include the HTTP
service. It should look something like the following:
<service name="all" serviceType="Compound" base="" >
<service name="odap" serviceType="OpenDAP" base="/thredds/
dodsC/" />
<service name="http" serviceType="HTTPServer"
base="/thredds/fileServer/" />
</service>
with datasets referencing the serviceName "all".
More information can be found at
http://www.unidata.ucar.edu/projects/THREDDS/tech/tutorial/ConfigCatalogs.html
Ethan
Roy Mendelssohn wrote:
Yes, but in reality it doesn't appear to work on even that big a
setting. I believe some people have THREDDS working in 64-bit (see
for example
http://www.unidata.ucar.edu/mailing_lists/archives/thredds/2008-March/001008.html)
, but we do not. Our experience is that is takes a reasonably
larger amount of heap than the file size to deal with it.
Happy Thanksgiving Nathan,
-Roy
On Nov 27, 2008, at 10:01 AM, Nathan Potter wrote:
Isn't the JVM heap a capped at 2GB for 32-bit operating systems/
JVM's?
On Nov 27, 2008, at 7:19 AM, Roy Mendelssohn wrote:
I do believe THREDDS does all f its work in memory, if so you
have
to
greatly increase the java hep settings. We run at 1500Mb and
still
run into this. I would be curious to know if even with a very
large
heap size if file sof tha size can be successfully transferred.
-Roy
On Nov 27, 2008, at 7:11 AM, Heiko Klein wrote:
Hi,
I use thredds as opendap server for some large netcdf-files
(CF-1.0,
size > 1.1GB). When selecting to download the whole file with
the
'Get
ASCII' or 'GET BINARY', thredds crashes with a 'Java heap error'.
I'm currently using a default memory setup, i.e. -Xmx512MB for
tomcat.
How much memory will I need to avoid this problem. Does DODS read
the
whole file at once or variable after variable, or even slice
after
slice? Or are there some adjustments I can make to reduce memory
consumption?
Best regards,
Heiko
Details:
DODServlet ERROR (anyExceptionHandler):
java.lang.OutOfMemoryError:
Java
heap space
ReqState:
serverClassName: 'thredds.server.opendap.NcDODSServlet'
dataSet: 'XXXX.nc'
requestSuffix: 'ascii'
CE: ''
compressOK: true
InitParameters:
DebugOn: ''
maxNetcdfFilesCached: '10'
java.lang.OutOfMemoryError: Java heap space
at ucar.nc2.N3raf.readData(N3raf.java:77)
at ucar.nc2.N3iosp.readData(N3iosp.java:159)
at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:1374)
at ucar.nc2.Variable._read(Variable.java:923)
at ucar.nc2.Variable.read(Variable.java:618)
at thredds.server.opendap.NcSDArray.read(NcSDArray.java:102)
at thredds.server.opendap.NcSDGrid.read(NcSDGrid.java:59)
at opendap.servlet.AsciiWriter.writeAsc(AsciiWriter.java:95)
at opendap.servlet.AsciiWriter.toASCII(AsciiWriter.java:56)
at
opendap.servlet.AbstractServlet.doGetASC(AbstractServlet.java:
1029)
at
opendap.servlet.AbstractServlet.doGet(AbstractServlet.java:
1630)
at
thredds.server.opendap.NcDODSServlet.doGet(NcDODSServlet.java:
269)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:
698)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:
803)
...
Thredds-Version:
Built-By: caron
Built-On: 2008-02-20 22:56:05
Implementation-Title: THREDDS Data Server
Implementation-Version: 3.16.33
Implementation-Vendor: UCAR/Unidata
Running on tomcat 6.0.16 with jdk.1.6.0_04 on debian 4.0.
--
Dr. Heiko Klein Tel. + 47 22 96 32 58
Development Section / IT Department Fax. + 47 22 69 63 55
Norwegian Meteorological Institute http://www.met.no
P.O. Box 43 Blindern 0313 Oslo NORWAY
_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
**********************
"The contents of this message do not reflect any position of the U.S.
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097
e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/
"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
