Re: [thredds] "Too Many Open Files" Error. Dataset too big?

• To: Kevin Manross <manross@xxxxxxxx>
• Subject: Re: [thredds] "Too Many Open Files" Error. Dataset too big?
• From: Michael McDonald <mcdonald@xxxxxxxxxxxxx>
• Date: Thu, 24 Oct 2013 17:09:23 -0400

Kevin,
We needed to add a script to "/etc/profile.d" that increases the stock
CentOS/RHEL 6.x number of open files. Change $USER == "tomcat" to
whatever you run tomcat/apache as (see below). Also assumes that the
user is running the sh/bash shell.

# cat /etc/profile.d/thredds.sh
export TOMCAT_HOME="/usr/share/tomcat6"
if [ $USER == "tomcat" ]; then ulimit -n 1000000; fi


Our THREDDS servers providing only OPENDAP requests are averaging 40k
open files. Our THREDDS servers providing only NCSS requests are
averaging 15k open files. We have a lot of files in some of our hycom
datasets.

/mike

On Thu, Oct 24, 2013 at 1:50 PM, Kevin Manross <manross@xxxxxxxx> wrote:
>
> Hi there,
>
> I am trying to aggregate netcdf files for some rather large datasets with
> tomcat6.0.36/TDS4.19 - 20131008.1358 and am running into a "Too many open
> files" error.
>
> My catalog scan is:
>
>   <dataset name="ASR 30 km 2d surface analysis"
>            ID="ds631.0_anl_2d-AGG"
>            urlPath="ds631.0/9/best"
>            harvest="true">
>      <serviceName>all</serviceName>
>      <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
>        <aggregation dimName="time" type="Union">
>          <scan location="/glade/p/rda/data/ds631.0/asr30km.anl.2d/"
> suffix=".nc" subdirs="false"/>
>        </aggregation>
>      </netcdf>
>    </dataset>
>
> which contains about 4020 files and totals 470G.
>
> The message that the server sends when trying to access the OPeNDAP service
> is:
>
>
> Error {
>     code = 404;
>     message =
> "/glade/p/rda/data/ds631.0/asr30km.anl.2d/asr30km.anl.2d.20010619.nc (Too
> many open files)";
> };
>
>
>
> When trying the CdmRemote service catalina.out repeatedly reads:
>
> Oct 24, 2013 11:20:39 AM org.apache.jk.common.ChannelSocket
> acceptConnections
> WARNING: Exception executing accept
> java.net.SocketException: Too many open files
>     at java.net.PlainSocketImpl.socketAccept(Native Method)
>     at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:375)
>     at java.net.ServerSocket.implAccept(ServerSocket.java:478)
>     at java.net.ServerSocket.accept(ServerSocket.java:446)
>     at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:311)
>     at
> org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:668)
>     at
> org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:879)
>     at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
>     at java.lang.Thread.run(Thread.java:679)
>
>
> The cache settings in my threddsConfig.xml are the default settings.
>
> Have I overlooked something?
>
> -kevin.
>
>
> --
> Kevin Manross
> NCAR/CISL/Data Support Section
> Phone: (303)-497-1218
> Email:manross@xxxxxxxx
> Web:http://rda.ucar.edu
>
> _______________________________________________
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit:
> http://www.unidata.ucar.edu/mailing_lists/

Attachment: ncss2.hycom.org_num_open_files.png
Description: PNG image

Attachment: ncss1.hycom.org_num_open_files.png
Description: PNG image

Attachment: tds2.hycom.org_num_open_files.png
Description: PNG image

Attachment: tds1.hycom.org_num_open_files.png
Description: PNG image

• Follow-Ups:
  • Re: [thredds] "Too Many Open Files" Error. Dataset too big?
    • From: Kevin Manross

• References:
  • [thredds] "Too Many Open Files" Error. Dataset too big?
    • From: Kevin Manross