[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Thredds out of memory
For small files, this actually works. For larger files, the interaction with the servlet breaks somehow, however the file sources from apache works okay.
I don't understand why this is the case. Software such as wget, firefox etc is happily able to download the file, resume partial downloads etc. Thredds is happily able to get smaller files. I fail to see why file size is affecting the system so badly.
generally a netcdf client like the thredds data viewer will treat the file as random access, and so may skip around in the file. if all you do is read the file sequentially, HTTP is ok. but for random access it can be really slow. Opendap is much better in this case.
That's exactly the goal - we want to use Opendap to give data to the various software clients that will use the data, in order to gain the many advantages offered. However, we have to get the files IN to thredds somehow.
yes, but the thredds AS server is a client of your HTTP server. When the AS server gets a request, it skips around the HTTP file to read it. So it depends what request the AS server gets, as to what its access pattern is.
try giving it very simple requests that are contiguous in the file and of known, reasonable size. those should work ok. then increase the size/complexity of your request and see where it degrades.
Is there any way you can give it access to the file directly, like through an NFS mount?
The file doesn't exist until the query is made, so, um, yes and no. It is no great difficulty to give the thredds machine access to the file, through NFS or even just by copying it to local disk using an executed script. However, in order to do that, I would have to modify thredds code, which means I get coupled to maintainance issues, learning the libraries etc etc. So there's no security or networking reason why not, but there are software design reasons why not.
I did actually write something like this at one stage, but I found the implementation to be exceedingly messy. I had to have independant configuration files, modify thredds internal code, etc etc. I ended up talking to Nathan Potter quite a lot while getting it working, and while he was very helpful, it was a lot more work than writing something neatly de-coupled like this. I'm reluctant to pursue a solution which requires code change to the thredds server, although it's not out of the question.
At the moment, the URL that thredds uses to access the file contains the database query information, so I can encode everything in the one place, which makes the system that much simpler to understand...
Is there a "magic number" in thredds which is a best window size to use? Would it "prefer" to get its data in any particular way? Thredds is basically the only client for this servlet, so I will just tune it for best performance. 


what do you mean by "window size" ?
When I'm serving data from my servlet, I create an 8k buffer which reads data from disk, then is flushed to the output stream.
you will see better performance as you increase this buffer size, at the cost of needing more heap space.
you should also tune the buffer size in HTTPRandomAccessFile, probably matching the sizes would be best.
I'll give that a go and see what happens. I don't really care about memory usage beyond avoiding OutOfMemoryExceptions... The machine it will be running on is pretty grunty, so it's just a tuning issue. 

Cheers,
-T

From address@hidden Fri Apr  1 10:03:27 2005
Received: (from majordo@localhost)
        by unidata.ucar.edu (UCAR/Unidata) id j31H23An002076
        for netcdf-java-out; Fri, 1 Apr 2005 10:02:03 -0700 (MST)
Received: from [127.0.0.1] (whoopee.unidata.ucar.edu [128.117.140.6])
        by unidata.ucar.edu (UCAR/Unidata) with ESMTP id j31H1sv3002037;
        Fri, 1 Apr 2005 10:01:55 -0700 (MST)
Organization: UCAR/Unidata
Keywords: 200504011701.j31H1sv3002037
Message-ID: <address@hidden>
Date: Fri, 01 Apr 2005 10:01:54 -0700
From: John Caron <address@hidden>
User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206)
X-Accept-Language: en-us, en
MIME-Version: 1.0
To: address@hidden
CC: netcdf-java <address@hidden>,
       address@hidden
Subject: Re: problem with writing netcdf file
References: <address@hidden>
In-Reply-To: <address@hidden>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Sender: address@hidden
Precedence: bulk

Hi tom:
nectdf-3 file format doesnt support "long" type, so NetcdfFileWriteable wont accept it. 
sorry the error message wasnt more useful.


Tom Whittaker wrote:
I tried joining the netcdf-java mailing list using the on-line form, but got no results emailed back to me...so I'm writing directly (sorry).
I'm trying to write my very first netcdf file from scratch using netcdf-2.2.jar. Just as I'm starting, I'm testing and it's throwing an interesting exception. I'm sure it's "Pilot Error", but I have no clue at this point. 

I'm using the current netcdf-2.2.jar file from the web site.
Here's the code (sorry it's in Jython, but that's what we're using these days -- if it turns out to be some data type conversion problems, then I'll have to find a way around that, but when I look at "DataType.LONG" it prints a "long" and is a class ucar.nc2.DataType): 

from ucar.nc2 import NetcdfFileWriteable
from ucar.nc2 import DataType

def makeNetCDFtest():

  ncfile = NetcdfFileWriteable()
  ncfile.setName("mydata.nc")
  oneDim = ncfile.addDimension("one",1)

  ncfile.addVariable("date", DataType.LONG, [oneDim,])
ncfile.addVariableAttribute("date","long_name","Date, in YYYYMMDD format") 

  ncfile.create()
  ncfile.close()

if __name__ == "__main__":
  makeNetCDFtest()
  print "done..."


Here's the exception:

raceback (innermost last):

  File "<string>", line 19, in ?
  File "<string>", line 15, in makeNetCDFtest

java.lang.IllegalStateException: unknown DataType == long
    at ucar.nc2.N3header.getType(N3header.java:405)
    at ucar.nc2.N3header.writeVars(N3header.java:586)
    at ucar.nc2.N3header.create(N3header.java:458)
    at ucar.nc2.N3iosp.create(N3iosp.java:255)
    at ucar.nc2.NetcdfFileWriteable.create(NetcdfFileWriteable.java:299)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
    at java.lang.reflect.Method.invoke(Method.java:324)
at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java) 
    at org.python.core.PyMethod.__call__(PyMethod.java)
    at org.python.core.PyObject.__call__(PyObject.java)
    at org.python.core.PyInstance.invoke(PyInstance.java)
    at org.python.pycode._pyx79.makeNetCDFtest$1(<string>:15)
    at org.python.pycode._pyx79.call_function(<string>)
    at org.python.core.PyTableCode.call(PyTableCode.java)
    at org.python.core.PyTableCode.call(PyTableCode.java)
    at org.python.core.PyFunction.__call__(PyFunction.java)
    at org.python.pycode._pyx79.f$0(<string>:19)
    at org.python.pycode._pyx79.call_function(<string>)
    at org.python.core.PyTableCode.call(PyTableCode.java)
    at org.python.core.PyCode.call(PyCode.java)
    at org.python.core.Py.runCode(Py.java)
    at org.python.core.Py.exec(Py.java)
    at org.python.util.PythonInterpreter.exec(PythonInterpreter.java)
    at org.jext.scripting.python.Run.execute(Unknown Source)
    at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
    at java.lang.reflect.Method.invoke(Method.java:324)
at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java) at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java) 
    at org.python.core.PyObject.__call__(PyObject.java)
    at org.python.core.PyObject.invoke(PyObject.java)
    at org.python.pycode._pyx78.f$0(<string>:3)
    at org.python.pycode._pyx78.call_function(<string>)
    at org.python.core.PyTableCode.call(PyTableCode.java)
    at org.python.core.PyCode.call(PyCode.java)
    at org.python.core.Py.runCode(Py.java)
    at org.python.core.Py.exec(Py.java)
    at org.python.util.PythonInterpreter.exec(PythonInterpreter.java)
    at org.jext.PythonAction.actionPerformed(Unknown Source)
at org.gjt.sp.jedit.textarea.InputHandler.executeAction(Unknown Source) at org.gjt.sp.jedit.textarea.DefaultInputHandler.keyPressed(DefaultInputHandler.java:259) at org.gjt.sp.jedit.textarea.JEditTextArea.processKeyEvent(Unknown Source) 
    at java.awt.Component.processEvent(Component.java:4909)
    at java.awt.Container.processEvent(Container.java:1569)
    at java.awt.Component.dispatchEventImpl(Component.java:3615)
    at java.awt.Container.dispatchEventImpl(Container.java:1627)
    at java.awt.Component.dispatchEvent(Component.java:3477)
at java.awt.KeyboardFocusManager.redispatchEvent(KeyboardFocusManager.java:1713) at java.awt.DefaultKeyboardFocusManager.dispatchKeyEvent(DefaultKeyboardFocusManager.java:627) at java.awt.DefaultKeyboardFocusManager.preDispatchKeyEvent(DefaultKeyboardFocusManager.java:831) at java.awt.DefaultKeyboardFocusManager.typeAheadAssertions(DefaultKeyboardFocusManager.java:741) at java.awt.DefaultKeyboardFocusManager.dispatchEvent(DefaultKeyboardFocusManager.java:592) 
    at java.awt.Component.dispatchEventImpl(Component.java:3506)
    at java.awt.Container.dispatchEventImpl(Container.java:1627)
    at java.awt.Window.dispatchEventImpl(Window.java:1606)
    at java.awt.Component.dispatchEvent(Component.java:3477)
    at java.awt.EventQueue.dispatchEvent(EventQueue.java:456)
at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchThread.java:201) at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:151) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:145) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:137) 
    at java.awt.EventDispatchThread.run(EventDispatchThread.java:100)
java.lang.IllegalStateException: java.lang.IllegalStateException: unknown DataType == long