[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Thredds out of memory

John Caron wrote:


Tennessee Leeuwenburg wrote:
For anyone who's interested, I am now quite happily serving large data sets via HTTP to thredds. The key was to re-write my java servlet in python. I think object creation overhead was killing me because of the sheer number of independant requests made by thredds. I used a default window size of 65,536 bytes, and was able to get 1.1Mb/s throughput downloading from thredds. My test file was 590Mb.
I like this approach, because my custom data source is neatly de-coupled from thredds code. All I have to do is deal with the HTTP protocol, and everything is handled for me.
If anyone else is having difficulty integrating a new data source into thredds, feel free to contact me for my code. 

Thanks for everyone's help...

Cheers,
-Tennessee
Are you sure the key was moving to python, and not moving to a 65K buffer?
That could well be the case, as I did both at once. Java is a lovely language, however I can also see how it could suffer in the face of repeated object creation when handling many small requests. In this case, however, the baby and the bathwater can't be easily differentiated :)
I'm going to stick with the Python code on the grounds that it works, and I've got more urgent things to do right now, but it could be that Java could do the same job. 

Cheers,
-T