Re: Questions about HttpClient library

Kris Nuttycombe wrote:

Hi, John,

I work for Ted Habermann over here at the National Geophysical Data Center, and I'm in the process of doing some work with your NetCDF libraries. I'd like to write a patch for your library that permits NetCDF files to be read over an FTP connection, in addition to the functionality you've provided for reading NetCDF remotely over HTTP. I have most of what I need from the source code that you distributed, but the only thing I'm lacking are javadocs for the HttpClient library so that I know what I need to drop in for the RandomAccessFile subclass I'm writing to emulate the functionality you use in HTTPRandomAccessFile exactly.

Can you point me to either the HTTPClient source code or the Javadocs directly? Also, I'm using the jakarta commons.net.ftp library in my implementation, so my patch (if you choose to accept it) will add that dependency if the user want's to use these methods.

Thanks,

Kris Nuttycombe

Hi Kris, that sounds great, we will be very excited to include your improvements in our package.

The HttpClient package lives here:

 http://jakarta.apache.org/commons/httpclient/

The overall idea is that one uses a replacement for RandomAccessFile, so that whenever the system calls for a disk access, an HTTP access is done instead. So all we need is to be able to randomly access bytes in a file on an HTTP server, which we can through the GetRange HTTP command. So to do the same for FTP, we just need to use the equivilent FTP call, whatever that is. So probably you dont needt ot study the HttpClient code very closely, just look at RandomAccessFile and see where it makes disk calls.

Of course, we dont really want to make a network call for everything, so RandomAccessFile, (and HTTPRandomAccessFile) tries to intelligently buffer. I am intending on trying to optimize that a bit more, so if you have an ideas, try them out. I think the main thing would be to try to read the entire header all at once, so perhaps the header parser needs to estimate the header size, and twiddle some buffer sizes in the RandomAccessFile.

Im quite sure we dont need all the functionality of HttpClient, but i havent had time to figure out how to replace it or reduce it (mostly just to reduce the footprint).

let me know what else i can help you with, and how it goes.

thanks again,

John.




  • 2003 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: