Squid is a proxy server -- it supports byte-range requests, and can thus
provide that service for a server than does not support byte-range
requests. Configuration is a bit tricky -- you need to ensure that squid
will download and cache the entire file for the initial partial requests,
otherwise it will simply convert each partial request into a complete
request and discard the rest.
On Mon, Jun 10, 2013 at 3:12 PM, Ethan Davis <edavis@xxxxxxxxxxxxxxxx>wrote:
> Hi Joe,
>
> HTTP 1.1 can support requests for byte-range [a] subsets of a resource.
> This capability is optional so not all servers (or resources on a
> server) need support it. If you look at the HTTP response headers you
> get back from your [1] and [2] URLs you will see that [2] includes an
> "Accept-Ranges: bytes" header whereas [1] does not.
>
> The netCDF-Java library can open remote datasets available over HTTP,
> without downloading the entire file, if the server supports byte range
> requests for that dataset. [However, depending on the access pattern,
> this can be very inefficient as the client is doing the subsetting which
> may require multiple HTTP requests. For OPeNDAP access, the subsetting
> is done on the server and generally only requires one HTTP request.]
>
> Hope that helps,
>
> Ethan
>
> [a] https://en.wikipedia.org/wiki/Byte_serving
>
> On 6/10/2013 8:04 AM, H. Joe Lee wrote:
> > Hi, John & Ethan!
> >
> > Thanks for the help!
> >
> > Now I figured it out that the "location" tag of NcML does not work
> > with the on-demand NetCDF-3 file that is automatically generated by
> > "fileout_netcdf" module by OPeNDAP server like [1]. If I save the
> > NetCDF-3 file from [1] and put it in a regular HTTP server like [2],
> > the NcML works fine.
> >
> > For comparison, the URL [3] is the DAP response that NCDump in
> > ToolsUI can also recognize. Thus, I could make my NcML work by
> > dropping .nc as Ethan suggested. Since [3] works fine with NcML, I
> > have no reason to save and put it in a separate web server like [2].
> >
> > However, I still don't know why OPeNDP fileout_netcdf module doesn't
> > work with NcML "location". What's the key difference between [1] and
> > [2]? It seems like an OPeNDAP sever configuration issue. I'm asking
> > opendap-tech mailing list to get some help.
> >
> > Regards,
> >
> >
> > [1]
> http://eosdap.hdfgroup.uiuc.edu:8080/opendap/data/hdf4/foo2.hdf.gz.nc
> > [2] http://hdfeos.org/foo2.hdf.gz.nc
> > [3] dods://eosdap.hdfgroup.uiuc.edu:8080/opendap/data/hdf4/foo2.hdf.gz
> >
> > --
> > HDF: Software that Powers Science
>
> ============================
> http://eosdap.hdfgroup.uiuc.edu:8080/opendap/data/hdf4/foo2.hdf.gz.nc
>
> GET /opendap/data/hdf4/foo2.hdf.gz.nc HTTP/1.1
> Host: eosdap.hdfgroup.uiuc.edu:8080
> User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101
> Firefox/21.0
> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> Accept-Language: en-US,en;q=0.5
> Accept-Encoding: gzip, deflate
> DNT: 1
> Connection: keep-alive
>
> HTTP/1.1 200 OK
> Server: Apache-Coyote/1.1
> Last-Modified: Tue, 02 Oct 2012 02:58:53 GMT
> Content-Disposition: attachment; filename="foo2.hdf.gz.nc"
> XDODS-Server: dods/3.2
> XOPeNDAP-Server: bes/3.11.0, libdap/3.11.7, dap-server/ascii/4.1.2,
> fileout_netcdf/1.1.3, hdf4_handler/3.10.1, hdf5_handler/2.1.2,
> netcdf_handler/3.10.2, ncml_module/1.2.2, dap-server/usage/4.1.2,
> dap-server/www/4.1.2
> XDAP: 3.2
> Content-Type: application/x-netcdf
> Transfer-Encoding: chunked
> Date: Mon, 10 Jun 2013 18:55:04 GMT
>
> ============================
> http://hdfeos.org/foo2.hdf.gz.nc
>
> GET /foo2.hdf.gz.nc HTTP/1.1
> Host: hdfeos.org
> User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101
> Firefox/21.0
> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> Accept-Language: en-US,en;q=0.5
> Accept-Encoding: gzip, deflate
> DNT: 1
> Connection: keep-alive
>
> HTTP/1.1 200 OK
> Date: Mon, 10 Jun 2013 18:53:42 GMT
> Server: Apache
> Last-Modified: Mon, 10 Jun 2013 13:56:13 GMT
> Etag: "a20b75-f0-4decd24c9da96"
> Accept-Ranges: bytes
> Content-Length: 240
> Keep-Alive: timeout=2, max=200
> Connection: Keep-Alive
> Content-Type: application/x-netcdf
> Content-Encoding: gzip
>
>
> --
> Ethan Davis UCAR Unidata Program
> edavis@xxxxxxxxxxxxxxxx http://www.unidata.ucar.edu
>
> --
> You received this message because you are subscribed to the Google Groups
> "OPeNDAP Tech" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to opendap-tech+unsubscribe@xxxxxxxxxxx.
> To post to this group, send email to opendap-tech@xxxxxxxxxxx.
> Visit this group at
> http://groups.google.com/a/opendap.org/group/opendap-tech/?hl=en.
> For more options, visit
> https://groups.google.com/a/opendap.org/groups/opt_out.
>
>
>
--
Dr. M. Benno Blumenthal benno@xxxxxxxxxxxxxxxx
International Research Institute for climate and society
The Earth Institute at Columbia University
Lamont Campus, Palisades NY 10964-8000 (845) 680-4450
