Hi Jesse:
Are you doing an NcML Aggregation?
Do you know if your HTTP server (where the files live) supports range
requests?
Can you send something to reproduce?
John
On Thu, Oct 17, 2019 at 4:39 PM Jesse Bickel - NOAA Affiliate via
netcdf-java <netcdf-java@xxxxxxxxxxxxxxxx> wrote:
> Hello,
>
> A nice feature of the java netcdf library is it allows us to use a netcdf
> resource in the same way regardless of locality. Our code can use the same
> library calls to open a netcdf resource whether it is on a local filesystem
> or on a web server.
>
> The java netcdf library makes use of the HTTP Range header in the
> HTTPRandomAccessFile class. This means the whole netcf resource does not
> need to be read or downloaded prior to use. It seems the netcdf library
> handles the details quite well, requesting byte ranges similar to the way
> it would if the resource were on a local filesystem.
>
> One downside of the approach is the amount of heap memory allocated by
> default for each netcdf resource, especially in the case of forecasts
> spanning multiple netcdf resources greater than ten million bytes each.
>
> When attempting to open a single ensemble forecast composed of multiple
> netcdf resources (in this case seven members times sixty-eight timesteps)
> prior to reading values from them, an OutOfMemoryError is encountered with
> the following stack trace:
>
> java.lang.OutOfMemoryError: Java heap space
> at ucar.unidata.io
> .RandomAccessFile.init(RandomAccessFile.java:376)
> at ucar.unidata.io
> .RandomAccessFile.setBufferSize(RandomAccessFile.java:387)
> at
> ucar.unidata.io.http.HTTPRandomAccessFile.<init>(HTTPRandomAccessFile.java:98)
> at
> ucar.unidata.io.http.HTTPRandomAccessFile.<init>(HTTPRandomAccessFile.java:40)
> at ucar.nc2.NetcdfFile.getRaf(NetcdfFile.java:615)
> at ucar.nc2.NetcdfFile.open(NetcdfFile.java:506)
> at ucar.nc2.NetcdfFile.open(NetcdfFile.java:473)
> at ucar.nc2.NetcdfFile.open(NetcdfFile.java:458)
> at ucar.nc2.NetcdfFile.open(NetcdfFile.java:446)
>
> Two possibilities come to mind as to workarounds. First, allocate a larger
> heap. How much larger? Perhaps ten million bytes times seven times
> sixty-eight, around 4.5GiB more. But it does not seem right to require an
> additional 4.5GiB heap to simply open several resources and suppose the
> user is on a 32-bit system. Second, perhaps we could find a way to
> progressively open, read, and close each resource. This might be possible,
> but seems clunky and incorrect. We should be able to open all the
> resources, then access what is needed across them, then close them. In this
> case, a forecast spans multiple resources and the goal is to read a single
> forecast.
>
> A third option is to consider code in HTTPRandomAccessFile near the
> OutOfMemoryError and the variables involved.
>
> public class HTTPRandomAccessFile extends ucar.unidata.io.RandomAccessFile
> {
> public static final int defaultHTTPBufferSize = 20 * 1000; // 20K
> public static final int maxHTTPBufferSize = 10 * 1000 * 1000; // 10 M
>
> ...
>
> if (total_length > 0) {
> // this means that we will read the file in one gulp then deal with
> it in memory
> int useBuffer = (int) Math.min(total_length, maxHTTPBufferSize); //
> entire file size if possible
> useBuffer = Math.max(useBuffer, defaultHTTPBufferSize); // minimum
> buffer
> setBufferSize(useBuffer);
> }
>
> The effect of the Math.min and Math.max calls appears to cause a buffer
> size of ten million bytes to be allocated for each netcdf resource greater
> than or equal to ten million bytes.
>
> Experimentation shows that there are more requests made when this
> maxHTTPBufferSize is reduced, but the OutOfMemoryError is avoided.
>
> The version control history shows it used to only use the twenty thousand
> value, not ten million.
>
> Is there any significance to the ten million byte buffer size?
>
> Would you be willing to make a new default of two hundred thousand or
> perhaps offer a Java System Property option to configure the value at JVM
> launch time? For example, -Ducar.unidata.io.http.maxHTTPBufferSize=200000.
> It is preferable to use a build tool to fetch the ucar-built and
> ucar-tested cdm artifact to get the latest and greatest updates rather than
> maintain a fork.
>
> I have not experimented with the setting with any datasets other than this
> narrow use case, so I also wonder about the impact on other uses. All I can
> see from experimentation is that a trade-off is made between
> request/response overhead on the one hand (higher when set lower) and data
> volume on the other hand (higher when set higher).
>
> The trace above is with cdm-5.1.0.jar, from the ucar artifact repository
> (fetched with gradle), with sha256sum
> d211d2b040aa1d63bc3a6898bb27f55fb116f743dee4572b7f1228e9d4cf37f1.
>
> Thank you for your consideration,
>
> Jesse Bickel
>
> Contractor, ERT, Inc.
> Federal Affiliation: NWC/OWP/NOAA/DOC
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> netcdf-java mailing list
> netcdf-java@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
>
