Can you do
for ( String arg : args ) {
try (NetcdfFile ncfile = NetcdfFile.open( arg )) {
// do all processing for this ncfile
....
}
}
Note that this is a "try with resource" that will close the ncfile
automatically at the end of the block.
Much preferred over manually closing, because it is guaranteed to close
(and free the memory) even when an exception is thrown.
On Wed, Oct 23, 2019 at 11:14 AM Jesse Bickel - NOAA Affiliate <
jesse.bickel@xxxxxxxx> wrote:
> Hi John,
>
> I did not know about the NcML Aggregation capability in the Java cdm
> library, thank you for pointing this out. So no, it was simply opening
> all the netcdf resources using a for-each loop over a Set<URI> using
> NetcdfFile.open( uri.toString() ).
>
> Yes, both HTTP servers support range requests and I can observe range
> requests and responses using Java network debug settings. One example
> visible to you would be the medium range members at
> https://nomads.ncep.noaa.gov/pub/data/nccf/com/nwm/prod/ (I would give
> more exact links, but the first subdirectory changes day-to-day).
>
> Below is an example to reproduce the issue. It requires either passing
> a list of resources that are larger than the default maximum heap on
> ones machine or setting the maximum default heap to be smaller than
> the size of the resources in bytes times ten million or the size of
> the resources themselves (whichever is smaller). I also attempted to
> attach the complete example in compressed form, with jars and all, but
> that was blocked. One can reconstruct it by looking at the commands
> below and finding the appropriate jars.
>
> Jesse
>
> --
> Contractor, ERT, Inc.
> Federal Affiliation: NWC/OWP/NOAA/DOC
>
>
> package com.example;
>
> import java.io.IOException;
> import java.util.Set;
> import java.util.HashSet;
>
> import org.slf4j.Logger;
> import org.slf4j.LoggerFactory;
> import ucar.nc2.NetcdfFile;
>
> public class CdmHttpResourceOomeExample
> {
> private static final Logger LOGGER = LoggerFactory.getLogger(
> CdmHttpResourceOomeExample.class );
>
> /**
> * Opens each argument using NetcdfFile.open(), keeps all of them
> open, then
> * after opening them all, closes them all.
> *
> * @param args An array of URLs to open as netCDF resources.
> * @throws IOException When opening an arg as a netCDF resource fails.
> * @throws IllegalArgumentException When no arg is specified.
> */
>
> public static void main( String[] args ) throws IOException
> {
> if ( args == null || args.length < 1 )
> {
> throw new IllegalArgumentException( "Please specify a URL." );
> }
>
> Set<NetcdfFile> ncResources = new HashSet<>();
>
> try
> {
> for ( String arg : args )
> {
> NetcdfFile ncResource = NetcdfFile.open( arg );
> ncResources.add( ncResource );
> }
> }
> finally
> {
> for ( NetcdfFile ncResource : ncResources )
> {
> try
> {
> ncResource.close();
> }
> catch ( IOException ioe )
> {
> LOGGER.warn( "Failed to close {}", ncResource );
> }
> }
> }
> }
> }
>
>
>
> To compile on Windows:
> %JAVA_HOME%\bin\javac -cp cdm-5.1.0.jar;slf4j-api-1.8.0-beta4.jar;.
> com\example\CdmHttpResourceOomeExample.java
>
> To run simplest example on Windows:
> %JAVA_HOME%\bin\java -Xms10m -Xmx10m -cp
>
> cdm-5.1.0.jar;slf4j-api-1.8.0-beta4.jar;logback-core-1.3.0-alpha4.jar;logback-classic-1.3.0-alpha4.jar;httpservices-5.1.0.jar;httpcore-4.4.12.jar;httpclient-4.5.10.jar;httpmime-4.5.10;jcl-over-slf4j-1.8.0-beta4.jar;re2j-1.3.jar;.
> com.example.CdmHttpResourceOomeExample
> https://any/nc/resource/bigger/than/ten/million/bytes.nc
>
