Hi John,
I did not know about the NcML Aggregation capability in the Java cdm
library, thank you for pointing this out. So no, it was simply opening
all the netcdf resources using a for-each loop over a Set<URI> using
NetcdfFile.open( uri.toString() ).
Yes, both HTTP servers support range requests and I can observe range
requests and responses using Java network debug settings. One example
visible to you would be the medium range members at
https://nomads.ncep.noaa.gov/pub/data/nccf/com/nwm/prod/ (I would give
more exact links, but the first subdirectory changes day-to-day).
Below is an example to reproduce the issue. It requires either passing
a list of resources that are larger than the default maximum heap on
ones machine or setting the maximum default heap to be smaller than
the size of the resources in bytes times ten million or the size of
the resources themselves (whichever is smaller). I also attempted to
attach the complete example in compressed form, with jars and all, but
that was blocked. One can reconstruct it by looking at the commands
below and finding the appropriate jars.
Jesse
--
Contractor, ERT, Inc.
Federal Affiliation: NWC/OWP/NOAA/DOC
package com.example;
import java.io.IOException;
import java.util.Set;
import java.util.HashSet;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import ucar.nc2.NetcdfFile;
public class CdmHttpResourceOomeExample
{
private static final Logger LOGGER = LoggerFactory.getLogger(
CdmHttpResourceOomeExample.class );
/**
* Opens each argument using NetcdfFile.open(), keeps all of them open, then
* after opening them all, closes them all.
*
* @param args An array of URLs to open as netCDF resources.
* @throws IOException When opening an arg as a netCDF resource fails.
* @throws IllegalArgumentException When no arg is specified.
*/
public static void main( String[] args ) throws IOException
{
if ( args == null || args.length < 1 )
{
throw new IllegalArgumentException( "Please specify a URL." );
}
Set<NetcdfFile> ncResources = new HashSet<>();
try
{
for ( String arg : args )
{
NetcdfFile ncResource = NetcdfFile.open( arg );
ncResources.add( ncResource );
}
}
finally
{
for ( NetcdfFile ncResource : ncResources )
{
try
{
ncResource.close();
}
catch ( IOException ioe )
{
LOGGER.warn( "Failed to close {}", ncResource );
}
}
}
}
}
To compile on Windows:
%JAVA_HOME%\bin\javac -cp cdm-5.1.0.jar;slf4j-api-1.8.0-beta4.jar;.
com\example\CdmHttpResourceOomeExample.java
To run simplest example on Windows:
%JAVA_HOME%\bin\java -Xms10m -Xmx10m -cp
cdm-5.1.0.jar;slf4j-api-1.8.0-beta4.jar;logback-core-1.3.0-alpha4.jar;logback-classic-1.3.0-alpha4.jar;httpservices-5.1.0.jar;httpcore-4.4.12.jar;httpclient-4.5.10.jar;httpmime-4.5.10;jcl-over-slf4j-1.8.0-beta4.jar;re2j-1.3.jar;.
com.example.CdmHttpResourceOomeExample
https://any/nc/resource/bigger/than/ten/million/bytes.nc
