Re: [netcdf-java] Aggregating NetCDF files programmatically

On 12/11/2013 5:23 AM, Guy Griffiths wrote:
Hi all,
>
> I am looking into ways of aggregating NetCDF files programmatically
> for a new version of ncWMS.  The aim is to allow expressions such as
> "*.nc" to result in a single dataset, so long as all NetCDF files
> share the same variables/dimensions and differ only in the values on
> their time axes.
>
> I can do this by writing an NcML file, but this is a bit of a hack,
> and leaves files around.  We previously did this by manually keeping
> track of which time steps appeared in which files, but this doesn't
> easily fit with our new data model, and seems unnecessary when there
> are several Aggregation objects in the NetCDF-Java libraries which
> should be able to do the job for us.
>
> Searching lead me to this message on the list:
>
> http://www.unidata.ucar.edu/mailing_lists/archives/netcdf-java/2013/msg00029.html
>
>
>
but unfortunately there were no responses. I was able to use the AggregationUtil class from that post to read the correct metadata, but like the person who posted I run into trouble when trying to actually read the data. I get a slightly different stack trace, which starts:

> java.io.IOException: BAD: missing spi: analysed_sst at
> ucar.nc2.NetcdfFile.readData(NetcdfFile.java:1753) at
> ucar.nc2.Variable.reallyRead(Variable.java:850) ...
>
> and is then the same as in the other post.
>
> I would be very grateful if anyone could suggest a way of making
> aggregation work programatically, either with an adaptation of the
> above method or something completely new.
>
> Best Regards,
>
> Guy Griffiths

Hi Guy:

One workaround would be to construct the NcML as a string in memory and pass it to NcMLReader.readNcML () with a ByteArrayInputStream:

   // read ncml through a InputStream.
    String ncml =
"<netcdf xmlns=\"http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2\";>\n" +
        "<aggregation type=\"union\">\n" +
" <netcdf location=\"file:G:/work/jasmin/SVM01_npp_d20120120_t0531354_e0532596_b01189_c20120120120033631216_noaa_ops.h5\" />\n" + " <netcdf location=\"file:G:/work/jasmin/GMTCO_npp_d20120120_t0531354_e0532596_b01189_c20120120115420527613_noaa_ops.h5\" />\n" +
        "  </aggregation>\n" +
        "</netcdf>";

NetcdfDataset aggregatedDataset = NcMLReader.readNcML( new ByteArrayInputStream(ncml.getBytes()), null );




  /**
   * Read NcML doc from an InputStream, and construct a NetcdfDataset.
   *
   * @param ins        the InputStream containing the NcML document
   * @param cancelTask allow user to cancel the task; may be null
   * @return the resulting NetcdfDataset
   * @throws IOException on read error, or bad referencedDatasetUri URI
   */
static public NetcdfDataset readNcML(InputStream ins, CancelTask cancelTask) throws IOException

Also could construct a JDOM tree yourself and use:

  /**
   * Read NcML from a JDOM Document, and construct a NetcdfDataset.
   *
* @param ncmlLocation the URL location string of the NcML document, used to resolve reletive path of the referenced dataset,
   *                     or may be just a unique name for caching purposes.
   * @param netcdfElem   the JDOM Document's root (netcdf) element
   * @param cancelTask   allow user to cancel the task; may be null
   * @return the resulting NetcdfDataset
   * @throws IOException on read error, or bad referencedDatasetUri URI
   */
static public NetcdfDataset readNcML(String ncmlLocation, Element netcdfElem, CancelTask cancelTask) throws IOException {


John

  • 2013 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: