Re: [thredds] Best way to handle missing grids in an aggregation.

• To: John Caron <caron@xxxxxxxxxxxxxxxx>
• Subject: Re: [thredds] Best way to handle missing grids in an aggregation.
• From: Ken Tanaka <Ken.Tanaka@xxxxxxxx>
• Date: Fri, 16 Oct 2009 11:29:50 -0600

Hi John,

I inserted some responsed below.

John Caron wrote:

Hi Ken:

Ken Tanaka wrote:

Hi,

I'm aggregating some gridded data with an ncml entry in our THREDDS catalog.xml file. The entry looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<catalog name="NGDC THREDDS Server Default Catalog"

xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0";

       xmlns:xlink="http://www.w3.org/1999/xlink";
       version="1.0.2">

 <service name="aggServices" serviceType="compound" base="">

<service name="dapService2" serviceType="OpenDAP" base="/thredds/dodsC/" />

   <service name="wcsService2" serviceType="WCS" base="/thredds/wcs/" />
   <service name="wms2" serviceType="WMS" base="/thredds/wms/" />
 </service>

 <dataset name="USTEC STEC Aggregation" ID="STEC-Agg"
          urlPath="stec-aggregation">
   <metadata inherited="true">
     <serviceName>aggServices</serviceName>
     <dataType>Grid</dataType>
   </metadata>

<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>

     <attribute name="Comment" value="Aggregation for USTEC STEC" />

<aggregation dimName="Time" type="joinExisting" recheckEvery="15 min">

       <scan location="/nfs/ustec_netcdf/stec/latest7days"
             suffix=".nc" />
     </aggregation>
   </netcdf>

 </dataset>
</catalog>

Inside the netCDF files are grids from satellites:
time 1:  satellite_01, satellite_02, satellite_03
time 2:  satellite_01, satellite_02, satellite_03
time 3:  satellite_01, satellite_03
time 4:  satellite_03, satellite_02, satellite_01
and so on.

Sometimes data from a satellite is not present, as satellite_02 is missing at time 3 at in the example above. This appears as an all white image in the time sequence when viewing the satellite_02 result with IDV. What is the best way to handle this. Is there a missing grid flag, or should there be an empty image inserted, so that there is always a satellite_02 grid?

Can you try setting timeUnitsChange = "true" on the aggregation element and see if the problem persists? thanks.

I made this change, so the new line looks like this:

We are also getting an error from IDV

java.io.IOException: Inconsistent array length read: 1165128303 != 1914731274

  at ucar.nc2.dt.grid.GeoGrid.readDataSlice(GeoGrid.java:616)
  at ucar.nc2.dt.grid.GeoGrid.readDataSlice(GeoGrid.java:524)
  at ucar.nc2.dt.grid.GeoGrid.readVolumeData(GeoGrid.java:445)

at ucar.visad.data.GeoGridFlatField.readData(GeoGridFlatField.java:220) at ucar.visad.data.CachedFlatField.getMyValues(CachedFlatField.java:479) at ucar.visad.data.CachedFlatField.getMyValues(CachedFlatField.java:467) at ucar.visad.data.CachedFlatField.unpackFloats(CachedFlatField.java:614)

  at visad.FlatField.getFloats(FlatField.java:1567)

at visad.ShadowFunctionOrSetType.doTransform(ShadowFunctionOrSetType.java:1404) at visad.java3d.ShadowFunctionOrSetTypeJ3D.doTransform(ShadowFunctionOrSetTypeJ3D.java:195) at visad.java3d.ShadowFunctionOrSetTypeJ3D.recurseRange(ShadowFunctionOrSetTypeJ3D.java:1209) at visad.java3d.ShadowFunctionOrSetTypeJ3D$1.run(ShadowFunctionOrSetTypeJ3D.java:181)

  at visad.util.ThreadManager$1.run(ThreadManager.java:292)
  at visad.util.ThreadPool$ThreadMinnow.run(ThreadPool.java:98)

this is the error on the client, the only thing it tells you is that there was an error on the server. Go to the sever logs and see what error is there.

There are errors in threddsServlet.log like this one:
=================================

java.lang.NullPointerException

   at ucar.nc2.dataset.VariableDS._read(VariableDS.java:498)
   at ucar.nc2.Variable.read(Variable.java:778)
   at ucar.nc2.Variable.read(Variable.java:752)
   at thredds.server.opendap.NcSDArray.read(NcSDArray.java:114)
   at thredds.server.opendap.NcSDGrid.read(NcSDGrid.java:72)
   at opendap.dap.Server.SDGrid.serialize(SDGrid.java:516)
   at opendap.dap.Server.CEEvaluator.send(CEEvaluator.java:275)

   at thredds.server.opendap.OpendapServlet.doGet(OpendapServlet.java:220)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)

   at thredds.servlet.filter.CookieFilter.doFilter(CookieFilter.java:54)

   at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
   at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
   at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)

   at java.lang.Thread.run(Thread.java:619)

=================================

Do you think this error is due to the missing satellite_02 grids?

One more question: Also, as can be seen at time 4, the satellite grids are not always in the same order in the netCDF file. Is this a problem? I can have my colleague regenerate the data with grids in a consistent order if needed.

Are the satellite_03, satellite_02, satellite_01 variable name? if so, order doesnt matter.

If I have answers to the above, I can probably figure out the following on my own. If all satellites must be present, maybe we need to start a new aggregation if a new satellite is added to the set or one is retired from service? Maybe we should be splitting out each satellite into their own netCDF files?

Thanks,
Ken

_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx

For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/

--
= Enterprise Data Services Division ===============
| CIRES, National Geophysical Data Center / NOAA  |
| 303-497-6221                                    |
= Ken.Tanaka@xxxxxxxx =============================

• References:
  • [thredds] Best way to handle missing grids in an aggregation.
    • From: Ken Tanaka
  • Re: [thredds] Best way to handle missing grids in an aggregation.
    • From: John Caron