Re: [thredds] Advantages of using FMRC over JoinExisting

• To: thredds@xxxxxxxxxxxxxxxx
• Subject: Re: [thredds] Advantages of using FMRC over JoinExisting
• From: John Caron <caron@xxxxxxxxxxxxxxxx>
• Date: Fri, 07 Dec 2012 20:12:42 -0700

hi rich:

what is varying across the files? times ?

heres a dump of one of your files:

netcdf G:/work/signell/ugrid/ugrid01.nc {
 dimensions:
   node = 48451;
   three = 3;
   nele = 90415;
   time = 3;
 variables:
   int fvcom_mesh;
     :standard_name = "mesh_topology";
     :dimension = 2; // int
     :node_coordinates = "lon lat";
     :face_coordinates = "lonc latc";
     :face_node_connectivity = "nv";
   float lat(node=48451);
     :long_name = "nodal latitude";
     :standard_name = "latitude";
     :units = "degrees_north";
   float lon(node=48451);
     :long_name = "nodal longitude";
     :standard_name = "longitude";
     :units = "degrees_east";
   int nv(three=3, nele=90415);
     :long_name = "nodes surrounding element";
     :standard_name = "face_node_connectivity";
     :start_index = 1; // int
   float time(time=3);
     :long_name = "time";
     :units = "days since 1858-11-17 00:00:00";
     :format = "modified julian day (MJD)";
     :time_zone = "UTC";
   float zeta(time=3, node=48451);
     :long_name = "Water Surface Elevation";
     :units = "meters";
     :standard_name = "sea_surface_height_above_geoid";
     :coordinates = "time lat lon";
     :type = "data";
     :mesh = "fvcom_mesh";
     :location = "node";

 :NCO = "20121207";
 :title = "GOM3 Nesting by yf.sun@xxxxxxxxxx";
 :institution = "School for Marine Science and Technology";
 :source = "FVCOM_3.0";

 :Conventions = "CF-1.0";
 :CoordinateSystem = "Cartesian";
 :CoordinateProjection = "none";
 :Tidal_Forcing = "TIDAL ELEVATION FORCING IS OFF!";

 :GroundWater_Forcing = "GROUND WATER FORCING IS OFF!";

}


On 12/6/2012 4:00 PM, Signell, Richard wrote:

Thredds folks,

  I have UGRID (unstructured grid) files I'm aggregating, so FMRC is out.

Plan B:  Is there a way to use NcML to trick joinExisting aggregation
to give me an unlimited dimension for time?

I found this page:

http://www.unidata.ucar.edu/software/netcdf-java/CDM/Opendap.html

where it sounds like adding a global attribute via NcML
      <attribute name="DODS_EXTRA/Unlimited_Dimension" value="time"/>
might have worked.  But I tried it, and it doesn't.

Does anyone have another idea?

Thanks,
-Rich

On Tue, Dec 4, 2012 at 6:22 PM, Signell, Richard <rsignell@xxxxxxxx> wrote:

Thredds folks,

We've got 33 years of ocean model output  in monthly NetCDF3 files
that we have been aggregating into a single virtual 33TB dataset using
joinExisting aggregation along the time dimension:
http://www.smast.umassd.edu:8080/thredds/hindcasts.html?dataset=fvcom/hindcasts/30yr_gom3

This works fine, but two issues have arisen:

1) the monthly files start at 0000 on the first day of the month, and
end on 2400 of the last day of the month.  Thus when aggregated with
joinExisting, we get duplicate time and data values at the end of
every month.

2) when accessing the data via OPeNDAP, the dataset doesn't have an
unlimited time dimension.   This causes the "nc3tonc4" conversion
program in NetCDF4-Python to try to read each variable in it's
entirety, which of course is problematic.  If "nc3tonc4" sees an
unlimited dimension, it can "chunk" its way along.

So we could eliminate both these problems by using a FMRC aggregation, right?

Thanks,
Rich

--
Dr. Richard P. Signell   (508) 457-2229
USGS, 384 Woods Hole Rd.
Woods Hole, MA 02543-1598

• References:
  • [thredds] Advantages of using FMRC over JoinExisting
    • From: Signell, Richard
  • Re: [thredds] Advantages of using FMRC over JoinExisting
    • From: Signell, Richard