[idvusers] [THREDDS #GCY-293173]: THREDDS 4.6 and Mixed Interval Variables

Hi Don,

Apologies for the troubles you are having. Perhaps you have some ideas on how to
make bundles more resilient to changes. Based on the bundle you provided, a 
simple
name change seemed to do the trick. Maybe it's time to rethink the current 
bundle format
and see how these kinds of issues could be addressed by creating a new bundle 
version.
Also remember that data access through a backup server running the previous 
version will 
continue to exists for a few weeks after today using threddsbackup.ucar.edu 
(there is also 
an IDV plugin for this). This should help facilitate any changes that need to 
be made in terms
of getting the details of previous bundle behavior (color scales, ranges, etc.)

Rather than taking this thread in a totally different direction (re: new bundle 
format), perhaps 
this item can be added to the next IDV Steering Committee meeting.

Thank you,

THREDDS Support

> Dear THREDDS Support-
>
> Not all IDV users are on an academic schedule.  I use this bundle continually 
> through the year
> (daily during times of precip) and it is served up by my RAMADDA server to 
> all my users.  
> In this case, I do have to remake the display (unless I edit the bundle) 
> which means I have 
> to redo the formula and set any colortables, ranges, etc that I had in the 
> original bundle (if 
> I can remember what they were).  It's not a trivial issue as you make it seem.
>
>Don
>
> Hi All,
> 
> Don - thank you for your initial report of the issue, and apologies for the 
> delay. Given that
> the academic summer semesters typically start at the beginning of June, there 
> should
> be enough time to address these changes.
> 
> I'm pretty sure the bundle will load what it can, and the new field can be 
> added quite easily
> (which I've done with your RR 20km bundle Don - would you like me to send a 
> copy?). I do
> not believe there is any need to remake a bundle from scratch in this case.
> 
> In terms of this change specifically, if anyone has issues with their bundles 
> not being able
> to be salvaged due to the mixed interval variable name changes, please send a 
> copy of your
> bundle to support@xxxxxxxxxxxxxxxx and we will get it fixed.
> 
> If you're feeling adventurous, you  can use the list of name changes provided 
> below to
> replace the old names in the .xidv  bundle file yourself, since xidv bundles 
> are just text files,
> which is exactly what we will be doing here. Generally it's a bad idea to 
> hand edit bundle
> files as they tend to be very brittle and not at all forgiving when it comes 
> to hand editing,
> but variable name changes are pretty straight forward. You would basically 
> need to change
> occurrences of the string "_Mixed_Intervals" with "_1_Hour" or "_3_Hour", 
> depending on
> the correct name.
> 
> Again, no need that I can see to remake a bundle.
> 
> Cheers,
> 
> THREDDS Support
> 
> >
> > Thank you for finally addressing this issue that was reported several weeks 
> > ago.  I wish
> >  there could have been more notification than on the day of the server 
> > switch so I could
> >  have time to fix things.
> >
> > One point of clarification.  Loading a bundle in IDV 5.1u2 will not prompt 
> > you for a new variable
> >  name. It just gives an error that the variable can't be found and hence no 
> > display is created.
> >  This means that anyone using bundles that have variables with new names 
> > will have to
> >  remake their bundles.
> >
> > Sigh.....
> >
> > Don
> 
> > Greetings IDV users,
> >
> > As thredds.ucar.edu is switched to the 4.6 version of the THREDDS Data 
> > Server (TDS),
> > you may notice the same names of variables have changed. Specifically, you 
> > will likely
> > encounter the case where Mixed Interval variable have been renamed to 
> > indicate that
> > they are no longer mixed, but rather fixed interval variables. As an 
> > example, if we consider
> > the Rapid Refresh Model, the following old variable name:
> >
> > Convective_precipitation_surface_Mixed_intervals_Accumulation
> >
> > has been changed to:
> >
> > Convective_precipitation_surface_1_Hour_Accumulation
> >
> > Why did this happen? The THREDDS group has looked over all cases of Mixed 
> > Interval
> > Variables in the data we serve out and we noticed that, in some cases, 
> > mixed intervals were
> > not needed, and we we able to reduce these to single interval variables.
> >
> > Again, considering the Convective Precip variable from the Rapid Refresh, 
> > the old variable:
> >
> > Convective_precipitation_surface_Mixed_intervals_Accumulation
> >
> > actually contained two types of intervals over which the accumulation 
> > occurred (thus
> > the "mixed interval" in the name). One interval was the 1 hour interval:
> >
> > accumulation over forecast hours 0-1, 1-2, 2-3, 3-4., etc.
> >
> > The second interval contained a total run accumulation:
> >
> > accumulation over forecast hours 0-1, 0-2, 0-3, 0-4, etc.
> >
> > When served as a mixed interval, the data would look something like:
> >
> > accumulation over forecast hours 0-1, 0-1, 1-2, 0-2,  2-3, 0-3...
> >
> > The end result is that, when displayed in the IDV, the precip field would 
> > appear to
> > "jump" around in magnitude when animated. However, given the simplicity of 
> > how
> > these intervals were actually "mixed", we were able to, on the server side, 
> > remove
> > the total accumulation over the run portion of the grids while only serving 
> > the 1 hour
> > accumulation grids. Since the total run accumulation can be computed from 
> > the
> > 1 hour interval accumulations, this seems like the reasonable thing to do.
> >
> > Depending on the model, the mixed interval grid may be rename as a
> > 1_hour_Accumulation, or a 3_hour_Accumulation (other interval lengths
> > will be possible in the future, I'm sure). That means replacing all of the 
> > Mixed
> > Interval Variable names with a 1 hour Accumulation variable name is not a
> > safe option.
> >
> > After much discussion with the IDV group over the past year+, we've agreed 
> > that
> > in these cases (where data fundamentally change meaning), it's best to allow
> > users of the IDV to see that these changes have occurred. We are not big 
> > fans of
> > silently changing the meaning of the variables. Further, we aren't sure how 
> > aware
> > users are of the complexity of these mixed interval data, so this will 
> > bring these issues
> > to the forefront. For users of IDV  bundles, this means you will need to 
> > re-save your
> > bundle after being prompted to select a new variable.
> >
> > We hope you find these reduced mixed interval variables are more useful, and
> > please know that we are working on a solution for the more complex types of
> > mixed intervals (for example, every mixed interval in the GFS output). 
> > Because
> > these more complex mixed interval variables still show up as Mixed Interval
> > variables (and are very nuanced), be very cautious when using them as they 
> > are
> > likely NOT what you think or hope they might be.
> >
> > In the coming weeks we will be doing a series of blog posts explaining in 
> > detail
> > how these mixed interval grids come about, and why there are cases where
> > the reduction of a mixed interval variable is not as simple as the case 
> > above.
> >
> > Here is a list of the datasets, and associated variables, where these 
> > changes occur:
> >
> > Rapid Refresh collections (for old name, replace 1_Hour with 
> > Mixed_intervals):
> >
> > Convective_precipitation_surface_1_Hour_Accumulation
> > Large-scale_precipitation_non-convective_surface_1_Hour_Accumulation
> >
> > NAM 12km NOAAPORT (for old name, replace 3_Hour with Mixed_intervals):
> >
> > Convective_precipitation_surface_3_Hour_Accumulation
> > Percent_of_Frozen_Precipitation_surface_3_Hour_Accumulation
> > Probability_of_0p01_inch_of_precipitation_POP_surface_3_Hour_Accumulation
> > Probability_of_Freezing_Precipitation_surface_3_Hour_Accumulation
> > Thunderstorm_probability_surface_3_Hour_Accumulation
> > Total_precipitation_surface_3_Hour_Accumulation
> >
> > NAM 12km CONDUIT  (for old name, replace 3_Hour with Mixed_intervals):
> >
> > Baseflow-Groundwater_Runoff_surface_3_Hour_Accumulation
> > Convective_precipitation_surface_3_Hour_Accumulation
> > Deep_Convective_Heating_Rate_entire_atmosphere_single_layer_3_Hour_Average
> > Downward_Short-Wave_Radiation_Flux_surface_3_Hour_Average
> > Large_Scale_Condensate_Heating_Rate_entire_atmosphere_single_layer_3_Hour_Average
> > Snow_melt_surface_3_Hour_Accumulation
> > Storm_Surface_Runoff_surface_3_Hour_Accumulation
> > Total_precipitation_surface_3_Hour_Accumulation
> > Water_equivalent_of_accumulated_snow_depth_surface_3_Hour_Accumulation
> >
> > NAM 20km NOAAPORT (for old name, replace 3_Hour with Mixed_intervals):
> >
> > Convective_precipitation_surface_3_Hour_Accumulation
> > Percent_of_frozen_precipitation_surface_3_Hour_Accumulation
> > Probability_of_freezing_precipitation_surface_3_Hour_Accumulation
> > Probability_of_precipitation_surface_3_Hour_Accumulation
> > Snow_depth_surface_3_Hour_Accumulation
> > Thunderstorm_probability_surface_3_Hour_Accumulation
> > Total_precipitation_surface_3_Hour_Accumulation
> >
> > NAM 40km CONDUIT  (for old name, replace 3_Hour with Mixed_intervals):
> >
> > Baseflow-Groundwater_Runoff_surface_3_Hour_Accumulation
> > Convective_precipitation_surface_3_Hour_Accumulation
> > Downward_Long-Wave_Radp_Flux_surface_3_Hour_Average
> > Downward_Short-Wave_Radiation_Flux_surface_3_Hour_Average
> > Evaporation_surface_3_Hour_Accumulation
> > Ground_Heat_Flux_surface_3_Hour_Average
> > Large-scale_precipitation_non-convective_surface_3_Hour_Accumulation
> > Latent_heat_net_flux_surface_3_Hour_Average
> > Potential_Evaporation_surface_3_Hour_Accumulation
> > Sensible_heat_net_flux_surface_3_Hour_Average
> > Snow_Phase_Change_Heat_Flux_surface_3_Hour_Average
> > Snow_melt_surface_3_Hour_Accumulation
> > Storm_Surface_Runoff_surface_3_Hour_Accumulation
> > Total_cloud_cover_entire_atmosphere_single_layer_3_Hour_Average
> > Total_precipitation_surface_3_Hour_Accumulation
> > Upward_Long-Wave_Radp_Flux_atmosphere_top_3_Hour_Average
> > Upward_Long-Wave_Radp_Flux_surface_3_Hour_Average
> > Upward_Short-Wave_Radiation_Flux_atmosphere_top_3_Hour_Average
> > Upward_Short-Wave_Radiation_Flux_surface_3_Hour_Average
> > Water_equivalent_of_accumulated_snow_depth_surface_3_Hour_Accumulation
> >
> > Thank you,
> >
> > The THREDDS Development Team
> >

Ticket Details
===================
Ticket ID: GCY-293173
Department: Support THREDDS
Priority: Low
Status: Open



  • 2015 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the idvusers archives: