Re: [thredds] NetCDF, WMS, and auto color scale (not set by admin)

Hi Jon

 

We updated one of our archives to have - actual_min/actual_max recorded in
each layer in each file.

 

We are now in a good position to progress the work. How do we go about this?
Does this fall into core development or is this something we need find
resources to write ourselves?

 

I found another back reference to 'auto color scaling' - all be it not quite
the same.

http://www.unidata.ucar.edu/mailing_lists/archives/thredds/2010/msg00295.htm
l

 

 

 

The other one we want to get going is the "out of range" customisation for
the WMS. There are a few posts on this - by us and others - and this would
be a great config option to have in the wms config file. 

Something like being able to set <outOfRange>transparent</outOfRange>
instead of it going black or white etc.

 

 

Regards

Terry

 

From: thredds-bounces@xxxxxxxxxxxxxxxx
[mailto:thredds-bounces@xxxxxxxxxxxxxxxx] On Behalf Of
Terry.Rankine@xxxxxxxx
Sent: Saturday, 23 August 2014 12:33 PM
To: j.d.blower@xxxxxxxxxxxxx
Cc: thredds@xxxxxxxxxxxxxxxx
Subject: [ExternalEmail] Re: [thredds] NetCDF, WMS, and auto color scale
(not set by admin)

 

Our grids are relatively small, 300gig is normal or high res scans are about
1T compressed.

I would think that the actual value scanner would be too slow as it would be
scanning millions of values. Our Python cleanup/inspection code takes over
30+ seconds for the large sets. 

I thing actuals would be great, but since i have no simple performance
measure (max file size? Max points?) to protect it, the usecase for me seems
"too dangerous" to use without protection. (Ie a user dumps a high res file 
In a location set up for the $actual scanner.)

What about having an "approx" scanner, some special 1000 point stride of the
data or something like that. In some vis packages they have an approx
scanner.
$quickApprox(1000), or a function called quickandwrong sounds like the right
name....


So:
* can you protect the $actual function
* can you have an approximate function.

Approx function would actually be a pretty good systemwide default.

As for the data providers wanting to tell me what they want the vis range to
be, 2/7 currently think that way, the others think I should know the range
from the data....

I had the chat about vis range to 1 from each opinion. Both think it would
be agreed they could add the extra variables, but some form of automation
would be better than -50 50. so for me it would be set and forget for either
'use their values' or roughly guessing by scanning the data. Both thought
that some stats in the header would be a good thing, and then argured about
the value if a histogram encoded as data variable ... 


Regards
Terry.

 

  _____  

From: Jon Blower
Sent: Friday, 22 August 2014 6:41:03 AM
To: Rankine, Terry (Mineral Resources, Kensington)
Cc:  <mailto:thredds@xxxxxxxxxxxxxxxx> thredds@xxxxxxxxxxxxxxxx
Subject: Re: NetCDF, WMS, and auto color scale (not set by admin)

P.S. I forgot to ask - what was your objection to using the actual range of
values within the variable as the default? I'm assuming your data are 2D
fields, hence reasonably small. It would be feasible to implement a scheme
like this:

<variable id="elevation">
  <defaultColorScaleRange>$actual</defaultColorScaleRange>
</variable>

Then THREDDS, on first reading the variable, would find the range of values
in the variable and use this as the scale range. It could use an actual_min
attribute, but I'm suggesting that it could just calculate this itself by
reading the whole variable array. It could then cache the value range for
future requests.

The advantage would be that the data provider doesn't need to do anything.
The disadvantage would be that performance would be affected, for the first
request at least. Performance would be particularly bad for large
aggregations, but this is not your use case. Sysadmins wouldn't need to
enable this feature for such cases.

This could be made more sophisticated for Mike's suggestion:

<defaultColorScaleRange>$percentile($actual, 2.75)</defaultColorScaleRange>

or something like that.


So I guess the question is - will your data providers need a default scale
range that is *not* based somehow on the actual range of values in the data
file?

Cheers,
Jon

-
Dr Jon Blower,
Technical Director, Reading e-Science Centre,
MELODIES project coordinator,
School of Mathematical and Physical Sciences,
University of Reading
Tel: +44 118 378 5213
Email:  <mailto:j.d.blower@xxxxxxxxxxxxx> j.d.blower@xxxxxxxxxxxxx

On 22 Aug 2014, at 07:31, Jon Blower < <mailto:j.d.blower@xxxxxxxxxxxxx>
j.d.blower@xxxxxxxxxxxxx> wrote:

> Hi Terry,
> 
> I think that your suggestion is implementable, I'm just trying to work
through the potential issues so that a means can be found to work for the
maximum number of users without causing surprising behaviour or the need for
special-case coding.
> 
> I think the remaining issues would be:
> 
> 1. Choice of the attribute names for the suggested scale range. Does
anyone else add this information to NetCDF files and are there any
commonly-used attribute names? I'll ask the CF community this question.
> 
> 2. Behaviour with packed variables. I would assume that the most useful
thing would be for the view range to be encoded with respect to unpacked
variable values. (Valid_min/max can be encoded with respect to either packed
or unpacked values.)
> 
> So I think we should probably do this, but it would be useful to have some
input from other users as they may raise things that we haven't thought of.
> 
> Cheers,
> Jon
> 
> --
> Dr Jon Blower,
> Technical Director, Reading e-Science Centre,
> MELODIES project coordinator,
> School of Mathematical and Physical Sciences,
> University of Reading
> Tel: +44 118 378 5213
> Email:  <mailto:j.d.blower@xxxxxxxxxxxxx> j.d.blower@xxxxxxxxxxxxx
> 
> On 22 Aug 2014, at 03:29,  <mailto:Terry.Rankine@xxxxxxxx>
Terry.Rankine@xxxxxxxx wrote:
> 
>> No we don't use aggregations.
>> 
>> I am an Australian, and all I really want: is to do less work! 
>> The ability for a server to do some of that work for me, is awesome! Your
>> thoughts and help is appreciated.
>> 
>> I am not suggesting that we change THREDDS defaults, I am suggesting that
we
>> 
>> have the option to allow an admin to delegate a default/preference to the
>> data 
>> owner.
>> 
>> The 'pseudo code' would allow arbitrary delegation using a selector to
pull 
>> the data out of the netcdf header.
>>>     <default color range>
>>>        <min>$delegate[elevation.someVar]</min>
>>>        <max>$delegate[elevation.otherVar]</max>
>>>     </default color range>
>> As an admin of multiple mixed systems, I would be free-ed, where
>> appropriate, 
>> to allow the 'view range' to come from a dataset. 
>> 
>> Just to give you an example, we sent a boat out to do some bathymetry
scans,
>> 
>> they dumped 3 netcdf's in the 'please host me folder'.
>> Three separate runs were done, our variable is 'elevation' in this
context.
>> The near shore one was (-200 -20), and the two offshore where
(~-6500,-50),
>> (-2000, -50).
>> 
>> Are you suggesting that:
>> * Because others use aggregations - you won't consider the ability
>> delegate and to read from the file? 
>> * Or are you saying that actual_min/max sound good - lets hard code them
>> in?
>> 
>> I think neither of those are wonderful conclusions. I see value in the
>> delegation pattern, with a 'configured' selector, today they can be read
>> from a header (simple), tomorrow they could be an extra function like
Mike
>> McCann < <mailto:mccann@xxxxxxxxx> mccann@xxxxxxxxx> uses manually now:
>> So the default min (max would be the same) could end up looking like:
>> * $select[file.layer.somvar]
>> * $calculate[$math.percentile(file.layer, 2.75)]
>> With the obvious wording - you can make assumptions on 'performance'
>> 
>> Once again - "I am just lazy" and looking for a better integration
between
>> data-owner, and administrator.
>> 
>> So for now - I will look at capturing the min/max (and hacking in a
>> percentile variable too - good idea Mike), and educating my data
providers
>> to do the same from now on.
>> What Thredds does with the data - well that's now up to you guys!
>> 
>> Regards
>> Terry
>> "That lazy architect guy"
>> 
>> 
>> -----Original Message-----
>> From: Jon Blower [ <mailto:j.d.blower@xxxxxxxxxxxxx>
mailto:j.d.blower@xxxxxxxxxxxxx]
>> Sent: Thursday, 21 August 2014 4:37 PM
>> To: Rankine, Terry (Mineral Resources, Kensington)
>> Cc:  <mailto:thredds@xxxxxxxxxxxxxxxx> thredds@xxxxxxxxxxxxxxxx
>> Subject: Re: NetCDF, WMS, and auto color scale (not set by admin)
>> 
>> Hi Terry,
>> 
>> Do you serve aggregations, or is each NetCDF file a separate dataset?
With
>> an 
>> aggregation, there isn't a one-to-one correspondence between a WMS layer
and
>> a 
>> NetCDF variable in a particular file. This is also true for things like 
>> velocity fields, which are derived on-the-fly from vector components. One
of
>> 
>> the THREDDS team could hopefully explain how variable attributes behave
>> under 
>> aggregation (do they just take the values from the first file in the 
>> aggregation?)
>> 
>> One problem with "baking" this information into the NetCDF file, is that 
>> people frequently change their mind about what the values should be. The 
>> concept of a "sensible view range" is not an intrinsic property of the
data,
>> 
>> hence we have always kept this as a WMS configuration issue.
>> 
>> By the way, valid_min/max isn't the solution I'm afraid. These attributes

>> define a range, outside of which data are considered to be missing. So
for
>> an 
>> ocean temperature field (with a sensible view range of [-2:35] degC
perhaps)
>> 
>> the valid_min/max could perfectly legitimately be set to [-100:100].
>> 
>> There was a proposal (which I don't think is official CF) to use 
>> "actual_min/max" attributes to denote the actual range of data in a
>> variable. 
>> There was reluctance to accept this into CF because it is redundant 
>> information, but I think it can be easily calculated and inserted into a 
>> NetCDF file using the NetCDF Operators (nco).
>> 
>> However, if you think that something like actual_min/max would work for
your
>> 
>> use case, we could consider whether this would be implementable as a
means
>> of 
>> providing the "sensible view range".
>> 
>> Cheers,
>> Jon
>> 
>> --
>> Dr Jon Blower,
>> Technical Director, Reading e-Science Centre, MELODIES project
coordinator, 
>> School of Mathematical and Physical Sciences, University of Reading
>> Tel: +44 118 378 5213
>> Email:  <mailto:j.d.blower@xxxxxxxxxxxxx> j.d.blower@xxxxxxxxxxxxx
>> 
>> On 21 Aug 2014, at 03:45, < <mailto:Terry.Rankine@xxxxxxxx>
Terry.Rankine@xxxxxxxx> < <mailto:Terry.Rankine@xxxxxxxx>
Terry.Rankine@xxxxxxxx> 
>> wrote:
>> 
>>> The two workflows we have are
>>> 1. Dump the data in "this" folder heriachy/structure - it will
>>> magically appear for you.
>>> 2. Email us and we will host it.
>>> 
>>> I cant see any issue with scripting it, but I was under the impression
>>> that changes to the xml file would require TDS to be restarted, plus
>>> its just another bit of data to keep in 'sync' somehow.
>>> 
>>> My desire here would be that, as an admin, I would tell the
>>> wmsConfig.xml <location /this/path/his-data-set-*.nc>
>>> <layer name=elevation>
>>>     <default color range>
>>>        <min>$delegate[elevation.vmin]</min>
>>>        <max>$delegate[elevation.vmax]</max>
>>>     </default color range>
>>>  </layer>
>>> </location>
>>> 
>>> As an admin I would deliberately hand over the 'setting the right view
>>> stuff' for those who could control the data production/attribution.
>>> Freeing me from that role.
>>> 
>>> I completely understand where it is not appropriate - but I don't
>>> believe that makes it 'inappropriate' for the simple cases.
>>> 
>>> Terry
>>> 
>>> -----Original Message-----
>>> From:  <mailto:thredds-bounces@xxxxxxxxxxxxxxxx>
thredds-bounces@xxxxxxxxxxxxxxxx
>>> [ <mailto:thredds-bounces@xxxxxxxxxxxxxxxx>
mailto:thredds-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Jon Blower
>>> Sent: Thursday, 21 August 2014 2:48 AM
>>> To:  <mailto:thredds@xxxxxxxxxxxxxxxx> thredds@xxxxxxxxxxxxxxxx
>>> Subject: Re: [thredds] NetCDF, WMS, and auto color scale (not set by
>>> admin)
>>> 
>>> Hi Terry,
>>> 
>>> As far as I remember THREDDS has always required setting of the
>>> default colour scale range by the admin. (However the standalone ncWMS
>>> could do this automatically, because it scans datasets during an
>>> ingestion step, which THREDDS does not perform.)
>>> 
>>> There is currently no way to read the default colour range from the
>>> NetCDF files, although a means could theoretically be invented, with 
>>> "special"
>>> attributes. One problem is that, in a multidimensional variable, the
>>> most appropriate range for one z or t level may not be so appropriate
>>> for another value of z or t. There are other interesting issues with
>>> the behaviour of NetCDF attributes in aggregations (what happens if
>>> different files in the same aggregation have different attributes?)
>>> 
>>> How do the data providers communicate their preferences to you at the
>>> moment? Could you, for example, craft a script that takes their values
>>> (from a simple text file maybe) and inserts it appropriately into the
>>> wmsConfig.xml automatically?
>>> 
>>> Cheers,
>>> Jon
>>> 
>>> 
>>>> 
>>>> 
>>>> ---------------------------------------------------------------------
>>>> -
>>>> 
>>>> Message: 1
>>>> Date: Wed, 20 Aug 2014 01:19:29 +0000
>>>> From: < <mailto:Terry.Rankine@xxxxxxxx> Terry.Rankine@xxxxxxxx>
>>>> To: < <mailto:thredds@xxxxxxxxxxxxxxxx> thredds@xxxxxxxxxxxxxxxx>
>>>> Subject: [thredds] NetCDF, WMS, and auto color scale (not set by
>>>>     admin)
>>>> Message-ID:
>>>> 
>>> <
<mailto:918EBF4F0E40D340B213B473CCAB6B126000492D@xxxxxxxxxxxxxxxxxxxxxxxxxx>
918EBF4F0E40D340B213B473CCAB6B126000492D@xxxxxxxxxxxxxxxxxxxxxxxxxx>
>>>> Content-Type: text/plain; charset="us-ascii"
>>>> 
>>>> Hi Guys
>>>> 
>>>> 
>>>> 
>>>> In older versions of thredds I could leave the auto color scaling for
>>>> a WMS to thredds to take care of for me. Can I have it back - but a
>>>> bit
>>> different.
>>>> 
>>>> 
>>>> 
>>>> As a system admin - I am just 'serving the data' not really
>>>> understanding what 'sensible viewing defaults' for
>>>> defaultColorScaleRange for any file in the directory should be.
>>>> However - the data owners I interact with all have a 'preference' on
>>>> how to color/scale their visualisations for their data files.
>>>> 
>>>> 
>>>> 
>>>> Is there a way to make thredds read the netCDF file - load the
>>>> variable, and load the visualisation range FROM THE DATA FILE - not
>>>> the wmsConfig.xml - where I would need to explicitly set it?
>>>> 
>>>> 
>>>> 
>>>> IE - if on the elevation layer for this service:
>>>> 
>>>>
<http://vhirl-dev.csiro.au/thredds/dodsC/dem/testconvert/bathymetry_su>
http://vhirl-dev.csiro.au/thredds/dodsC/dem/testconvert/bathymetry_su
>>>> r
>>>> veys_G
>>>> A-2476_100m_version1_WGS84UTM49S_esri_grid_grid100m_GA-2476_Exmouth_1
>>>> 0
>>>> 0m_exm
>>>> outh100m.nc4.html
>>>> 
>>>> 
>>>> 
>>>> I set a netcdf parameter like defaultColorScaleRange or something
>>>> less thredds specific, max/min or something which could guide the
'lazy'
>>>> configuration option for the data?
>>>> 
>>>> 
>>>> 
>>>> Regards
>>>> 
>>>> Terry
>>>> 
>>>> 
>>> 
>>> _______________________________________________
>>> thredds mailing list
>>>  <mailto:thredds@xxxxxxxxxxxxxxxx> thredds@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe,  visit:
>>>  <http://www.unidata.ucar.edu/mailing_lists/>
http://www.unidata.ucar.edu/mailing_lists/
>> 
> 

Attachment: smime.p7s
Description: S/MIME cryptographic signature

  • 2014 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: