Hi Jon We updated one of our archives to have - actual_min/actual_max recorded in each layer in each file. We are now in a good position to progress the work. How do we go about this? Does this fall into core development or is this something we need find resources to write ourselves? I found another back reference to 'auto color scaling' - all be it not quite the same. http://www.unidata.ucar.edu/mailing_lists/archives/thredds/2010/msg00295.htm l The other one we want to get going is the "out of range" customisation for the WMS. There are a few posts on this - by us and others - and this would be a great config option to have in the wms config file. Something like being able to set <outOfRange>transparent</outOfRange> instead of it going black or white etc. Regards Terry From: thredds-bounces@xxxxxxxxxxxxxxxx [mailto:thredds-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Terry.Rankine@xxxxxxxx Sent: Saturday, 23 August 2014 12:33 PM To: j.d.blower@xxxxxxxxxxxxx Cc: thredds@xxxxxxxxxxxxxxxx Subject: [ExternalEmail] Re: [thredds] NetCDF, WMS, and auto color scale (not set by admin) Our grids are relatively small, 300gig is normal or high res scans are about 1T compressed. I would think that the actual value scanner would be too slow as it would be scanning millions of values. Our Python cleanup/inspection code takes over 30+ seconds for the large sets. I thing actuals would be great, but since i have no simple performance measure (max file size? Max points?) to protect it, the usecase for me seems "too dangerous" to use without protection. (Ie a user dumps a high res file In a location set up for the $actual scanner.) What about having an "approx" scanner, some special 1000 point stride of the data or something like that. In some vis packages they have an approx scanner. $quickApprox(1000), or a function called quickandwrong sounds like the right name.... So: * can you protect the $actual function * can you have an approximate function. Approx function would actually be a pretty good systemwide default. As for the data providers wanting to tell me what they want the vis range to be, 2/7 currently think that way, the others think I should know the range from the data.... I had the chat about vis range to 1 from each opinion. Both think it would be agreed they could add the extra variables, but some form of automation would be better than -50 50. so for me it would be set and forget for either 'use their values' or roughly guessing by scanning the data. Both thought that some stats in the header would be a good thing, and then argured about the value if a histogram encoded as data variable ... Regards Terry. _____ From: Jon Blower Sent: Friday, 22 August 2014 6:41:03 AM To: Rankine, Terry (Mineral Resources, Kensington) Cc: <mailto:thredds@xxxxxxxxxxxxxxxx> thredds@xxxxxxxxxxxxxxxx Subject: Re: NetCDF, WMS, and auto color scale (not set by admin) P.S. I forgot to ask - what was your objection to using the actual range of values within the variable as the default? I'm assuming your data are 2D fields, hence reasonably small. It would be feasible to implement a scheme like this: <variable id="elevation"> <defaultColorScaleRange>$actual</defaultColorScaleRange> </variable> Then THREDDS, on first reading the variable, would find the range of values in the variable and use this as the scale range. It could use an actual_min attribute, but I'm suggesting that it could just calculate this itself by reading the whole variable array. It could then cache the value range for future requests. The advantage would be that the data provider doesn't need to do anything. The disadvantage would be that performance would be affected, for the first request at least. Performance would be particularly bad for large aggregations, but this is not your use case. Sysadmins wouldn't need to enable this feature for such cases. This could be made more sophisticated for Mike's suggestion: <defaultColorScaleRange>$percentile($actual, 2.75)</defaultColorScaleRange> or something like that. So I guess the question is - will your data providers need a default scale range that is *not* based somehow on the actual range of values in the data file? Cheers, Jon - Dr Jon Blower, Technical Director, Reading e-Science Centre, MELODIES project coordinator, School of Mathematical and Physical Sciences, University of Reading Tel: +44 118 378 5213 Email: <mailto:j.d.blower@xxxxxxxxxxxxx> j.d.blower@xxxxxxxxxxxxx On 22 Aug 2014, at 07:31, Jon Blower < <mailto:j.d.blower@xxxxxxxxxxxxx> j.d.blower@xxxxxxxxxxxxx> wrote: > Hi Terry, > > I think that your suggestion is implementable, I'm just trying to work through the potential issues so that a means can be found to work for the maximum number of users without causing surprising behaviour or the need for special-case coding. > > I think the remaining issues would be: > > 1. Choice of the attribute names for the suggested scale range. Does anyone else add this information to NetCDF files and are there any commonly-used attribute names? I'll ask the CF community this question. > > 2. Behaviour with packed variables. I would assume that the most useful thing would be for the view range to be encoded with respect to unpacked variable values. (Valid_min/max can be encoded with respect to either packed or unpacked values.) > > So I think we should probably do this, but it would be useful to have some input from other users as they may raise things that we haven't thought of. > > Cheers, > Jon > > -- > Dr Jon Blower, > Technical Director, Reading e-Science Centre, > MELODIES project coordinator, > School of Mathematical and Physical Sciences, > University of Reading > Tel: +44 118 378 5213 > Email: <mailto:j.d.blower@xxxxxxxxxxxxx> j.d.blower@xxxxxxxxxxxxx > > On 22 Aug 2014, at 03:29, <mailto:Terry.Rankine@xxxxxxxx> Terry.Rankine@xxxxxxxx wrote: > >> No we don't use aggregations. >> >> I am an Australian, and all I really want: is to do less work! >> The ability for a server to do some of that work for me, is awesome! Your >> thoughts and help is appreciated. >> >> I am not suggesting that we change THREDDS defaults, I am suggesting that we >> >> have the option to allow an admin to delegate a default/preference to the >> data >> owner. >> >> The 'pseudo code' would allow arbitrary delegation using a selector to pull >> the data out of the netcdf header. >>> <default color range> >>> <min>$delegate[elevation.someVar]</min> >>> <max>$delegate[elevation.otherVar]</max> >>> </default color range> >> As an admin of multiple mixed systems, I would be free-ed, where >> appropriate, >> to allow the 'view range' to come from a dataset. >> >> Just to give you an example, we sent a boat out to do some bathymetry scans, >> >> they dumped 3 netcdf's in the 'please host me folder'. >> Three separate runs were done, our variable is 'elevation' in this context. >> The near shore one was (-200 -20), and the two offshore where (~-6500,-50), >> (-2000, -50). >> >> Are you suggesting that: >> * Because others use aggregations - you won't consider the ability >> delegate and to read from the file? >> * Or are you saying that actual_min/max sound good - lets hard code them >> in? >> >> I think neither of those are wonderful conclusions. I see value in the >> delegation pattern, with a 'configured' selector, today they can be read >> from a header (simple), tomorrow they could be an extra function like Mike >> McCann < <mailto:mccann@xxxxxxxxx> mccann@xxxxxxxxx> uses manually now: >> So the default min (max would be the same) could end up looking like: >> * $select[file.layer.somvar] >> * $calculate[$math.percentile(file.layer, 2.75)] >> With the obvious wording - you can make assumptions on 'performance' >> >> Once again - "I am just lazy" and looking for a better integration between >> data-owner, and administrator. >> >> So for now - I will look at capturing the min/max (and hacking in a >> percentile variable too - good idea Mike), and educating my data providers >> to do the same from now on. >> What Thredds does with the data - well that's now up to you guys! >> >> Regards >> Terry >> "That lazy architect guy" >> >> >> -----Original Message----- >> From: Jon Blower [ <mailto:j.d.blower@xxxxxxxxxxxxx> mailto:j.d.blower@xxxxxxxxxxxxx] >> Sent: Thursday, 21 August 2014 4:37 PM >> To: Rankine, Terry (Mineral Resources, Kensington) >> Cc: <mailto:thredds@xxxxxxxxxxxxxxxx> thredds@xxxxxxxxxxxxxxxx >> Subject: Re: NetCDF, WMS, and auto color scale (not set by admin) >> >> Hi Terry, >> >> Do you serve aggregations, or is each NetCDF file a separate dataset? With >> an >> aggregation, there isn't a one-to-one correspondence between a WMS layer and >> a >> NetCDF variable in a particular file. This is also true for things like >> velocity fields, which are derived on-the-fly from vector components. One of >> >> the THREDDS team could hopefully explain how variable attributes behave >> under >> aggregation (do they just take the values from the first file in the >> aggregation?) >> >> One problem with "baking" this information into the NetCDF file, is that >> people frequently change their mind about what the values should be. The >> concept of a "sensible view range" is not an intrinsic property of the data, >> >> hence we have always kept this as a WMS configuration issue. >> >> By the way, valid_min/max isn't the solution I'm afraid. These attributes >> define a range, outside of which data are considered to be missing. So for >> an >> ocean temperature field (with a sensible view range of [-2:35] degC perhaps) >> >> the valid_min/max could perfectly legitimately be set to [-100:100]. >> >> There was a proposal (which I don't think is official CF) to use >> "actual_min/max" attributes to denote the actual range of data in a >> variable. >> There was reluctance to accept this into CF because it is redundant >> information, but I think it can be easily calculated and inserted into a >> NetCDF file using the NetCDF Operators (nco). >> >> However, if you think that something like actual_min/max would work for your >> >> use case, we could consider whether this would be implementable as a means >> of >> providing the "sensible view range". >> >> Cheers, >> Jon >> >> -- >> Dr Jon Blower, >> Technical Director, Reading e-Science Centre, MELODIES project coordinator, >> School of Mathematical and Physical Sciences, University of Reading >> Tel: +44 118 378 5213 >> Email: <mailto:j.d.blower@xxxxxxxxxxxxx> j.d.blower@xxxxxxxxxxxxx >> >> On 21 Aug 2014, at 03:45, < <mailto:Terry.Rankine@xxxxxxxx> Terry.Rankine@xxxxxxxx> < <mailto:Terry.Rankine@xxxxxxxx> Terry.Rankine@xxxxxxxx> >> wrote: >> >>> The two workflows we have are >>> 1. Dump the data in "this" folder heriachy/structure - it will >>> magically appear for you. >>> 2. Email us and we will host it. >>> >>> I cant see any issue with scripting it, but I was under the impression >>> that changes to the xml file would require TDS to be restarted, plus >>> its just another bit of data to keep in 'sync' somehow. >>> >>> My desire here would be that, as an admin, I would tell the >>> wmsConfig.xml <location /this/path/his-data-set-*.nc> >>> <layer name=elevation> >>> <default color range> >>> <min>$delegate[elevation.vmin]</min> >>> <max>$delegate[elevation.vmax]</max> >>> </default color range> >>> </layer> >>> </location> >>> >>> As an admin I would deliberately hand over the 'setting the right view >>> stuff' for those who could control the data production/attribution. >>> Freeing me from that role. >>> >>> I completely understand where it is not appropriate - but I don't >>> believe that makes it 'inappropriate' for the simple cases. >>> >>> Terry >>> >>> -----Original Message----- >>> From: <mailto:thredds-bounces@xxxxxxxxxxxxxxxx> thredds-bounces@xxxxxxxxxxxxxxxx >>> [ <mailto:thredds-bounces@xxxxxxxxxxxxxxxx> mailto:thredds-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Jon Blower >>> Sent: Thursday, 21 August 2014 2:48 AM >>> To: <mailto:thredds@xxxxxxxxxxxxxxxx> thredds@xxxxxxxxxxxxxxxx >>> Subject: Re: [thredds] NetCDF, WMS, and auto color scale (not set by >>> admin) >>> >>> Hi Terry, >>> >>> As far as I remember THREDDS has always required setting of the >>> default colour scale range by the admin. (However the standalone ncWMS >>> could do this automatically, because it scans datasets during an >>> ingestion step, which THREDDS does not perform.) >>> >>> There is currently no way to read the default colour range from the >>> NetCDF files, although a means could theoretically be invented, with >>> "special" >>> attributes. One problem is that, in a multidimensional variable, the >>> most appropriate range for one z or t level may not be so appropriate >>> for another value of z or t. There are other interesting issues with >>> the behaviour of NetCDF attributes in aggregations (what happens if >>> different files in the same aggregation have different attributes?) >>> >>> How do the data providers communicate their preferences to you at the >>> moment? Could you, for example, craft a script that takes their values >>> (from a simple text file maybe) and inserts it appropriately into the >>> wmsConfig.xml automatically? >>> >>> Cheers, >>> Jon >>> >>> >>>> >>>> >>>> --------------------------------------------------------------------- >>>> - >>>> >>>> Message: 1 >>>> Date: Wed, 20 Aug 2014 01:19:29 +0000 >>>> From: < <mailto:Terry.Rankine@xxxxxxxx> Terry.Rankine@xxxxxxxx> >>>> To: < <mailto:thredds@xxxxxxxxxxxxxxxx> thredds@xxxxxxxxxxxxxxxx> >>>> Subject: [thredds] NetCDF, WMS, and auto color scale (not set by >>>> admin) >>>> Message-ID: >>>> >>> < <mailto:918EBF4F0E40D340B213B473CCAB6B126000492D@xxxxxxxxxxxxxxxxxxxxxxxxxx> 918EBF4F0E40D340B213B473CCAB6B126000492D@xxxxxxxxxxxxxxxxxxxxxxxxxx> >>>> Content-Type: text/plain; charset="us-ascii" >>>> >>>> Hi Guys >>>> >>>> >>>> >>>> In older versions of thredds I could leave the auto color scaling for >>>> a WMS to thredds to take care of for me. Can I have it back - but a >>>> bit >>> different. >>>> >>>> >>>> >>>> As a system admin - I am just 'serving the data' not really >>>> understanding what 'sensible viewing defaults' for >>>> defaultColorScaleRange for any file in the directory should be. >>>> However - the data owners I interact with all have a 'preference' on >>>> how to color/scale their visualisations for their data files. >>>> >>>> >>>> >>>> Is there a way to make thredds read the netCDF file - load the >>>> variable, and load the visualisation range FROM THE DATA FILE - not >>>> the wmsConfig.xml - where I would need to explicitly set it? >>>> >>>> >>>> >>>> IE - if on the elevation layer for this service: >>>> >>>> <http://vhirl-dev.csiro.au/thredds/dodsC/dem/testconvert/bathymetry_su> http://vhirl-dev.csiro.au/thredds/dodsC/dem/testconvert/bathymetry_su >>>> r >>>> veys_G >>>> A-2476_100m_version1_WGS84UTM49S_esri_grid_grid100m_GA-2476_Exmouth_1 >>>> 0 >>>> 0m_exm >>>> outh100m.nc4.html >>>> >>>> >>>> >>>> I set a netcdf parameter like defaultColorScaleRange or something >>>> less thredds specific, max/min or something which could guide the 'lazy' >>>> configuration option for the data? >>>> >>>> >>>> >>>> Regards >>>> >>>> Terry >>>> >>>> >>> >>> _______________________________________________ >>> thredds mailing list >>> <mailto:thredds@xxxxxxxxxxxxxxxx> thredds@xxxxxxxxxxxxxxxx >>> For list information or to unsubscribe, visit: >>> <http://www.unidata.ucar.edu/mailing_lists/> http://www.unidata.ucar.edu/mailing_lists/ >> >
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
thredds archives: