Re: [thredds] nco as a web service

having taken a quick look at the WPS spec,
I think my main objection to it is its
verbosity. I realize that part of that comes from using
XML. However, a spec that takes 30+ tables to
describe the format is, in my opinion, way out of control.
I strongly prefer concise, even terse, and at least semi-formal
descriptions.

A specific request: does anyone have a RELAX-NG grammar
for the WPS spec, or any sort of grammar at all?

=Dennis Heimbigner
 Unidata

Roy Mendelssohn wrote:
Thanks for this  (and Rich's follow-up).    This is very interesting. And it 
can be used as service, correct, not just from a web page?

Is it possible to see some (or all) of the WPS code? I know Eoin's group at ASA is working on using WPS in an implementation perhaps we can get them to chime in.

I can't promise any help at the moment (but am very interested)- we are already stretched 
way beyond our capabilities and  are sort of going through an "institutional 
existential crisis" as it were, but in this case it is real rather than 
psychological.

BTW - I look at services based on how well they work and how easy they are to 
use  (from the client side) not what adjectives are attached to them  (REST, 
OGC, etc etc).  That is why I have always and keep pushing for working code, 
not paper.   A lot of things that look good  on paper are dogs in practice.

-Roy


On Jul 1, 2012, at 10:55 PM, Tom Kunicki wrote:

On (C) I definitely concur. I am not against simplicity and HTTP GET requests. I just want to make sure that the approach is discussed and that one doesn't fall into the trap of believing HTTP GET is a panacea of simplicity. These URLs that have been posted are pretty complex and aren't the kinds of things that anyone but expert users will be crafting by hand. There will be a client implementation in front of them and they will need to be updated if the server processing API behind them changes. In this case, the client implementation will have to change in tandem with the server side processing API. This will be true regardless of whether the request is GET, POST, PUT, etc. One benefit of GET is an embeddable link, to my knowledge this isn't easily done with POST or PUT.
Our group uses WPS.  We had issues with some holes with some implementation and 
the specification so we made a choice to join on to the WPS 2.0 SWG.

There are advantages to the WPS specification.  Implementations can list a set 
of supported operations and processes using the GetCapabilities request (a GET 
or POST, we use GET).  Each process can be queried for it's API including 
supported inputs and outputs (name, mime-type and schema if xml) using a 
DescribeProcess request (GET or POST, we use GET). If you know the arguments 
and types you can parse the DescribeProcess response and automatically generate 
a UI.  We have implemented this in JavaScript for our Web-based brokering 
services.  There are python clients as well as an Arc plugin in-progress 
(completed?) by ERSI and 52n, also a qGIS plugin among others.  Processes can 
be executed with an Execute request (a GET or POST request, we use POST).  POST 
for us because we deal with some pretty complex inputs (WFS calls with server 
side geometry filtering by reference to a GET or POST request; or Base64 
encoded shapefiles sent in-line).  These would bump us into some URL
l
 en
gth restrictions we have dealt with in the past.  We don't have to use these 
complex inputs but since WPS offers this flexibility we are happy to leverage 
it.  When we execute processes we have the options to execute them 
synchronously or asynchronously (and an implementation can control these 
options by advertising them per process.)  We can query the executing process 
for it's completion state (POST, don't know if GET is possible as I haven't 
looked into it).  We can request executions results in-line with the response 
or by reference.  We provide inputs to WPS calls as the results of other WPS 
calls.  WPS processing implementations can be complex or simple.  Given our use 
cases we made an architectural decision to leverage some of the more advanced 
components of the specification.  We've developed some complex processing that 
does some cool and useful things that we are able to leverage in other projects 
and share with other groups.  With our processing endpoints we ca
n
 a
dd a process and have it automatically be displayed in our UIs.  One of the 
benefits of WPS was processing end-points became self-documenting.

Now, the WPS execute by GET is pretty tricky as it requires so double URL 
encoding.  We are happy using POST and didn't delve too much into GET. If there 
was a need and someone wanted to look at this with me (ahem, Roy?) I would be 
more that happy to submit some change requests to simplify the specification 
for some use cases.  In my experience with the OGC standards almost everything 
can be done with GET, it's when you get into the outlying use cases you have to 
represent your requests with POST.

WPS is an OGC specification.  I think the last 2 words of the previous sentence 
instantly turn people off.  But there's some real value to the work that's been 
done.  We've used it as a thin wrapper on process execution.  Our initial cut 
at processing involved using simple GET-based services.  We found we had to 
generate a whole suite of utility/supporting GET-based services relying on 
clients to perform operations with correct ordering.  The architecture was 
becoming difficult to maintain and document. A large number of tasks have now 
been implemented with the OGC standards suite and available standards 
implementations.  This has saved our group a lot of development time and in 
turn taxpayer dollars.

Tom Kunicki
Center for Integrated Data Analytics
U.S. Geological Survey
8505 Research Way
Middleton, WI 53562


On Jul 1, 2012, at 11:34 PM, Gerry Creager wrote:

Roy,

That's a good explanation, and one I can live with. However, I also agree with 
Jeff's later comments, that A) in general, the same interpreter can handle GET 
and POST, and B) file uploads can't happen with a GET.

And, most important: C) KISS is a good mantra.

I'll sit back and listen to the debate again.

gerry

On Sun, Jul 1, 2012 at 3:13 PM, Roy Mendelssohn <roy.mendelssohn@xxxxxxxx> 
wrote:
BTW - a discussion we have been having around these parts is can you do enough 
in the way of server-side functions without a POST  (ie the URL defines the 
function).  That is why I would like to hear more from people who are running 
F-TDS and GDS - how many requests do they get for server side functions, but is 
the usual response time and download for these request, how large are the usual 
expressions?  And then contrast it with a WPS or WCPS approach.    I clearly 
believe in one approach, but I would welcome people who are using some of these 
other approaches to describe what they have done, the benefits of doing things 
that way, and what it means for a client.

Thanks,

-Roy

On Jul 1, 2012, at 11:25 AM, Dennis Heimbigner wrote:

Roy-

...  One comment.  I think you misunderstood my point about
Matlab and R.  I am not interested in Matlab specific
implementations.  The point was because the URL completely
defines the request, I can implement scripts in any application
that can send an URL and receive a file in terms of functions
built-in to that application - that is my clients do not break as
the application or operating system change.
Not quite sure I understand. This phrase "...receive a file in
terms of functions built-in to that application" sounds
like you are creating an association between functions defined
on the client side and functions defined on the server side.
Can you elaborate?

Why I strongly prefer, if it is at all reasonable, services that
only use GET, not POST.
Again, that is only possible if you keep your requests
short enough to not violate the URL length restrictions.

=Dennis Heimbigner
Unidata



Roy Mendelssohn wrote:
Hi Dennis:
Thanks.  One comment.  I think you misunderstood my point about Matlab and R.  
I am not interested in Matlab specific implementations.  The point was because 
the URL completely defines the request, I can implement scripts in any 
application that can send an URL and receive a file in terms of functions 
built-in to that application  - that is my clients do not break as the 
application or operating system change.
While I understand why this occurred, a few years ago we had straight OPeNDAP 
implementations.  We had a lot of users using scripts we developed for Matlab, 
running under Windows.  Due to updates in both Windows and Matlab, the OPeNDAP 
files for Windows stopped working (at least for Matlab).  We had a lot of users 
that were left stranded and stranded for quite a long time.  Developing and 
maintaining clients, particularly clients that are working within an 
application for which you have to write code, very quickly becomes a 
non-trivial exercise.
Since we switched to a service where the URL completely defines the request, 
our Matlab and R scripts have survived quite nicely any number of updates both 
to the applications themselves and to the operating systems.  That is because 
the clients now only use functions built into the applications.
Why I strongly prefer, if it is at all reasonable, services that only use GET, 
not POST.
-Roy
On Jun 28, 2012, at 1:03 PM, Dennis Heimbigner wrote:
I am old and slow, but suppose I am in OpeNDAP, are you proposing
to separate say constraint expressions and server-side function
requests basically the same (ie I just scan what is after each
comma) or do you propose some method that signifies in the URL
that what follows is an expression?  In F-TDS and GDS the form of
the URL is:
First, I am proposing to subsume DAP constraints.
Second, I am proposing, like DAP, to put the expressions
in the query part of the URL (i.e. after the '?').

http://machine:port/thredds/dodsC/dataset_expr_{dataset2,dataset3,...}{expression1;expression2;...}.URLsuffix?constraint
So, I would rewrite this as something more-or-less like this:
http://machine.../dataset?expression1,expression2,...
Where the expressions would include the references to dataset2, dataset3,
and the constraint.

BTW, the reason I have asked about the experience of people who
are using F-TDS and GDS on whether synchronous requests can cover
the large majority of cases, is because I am very partial to
systems where the URL completely defines the request, and hence
essentially use GET as the verb.
The synchronous/asynchronous issue is, for me, a separable issue.
I should note that GET has a limit on the size of URLS, so
there needs to be ways to deal with that. Two possibilities are
1) use POST or PUT, or 2) provide a way to upload a long expression
in parts USING multiple GETs.

The reason for this is long
experience.  where client code has broken with changes in
operating system and/or application, fixes were slow in coming,
so many users were left with nothing working.  In a system where
the URL completely defined the request, say ERDDAP, in Matlab:

link='http://coastwatch.pfeg.noaa.gov/erddap/griddap/erdBAsstamday.mat?sst[(2010-01-16T12:00:00Z):1:(2010-01-16T12:00:00Z)][(0.0):1:(0.0)][(30):1:(50.0)][(220):1:(240.0)]';
F=urlwrite(link,'cwatch.mat');

Will get the related file, and the entire command is in Matlab,
no extra code required.  The same in R is:

download.file(url="http://coastwatch.pfeg.noaa.gov/erddap/griddap/erdBAsstamday.nc?sst[(2010-01-16T12:00:00Z):1:(2010-01-16T12:00:00Z)][(0.0):1:(0.0)][(30):1:(50.0)][(220):1:(240.0)]",
 destfile="AGssta.nc",mode='wb')

again, "download.file" is an R command.
I think that we do not want to be R/MATLAB specific
in a proposal to put stuff in URLs. I would rather
propose to allow uploading of R/MATLAB scripts to serve
as additional, user-defined functions.

I would prefer to
maintain this simplicity and cover 80% of the cases if possible,
than cover the rest but where more complex, application specific
code would have to be developed and maintained.
Agreed. However my assumption is the the output of any function that
is not assigned to a single-assignment variable will be returned as part
of the response; but other ways of specifying this are possible within
the functional framework I am proposing.

=Dennis Heimbigner
Unidata
**********************
"The contents of this message do not reflect any position of the U.S. Government or 
NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097
e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/
"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected" "the arc of the moral 
universe is long, but it bends toward justice" -MLK Jr.
**********************
"The contents of this message do not reflect any position of the U.S. Government or 
NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/

_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/

_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/

**********************
"The contents of this message do not reflect any position of the U.S. Government or 
NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected" "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/