Thanks to Jeff and Tom for their responses. Of the last several presentations
we have given on web services, one of the first questions has been server-side
function related (usually the mean over some dimension(s).). I am hoping
people who have implement W(C)PS solutions would go into more detail of the
service, and most importantly what that would require of the client software.
If 20% of the request will require a POST, then my client needs to code for
that. Again, I give the Matlab example, I can no longer count on using
"url.write". Also, while I have seen a number of "implementations" of some
OGC services that use GET, the standard usually calls for a POST. If i don't
write to the standard, then it is no longer a standard. IOOS ran into this
with their initial SOS service, they created a GET version, but then of course
this no longer followed the SOS standard at that time
Tom also raises the question of whether a GET based service could deal with
non-gridded data. Again, look at ERDDAP, we serve some very large in-situ
datasets (GTSPP for example).
Let's continue the discussion. I know ASA is working on a WPS service, maybe I
can get them to chime in on what they are doing and how it works.
I know Steve Hankin is on extended leave, but maybe Roland or Kevin can chime
in on experiences with F-TDS. As I initially said, what I am hoping for is
some initial agreement on basic approach, and basic syntax, so hopefully a
variety of services, including THREDDS, would be interesting at least on an
experimental basis be interested in implementing them.
-Roy
On Jul 1, 2012, at 3:08 PM, Tom Kunicki wrote:
>
> We've done a lot of web geo-processing and few things we've done could be
> represented with a GET request without some serious configuration
> (hard-coding arguments or making assumptions) on the server side. Also
> consider that while one may be able to configure a particular service
> endpoint for a maximum URL length, this will need to be respected along every
> hop (HTTP proxy, Web Application Firewalls, etc.) between the client and
> server. Also consider URL length restrictions placed by the HTTP client
> (whether it be an API implementation or a browser). I appreciate the desire
> for an simple and easy to use interface. Being able to use GET for all
> requests types would be great, but it just don't think its feasible for all
> but a few types of processing. For this to be of use the processing GET API
> will have to respect the lowest common denominator as for as maximum URL
> length (1024 bytes?).
>
> If you want to implement a processing API with a GET-only interface, then
> certainly do it. I don't think it would be of much general use outside of
> being able to perform an operation on a one or more grids on a single TDS
> endpoints.
>
> We use WPS, most often with POST (but GET is possible if the request is
> simple enough). As an aside, OGC is not all POST. 80% of requests (WFS,
> WCS, WMS) can be covered with GET requests. If you need something complex
> then you fall back to POST.
>
> Tom Kunicki
> Center for Integrated Data Analytics
> U.S. Geological Survey
> 8505 Research Way
> Middleton, WI 53562
>
>
> On Jul 1, 2012, at 3:13 PM, Roy Mendelssohn wrote:
>
>> BTW - a discussion we have been having around these parts is can you do
>> enough in the way of server-side functions without a POST (ie the URL
>> defines the function). That is why I would like to hear more from people
>> who are running F-TDS and GDS - how many requests do they get for server
>> side functions, but is the usual response time and download for these
>> request, how large are the usual expressions? And then contrast it with a
>> WPS or WCPS approach. I clearly believe in one approach, but I would
>> welcome people who are using some of these other approaches to describe what
>> they have done, the benefits of doing things that way, and what it means for
>> a client.
>>
>> Thanks,
>>
>> -Roy
>>
>> On Jul 1, 2012, at 11:25 AM, Dennis Heimbigner wrote:
>>
>>> Roy-
>>>
>>>> ... One comment. I think you misunderstood my point about
>>>> Matlab and R. I am not interested in Matlab specific
>>>> implementations. The point was because the URL completely
>>>> defines the request, I can implement scripts in any application
>>>> that can send an URL and receive a file in terms of functions
>>>> built-in to that application - that is my clients do not break as
>>>> the application or operating system change.
>>>
>>> Not quite sure I understand. This phrase "...receive a file in
>>> terms of functions built-in to that application" sounds
>>> like you are creating an association between functions defined
>>> on the client side and functions defined on the server side.
>>> Can you elaborate?
>>>
>>>> Why I strongly prefer, if it is at all reasonable, services that
>>>> only use GET, not POST.
>>>
>>> Again, that is only possible if you keep your requests
>>> short enough to not violate the URL length restrictions.
>>>
>>> =Dennis Heimbigner
>>> Unidata
>>>
>>>
>>>
>>> Roy Mendelssohn wrote:
>>>> Hi Dennis:
>>>> Thanks. One comment. I think you misunderstood my point about Matlab and
>>>> R. I am not interested in Matlab specific implementations. The point was
>>>> because the URL completely defines the request, I can implement scripts in
>>>> any application that can send an URL and receive a file in terms of
>>>> functions built-in to that application - that is my clients do not break
>>>> as the application or operating system change.
>>>> While I understand why this occurred, a few years ago we had straight
>>>> OPeNDAP implementations. We had a lot of users using scripts we developed
>>>> for Matlab, running under Windows. Due to updates in both Windows and
>>>> Matlab, the OPeNDAP files for Windows stopped working (at least for
>>>> Matlab). We had a lot of users that were left stranded and stranded for
>>>> quite a long time. Developing and maintaining clients, particularly
>>>> clients that are working within an application for which you have to write
>>>> code, very quickly becomes a non-trivial exercise.
>>>> Since we switched to a service where the URL completely defines the
>>>> request, our Matlab and R scripts have survived quite nicely any number of
>>>> updates both to the applications themselves and to the operating systems.
>>>> That is because the clients now only use functions built into the
>>>> applications.
>>>> Why I strongly prefer, if it is at all reasonable, services that only use
>>>> GET, not POST.
>>>> -Roy
>>>> On Jun 28, 2012, at 1:03 PM, Dennis Heimbigner wrote:
>>>>>> I am old and slow, but suppose I am in OpeNDAP, are you proposing
>>>>>> to separate say constraint expressions and server-side function
>>>>>> requests basically the same (ie I just scan what is after each
>>>>>> comma) or do you propose some method that signifies in the URL
>>>>>> that what follows is an expression? In F-TDS and GDS the form of
>>>>>> the URL is:
>>>>> First, I am proposing to subsume DAP constraints.
>>>>> Second, I am proposing, like DAP, to put the expressions
>>>>> in the query part of the URL (i.e. after the '?').
>>>>>
>>>>>> http://machine:port/thredds/dodsC/dataset_expr_{dataset2,dataset3,...}{expression1;expression2;...}.URLsuffix?constraint
>>>>> So, I would rewrite this as something more-or-less like this:
>>>>> http://machine.../dataset?expression1,expression2,...
>>>>> Where the expressions would include the references to dataset2, dataset3,
>>>>> and the constraint.
>>>>>
>>>>>> BTW, the reason I have asked about the experience of people who
>>>>>> are using F-TDS and GDS on whether synchronous requests can cover
>>>>>> the large majority of cases, is because I am very partial to
>>>>>> systems where the URL completely defines the request, and hence
>>>>>> essentially use GET as the verb.
>>>>> The synchronous/asynchronous issue is, for me, a separable issue.
>>>>> I should note that GET has a limit on the size of URLS, so
>>>>> there needs to be ways to deal with that. Two possibilities are
>>>>> 1) use POST or PUT, or 2) provide a way to upload a long expression
>>>>> in parts USING multiple GETs.
>>>>>
>>>>>> The reason for this is long
>>>>>> experience. where client code has broken with changes in
>>>>>> operating system and/or application, fixes were slow in coming,
>>>>>> so many users were left with nothing working. In a system where
>>>>>> the URL completely defined the request, say ERDDAP, in Matlab:
>>>>>>
>>>>>>>> link='http://coastwatch.pfeg.noaa.gov/erddap/griddap/erdBAsstamday.mat?sst[(2010-01-16T12:00:00Z):1:(2010-01-16T12:00:00Z)][(0.0):1:(0.0)][(30):1:(50.0)][(220):1:(240.0)]';
>>>>>>>> F=urlwrite(link,'cwatch.mat');
>>>>>>>>
>>>>>> Will get the related file, and the entire command is in Matlab,
>>>>>> no extra code required. The same in R is:
>>>>>>
>>>>>>>> download.file(url="http://coastwatch.pfeg.noaa.gov/erddap/griddap/erdBAsstamday.nc?sst[(2010-01-16T12:00:00Z):1:(2010-01-16T12:00:00Z)][(0.0):1:(0.0)][(30):1:(50.0)][(220):1:(240.0)]",
>>>>>>>> destfile="AGssta.nc",mode='wb')
>>>>>>>>
>>>>>> again, "download.file" is an R command.
>>>>> I think that we do not want to be R/MATLAB specific
>>>>> in a proposal to put stuff in URLs. I would rather
>>>>> propose to allow uploading of R/MATLAB scripts to serve
>>>>> as additional, user-defined functions.
>>>>>
>>>>> I would prefer to
>>>>>> maintain this simplicity and cover 80% of the cases if possible,
>>>>>> than cover the rest but where more complex, application specific
>>>>>> code would have to be developed and maintained.
>>>>> Agreed. However my assumption is the the output of any function that
>>>>> is not assigned to a single-assignment variable will be returned as part
>>>>> of the response; but other ways of specifying this are possible within
>>>>> the functional framework I am proposing.
>>>>>
>>>>> =Dennis Heimbigner
>>>>> Unidata
>>>> **********************
>>>> "The contents of this message do not reflect any position of the U.S.
>>>> Government or NOAA."
>>>> **********************
>>>> Roy Mendelssohn
>>>> Supervisory Operations Research Analyst
>>>> NOAA/NMFS
>>>> Environmental Research Division
>>>> Southwest Fisheries Science Center
>>>> 1352 Lighthouse Avenue
>>>> Pacific Grove, CA 93950-2097
>>>> e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
>>>> voice: (831)-648-9029
>>>> fax: (831)-648-8440
>>>> www: http://www.pfeg.noaa.gov/
>>>> "Old age and treachery will overcome youth and skill."
>>>> "From those who have been given much, much will be expected" "the arc of
>>>> the moral universe is long, but it bends toward justice" -MLK Jr.
>>
>> **********************
>> "The contents of this message do not reflect any position of the U.S.
>> Government or NOAA."
>> **********************
>> Roy Mendelssohn
>> Supervisory Operations Research Analyst
>> NOAA/NMFS
>> Environmental Research Division
>> Southwest Fisheries Science Center
>> 1352 Lighthouse Avenue
>> Pacific Grove, CA 93950-2097
>>
>> e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
>> voice: (831)-648-9029
>> fax: (831)-648-8440
>> www: http://www.pfeg.noaa.gov/
>>
>> "Old age and treachery will overcome youth and skill."
>> "From those who have been given much, much will be expected"
>> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
>>
>> _______________________________________________
>> thredds mailing list
>> thredds@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>
>
**********************
"The contents of this message do not reflect any position of the U.S.
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097
e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/
"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
