Hi, Dennis --
I would argue that a better strategy for the client library would be
to minimize the size of data transfers from the servers at all times.
Most if not all OPeNDAP users are motivated by limited internet
bandwidth that makes it impossible to download entire data sets via
FTP. Why pad their requests unnecessarily? The client library should
not anticipate what the next request might be, it should get the
requested data as fast as possible. How about disabling over-requests
and client side cacheing by default, and giving the user the option to
override the settings in .dodsrc?
In the GrADS community, many unexplained problems with OPeNDAP access
were solved by deleting the client-side cache and re-issuing the
request. I don't know how the cache got corrupted, but this problem/
solution came up so many times, I ended up recommending users abandon
client-side cacheing completely.
--Jennifer
On Oct 26, 2009, at 10:11 PM, Dennis Heimbigner wrote:
The reason to download extra information is to cache it
on the client side so that some other client side request
can be addressed without contacting the server.
Currently, my netcdf client code does over-request
when that request is not "too large" as defined by a
user-configurable parameter.
=Dennis Heimbigner
Jennifer Adams wrote:
I've looked more closely at the GDS code ... although a .dods
request that is missing any constraints is rejected automatically,
there is no restriction on the size of a subset request; it is not
even configurable. I guess this is because the subsets are never
cached. If the client has the bandwidth and the patience to attempt
to download a terabyte of data in a single request, the GDS will
try to comply. The server does allow me to constrain the size of
server-side analysis results -- that's because these are cached.
In any event, even if GDS did allow a .dods request without a
constraint, I think it would be a mistake to use that syntax to get
the coordinate data values. Why download ALL data when you just
need a 1-D array for a single variable?
--Jennifer
On Oct 26, 2009, at 4:41 PM, Dennis Heimbigner wrote:
The issue is really the size of the returned dataset.
I would much prefer that the server complain that
the caller is asking for too much data (defined by some
preset limit) than forcing the addition of constraints.
Note that I can still ask for the whole dataset by just
putting in a set of constraint projections that ask for everything.
So, using the presence of constraints is not IMO a good idea.
=Dennis Heimbigner
Jennifer Adams wrote:
Hi, Dennis, James, et al. -- I didn't know before today that a
GDS request with only the ".dods" extension is automatically
rejected by the server, but it makes sense ... it would mean
delivering the entire data set -- coordinate and data variables
in their entirety -- to the client. For large data sets this
would be doomed to fail. But in order to fulfill the 'ncdump -c'
request, the entire data set is not required, only the .das and
the .dds and the data values for the coordinate variables need to
be retrieved. One other detail ... when I try 'ncdump -h', the
server delivers the .das and .dds without errors, but ncdump only
shows 1 out of 4 coordinate variables. --Jennifer
On Oct 26, 2009, at 2:29 PM, Dennis Heimbigner wrote:
Jennifer-
As you discovered, some servers (including GrADS)
will not serve up a whole dataset, which is what you
may get if you ask for, for example,
http://monsoondata.org:9090/dods/model
Other servers do not require constraints as
part of the DAP request.
[James- is the GrADS behavior (requiring a constraint)
proper or improper behavior for an OPeNDAP server?]
=Dennis Heimbigner
Jennifer Adams wrote:
Dear Experts -- I am new to this group but not to NetCDF or
OPeNDAP. I have been testing netcdf-4.1 for use with GrADS. I
had noticed some problems ncdump was having in getting the
attribute metadata for the coordinate axes of an OPeNDAP data
set. I tried it with the latest snapshot (dated 2009102000) and
ncdump is working much better, but it is not yet getting it
right. The data set in question is behind a GrADS Data Server,
a 4-dimensional low-resolution demo data set used for the
tutorial and testing. Before running ncdump, I followed the
recommendation from an earlier post and set the environment
variable OCLOGFILE to <blank>. Here is the output:
ncdump -c http://monsoondata.org:9090/dods/model
# ./ncdump -c http://cola51x:9090/dods/model
Warning::USE_CACHE: not currently supported
Warning::MAX_CACHE_SIZE: not currently supported
Warning::MAX_CACHED_OBJ: not currently supported
Warning::IGNORE_EXPIRES: not currently supported
Warning::CACHE_ROOT: not currently supported
Warning::DEFAULT_EXPIRES: not currently supported
Warning::ALWAYS_VALIDATE: not currently supported
netcdf model {
dimensions:
lat = 46 ;
lev = 7 ;
lon = 72 ;
time = 5 ;
variables:
double lat(lat) ;
lat:grads_dim = "y" ;
lat:grads_mapping = "linear" ;
lat:grads_size = "46" ;
lat:units = "degrees_north" ;
lat:long_name = "latitude" ;
lat:minimum = -90. ;
lat:maximum = 90. ;
lat:resolution = 4.f ;
float ua(time, lev, lat, lon) ;
ua:_FillValue = 1.e+20f ;
ua:missing_value = 1.e+20f ;
ua:long_name = "eastward wind [m/s] " ;
float ps(time, lat, lon) ;
ps:_FillValue = 1.e+20f ;
ps:missing_value = 1.e+20f ;
ps:long_name = "surface pressure [hpa] " ;
float va(time, lev, lat, lon) ;
va:_FillValue = 1.e+20f ;
va:missing_value = 1.e+20f ;
va:long_name = "northward wind [m/s] " ;
float zg(time, lev, lat, lon) ;
zg:_FillValue = 1.e+20f ;
zg:missing_value = 1.e+20f ;
zg:long_name = "geopotential height [m] " ;
float ta(time, lev, lat, lon) ;
ta:_FillValue = 1.e+20f ;
ta:missing_value = 1.e+20f ;
ta:long_name = "air temperature [k] " ;
float hus(time, lev, lat, lon) ;
hus:_FillValue = 1.e+20f ;
hus:missing_value = 1.e+20f ;
hus:long_name = "specific humidity [kg/kg] " ;
float ts(time, lat, lon) ;
ts:_FillValue = 1.e+20f ;
ts:missing_value = 1.e+20f ;
ts:long_name = "surface (2m) air temperature [k] " ;
float pr(time, lat, lon) ;
pr:_FillValue = 1.e+20f ;
pr:missing_value = 1.e+20f ;
pr:long_name = "total precipitation rate [kg/
(m^2*s)] " ;
// global attributes:
:title = "Sample Model Data" ;
:Conventions = "COARDSGrADS" ;
:dataType = "Grid" ;
:history = "Mon Oct 26 12:59:29 EDT 2009 :
imported by GrADS Data Server 2.0" ;
data:
Error: :oc_open: server error retrieving url: code=0
message="subset requests must include a constraint expression"
./ncdump: NetCDF: DAP server side error
lat = Since I am the administrator of the server, I can look in
the logs and see what went wrong. The request that failed and
led to the error message above was: ...GET /model.dods
The request from the client (ncdump) should have the ?
varname[constraints] syntax following ".dods" , like
this: ...GET /model.dods?time[0:1:4] ...GET /model.dods?
lev[0:1:6] ...GET /model.dods?lat[0:1:45]
...GET /model.dods?lon[0:1:71] I also note that only 1 out of
the 4 coordinate axes (lat) shows up as a data variable. I hope
this is helpful information. The DAP interface in netcdf-4.1 is
vital to GrADS, I am eager to adopt it as soon as it is working
properly. --Jennifer
--
Jennifer M. Adams
IGES/COLA
4041 Powder Mill Road, Suite 302
Calverton, MD 20705
jma@xxxxxxxxxxxxx <mailto:jma@xxxxxxxxxxxxx>
------------------------------------------------------------------------
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
<mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
--
Jennifer M. Adams
IGES/COLA
4041 Powder Mill Road, Suite 302
Calverton, MD 20705
jma@xxxxxxxxxxxxx
------------------------------------------------------------------------
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
--
Jennifer M. Adams
IGES/COLA
4041 Powder Mill Road, Suite 302
Calverton, MD 20705
jma@xxxxxxxxxxxxx
