Hi Ashwin,
If you have Netcdf library, version above 4.1 and compile with opendap
capabilities you can read opendap URLs as like a local netcdf:
http://www.unidata.ucar.edu/software/netcdf/docs/dap_support.html
This means that you can use as "file"name the opendap URL, including the
constraint expressions.
For example, you can use ncdump to see the remote metadata:
$ ncdump -h
"http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc";
or you can use James's CE:
$ ncdump -h
"http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[480:603][2][20:34][26:40]";
(Important: I have removed the .ascii after .nc, this is required)
Or copy the remote data to a local netcdf file using nccopy:
$ nccopy
"http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[480:603][2][20:34][26:40]";
air.nc
but the auxiliary coordinates are missing, therefore you also need to
"copy them"
$ nccopy
"http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[480:603][2][20:34][26:40],time[480:603],level[2],lat[20:34],lon[26:40]";
air.nc
And you will have a local subset netcdf file:
$ ncdump -h air.nc
As I mention this depends on the netcdf library do you have, but if it's
the right one, if your Fortran, C program it's linked against it you
will able to opendap URLS as local files. This includes R and Python
packages using this library.
Netcdf-java has these features but many others interesting ones.
Regards
Antonio
--
Antonio S. Cofiño
Grupo de Meteorología de Santander
Dep. de Matemática Aplicada y
Ciencias de la Computación
Universidad de Cantabria
http://www.meteo.unican.es
El 27/05/2016 a las 11:13, ashwinD12 . escribió:
Hi Don,
Thanks for your detailed response. Please allow me to
confirm what you have written in my own words so that I can begin
programming at the earliest.
As an aside I would like to say my ultimate goal is to automate this
process i.e. the download of NOAA reanalysis data and the subsequent
uptake by the Fortran program will have NO human intervention
whatsoever. The subsetting and generation of the netCDF file will be
automated and it will be part of a server process run every day(same
subset will be generated four times a day). From my understanding of
what you have written this is certainly and clearly possible with what
you have written.
Regarding the specifics of what you have written in relation to the
programming language of choice
1) If I use java netcdf(the latest version) I can use the OPeNDAP URL
that James explained to me and send it to the URL -
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.ascii?air[480:603][2][20:34][26:40]
<http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.ascii?air[480:603][2][20:34][26:40]>
(as an example)
the data returned will be ascii but the java netcdf library will allow
me to generate a netcdf file containing that data precisely that can
be written to disk. Am I correct in this ?
2) The other option is to use the URL you gave me and generate the URL
with the required parameters and then use Python and pycurl and write
out the netcdf file to disk.
Your input is appreciated,
Regards,
Ashwin.
On Fri, May 27, 2016 at 8:16 AM, Don Murray <don.murray@xxxxxxxx
<mailto:don.murray@xxxxxxxx>> wrote:
Hi Ashwin-
On 5/26/16 7:12 PM, ashwinD12 . wrote:
Don/Ryan,
Thanks for your response. I suppose I was
not clear on
my question. All other doubts have been resolved except this one.
I am wanting to download a subset of the May 2014 file for air
temperature programmatically(either python or java) and I
would like to
store this file as a netcdf file on my disk that will be read
downstream
by a Fortran program. The URL Ryan gave me will give ASCII
file but I am
wanting netcdf binary format. Regarding the URL Don gave me
would it be
possible to construct an OpenDap constraint expression and
then send it
to the RAMADDA server ?
The subset URL I sent will return a netCDF file. You can replace
certain parameters in the url arguments for different levels and
regions and times, or replace the entry id with that of another
file on the server. You can use the pycurl module in Python to
retrieve the file using that URL. The subset from that URL is
attached.
You could also use the netcdf-java library to make the subsets
programmatically in Java, pointing to the OPeNDAP URL. You can do
the subsets in real world space (e.g. lat/lon, time (with a
stride), etc) and it will convert that to the appropriate index
space constraints.
You cannot use the constraint expressions with netcdf-java or
through the RAMADDA url (which in the end calls the netcdf-java
library). I'm not sure why you would need that when you can use a
higher level interface.
Don
This is the only outstanding question.
Regards,
Ashwin.
On Fri, May 27, 2016 at 12:57 AM, Don Murray (NOAA Affiliate)
<don.murray@xxxxxxxx <mailto:don.murray@xxxxxxxx>
<mailto:don.murray@xxxxxxxx <mailto:don.murray@xxxxxxxx>>> wrote:
Ashwin-
We do not expose the netCDF subset service (NCSS) on the
PSD THREDDS
server, but the same files can be accessed through the PSD
RAMADDA
server, starting from here:
http://www.esrl.noaa.gov/psd/repository/entry/show?entryid=e570c8f9-ec09-4e89-93b4-babd5651e7a9
If you drill down to the 2014 file, you'll get to here:
http://www.esrl.noaa.gov/psd/repository/entry/show?entryid=synth%3Ae570c8f9-ec09-4e89-93b4-babd5651e7a9%3AL25jZXAucmVhbmFseXNpczIvcHJlc3N1cmUvYWlyLjIwMTQubmM%3D
where you can download the entire file, or select the
"Subset Grid"
option on the right side of the page. There you can
subset by time,
level and region. (however, you can't pick specific
hours, just a
range in this interface).
For for month of May 2014, latitude between 5 N
and 40 N and longitude 65 E and 100 E for the pressure level
850 hPa, the URL would be:
http://www.esrl.noaa.gov/psd/repository/entry/show/air.2014_subset.nc?submit=Subset&output=data.gridsubset&entryid=synth%3Ae570c8f9-ec09-4e89-93b4-babd5651e7a9%3AL25jZXAucmVhbmFseXNpczIvcHJlc3N1cmUvYWlyLjIwMTQubmM%3D&hstride=1&area_north=40&area_west=65&area_east=100&area_south=5&calendar=gregorian&fromdate=2014-05-01%2000%3A00%3A00%20UTC&todate=2014-05-31%2018%3A00%3A00%20UTC&addlatlon=true&level=2&variable=air
(shown at the bottom of the Grid Subset form).
Don Murray
--
Don Murray
NOAA/ESRL/PSD and CU-CIRES
303-497-3596
http://www.esrl.noaa.gov/psd/people/don.murray/
On 5/26/16 12:32 PM, Ryan May wrote:
Regarding #2, the URL from thredds to just download
the file
would be:
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.ascii?air[480:603][2][20:34][26:40]
You can find that in the catalog here:
http://www.esrl.noaa.gov/psd/thredds/catalog/Datasets/ncep.reanalysis2/pressure/catalog.html?dataset=Datasets/ncep.reanalysis2/pressure/air.2014.nc
(The HTTPServer link.)
Ryan
On Thu, May 26, 2016 at 11:10 AM, ashwinD12 .
<winash12@xxxxxxxxx <mailto:winash12@xxxxxxxxx>
<mailto:winash12@xxxxxxxxx <mailto:winash12@xxxxxxxxx>>
<mailto:winash12@xxxxxxxxx <mailto:winash12@xxxxxxxxx>
<mailto:winash12@xxxxxxxxx <mailto:winash12@xxxxxxxxx>>>> wrote:
James,
What an OpenDap primer from the person
who invented
OpenDap ! Thank you very much indeed. I absorbed
all of that
information.
I had three more follow up questions(maybe related)
1) I presume this URL relates to air temperatures -
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.ascii?air[480:603][2][20:34][26:40]
Why are the values negative ?
2) I presume I can download this file as a netcdf
file in
Python or
Java ?
3) Finally is there a way in the constraint
expression that
I can
add what time of day I want(00,06,12,18) ?
Regards,
Ashwin.
On Thu, May 26, 2016 at 8:41 PM, James Gallagher
<jgallagher@xxxxxxxxxxx
<mailto:jgallagher@xxxxxxxxxxx> <mailto:jgallagher@xxxxxxxxxxx
<mailto:jgallagher@xxxxxxxxxxx>>
<mailto:jgallagher@xxxxxxxxxxx
<mailto:jgallagher@xxxxxxxxxxx> <mailto:jgallagher@xxxxxxxxxxx
<mailto:jgallagher@xxxxxxxxxxx>>>>
wrote:
On May 26, 2016 at 07:06:38, ashwinD12 .
(winash12@xxxxxxxxx <mailto:winash12@xxxxxxxxx>
<mailto:winash12@xxxxxxxxx <mailto:winash12@xxxxxxxxx>>
<mailto:winash12@xxxxxxxxx
<mailto:winash12@xxxxxxxxx> <mailto:winash12@xxxxxxxxx
<mailto:winash12@xxxxxxxxx>>>)
wrote:
Hello,
I am not sure whether I will
get any help for
asking such a basic question on OpenDAP
constraint
expression
but if this is not the forum for getting
clarifications on
OpenDAP constraint expression please let
me know the
appropriate forum.
I I have this URL
-
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.ascii?air[408:603][2][20:34][26:40]'
This is the output of a program that
fetches data from a
THREDDS server. I am wanting to replicate the
functionality by
coming up with that URL myself.
Can somebody explain to me what the values in
parenthesis are ?
This is the input I give
variable='air',level=850,
months.minmax=c(5,5),
years.minmax=c(2014,2014),
lat.southnorth=c(5, 40),
lon.westeast=c(65, 100),
I am wanting data for month of May 2014,
latitude
between 5 N
and 40 N and longitude 65 E and 100 E for the
pressure level
850 hPa.
How does that input get translated to the
above URL ?
Here’s how the URL breaks down: air.2014.nc
<http://air.2014.nc>
<http://air.2014.nc> .ascii ? air
[408:603 ] [2] [20:34] [26:40]
<http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.ascii?air[408:603][2][20:34][26:40]>
The dataset (which is a file in this case) is
‘air.2014.nc <http://air.2014.nc> <http://air.2014.nc>
<http://air.2014.nc>’. You’re asking the server to
subset that
dataset and return just the variable ‘air’ and to
translate the
result into ASCII (text). When the variable
‘air’ is
extracted
from the data set its a four-dimensional array and
you’re asking
to have those dimensions ‘sliced’ (or subset)
so that,
for the
first dimension, you see only elements 408 to
603, for the
second dim only element 2, for the third
elements 20 to
34 and
elements 26 to 40 for the fourth.
You can look at the dataset to see how your
request
(level=850,
…) translated into those array indices by
looking at the
datasets’s metadata. To do that, removed the
.ascii and
replace
it with .dds, .das and/or .info and don’t
include the query
string (the part after the ‘?’). Like this:
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.dds
Dataset {
Float32 level[level = 17];
Float32 lat[lat = 73];
Float32 lon[lon = 144];
Float64 time[time = 1460];
Grid {
ARRAY:
Int16 air[time = 1460][level = 17][lat
= 73][lon
= 144];
MAPS:
Float64 time[time = 1460];
Float32 level[level = 17];
Float32 lat[lat = 73];
Float32 lon[lon = 144];
} air;
}
Datasets/ncep.reanalysis2/pressure/air.2014.nc
<http://air.2014.nc>
<http://air.2014.nc> <http://air.2014.nc>;
One thing you’ll see is that ‘air’ is a Grid -
so it’s not
really a simple array, but rather a collection of
arrays: air,
time, level, lat and lon. The ‘air’ array
holds the data
while
the other four hold what could be described as the
independent
variables (or dimensions).
You can see more information about the
variables by
looking at
‘semantic metadata’ like this:
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.das
Attributes {
level {
String units "millibar";
Float32 actual_range 1000.0, 10.0;
String long_name "Level";
String positive "down";
…
Hope this helps,
James
Regards,
Ashwin.
_______________________________________________
NOTE: All exchanges posted to Unidata
maintained
email lists are
recorded in the Unidata inquiry tracking
system and
made publicly
available through the web. Users who post
to any of
the lists we
maintain are reminded to remove any personal
information that
they
do not want to be made public.
thredds mailing list
thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
<mailto:thredds@xxxxxxxxxxxxxxxx
<mailto:thredds@xxxxxxxxxxxxxxxx>>
<mailto:thredds@xxxxxxxxxxxxxxxx
<mailto:thredds@xxxxxxxxxxxxxxxx>
<mailto:thredds@xxxxxxxxxxxxxxxx
<mailto:thredds@xxxxxxxxxxxxxxxx>>>
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
--
James Gallagher
jgallagher@xxxxxxxxxxx <mailto:jgallagher@xxxxxxxxxxx>
<mailto:jgallagher@xxxxxxxxxxx <mailto:jgallagher@xxxxxxxxxxx>>
<mailto:jgallagher@xxxxxxxxxxx
<mailto:jgallagher@xxxxxxxxxxx> <mailto:jgallagher@xxxxxxxxxxx
<mailto:jgallagher@xxxxxxxxxxx>>>
_______________________________________________
NOTE: All exchanges posted to Unidata maintained
email lists are
recorded in the Unidata inquiry tracking system
and made
publicly
available through the web. Users who post to any
of the
lists we
maintain are reminded to remove any personal
information
that they
do not want to be made public.
thredds mailing list
thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
<mailto:thredds@xxxxxxxxxxxxxxxx
<mailto:thredds@xxxxxxxxxxxxxxxx>>
<mailto:thredds@xxxxxxxxxxxxxxxx
<mailto:thredds@xxxxxxxxxxxxxxxx>
<mailto:thredds@xxxxxxxxxxxxxxxx
<mailto:thredds@xxxxxxxxxxxxxxxx>>>
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
--
Ryan May, Ph.D.
Software Engineer
UCAR/Unidata
Boulder, CO
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email
lists are
recorded in the Unidata inquiry tracking system and
made publicly
available through the web. Users who post to any of
the lists we
maintain are reminded to remove any personal
information that they
do not want to be made public.
thredds mailing list
thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
<mailto:thredds@xxxxxxxxxxxxxxxx
<mailto:thredds@xxxxxxxxxxxxxxxx>>
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
--
Don Murray
NOAA/ESRL/PSD and CIRES
303-497-3596
http://www.esrl.noaa.gov/psd/people/don.murray/
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
