Hi Don,
Thanks for your detailed response. Please allow me to confirm
what you have written in my own words so that I can begin programming at
the earliest.
As an aside I would like to say my ultimate goal is to automate this
process i.e. the download of NOAA reanalysis data and the subsequent uptake
by the Fortran program will have NO human intervention whatsoever. The
subsetting and generation of the netCDF file will be automated and it will
be part of a server process run every day(same subset will be generated
four times a day). From my understanding of what you have written this is
certainly and clearly possible with what you have written.
Regarding the specifics of what you have written in relation to the
programming language of choice
1) If I use java netcdf(the latest version) I can use the OPeNDAP URL that
James explained to me and send it to the URL -
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.ascii?air[480:603][2][
20:34][26:40] (as an example)
the data returned will be ascii but the java netcdf library will allow me
to generate a netcdf file containing that data precisely that can be
written to disk. Am I correct in this ?
2) The other option is to use the URL you gave me and generate the URL with
the required parameters and then use Python and pycurl and write out the
netcdf file to disk.
Your input is appreciated,
Regards,
Ashwin.
On Fri, May 27, 2016 at 8:16 AM, Don Murray <don.murray@xxxxxxxx> wrote:
> Hi Ashwin-
>
> On 5/26/16 7:12 PM, ashwinD12 . wrote:
>
>> Don/Ryan,
>> Thanks for your response. I suppose I was not clear on
>> my question. All other doubts have been resolved except this one.
>>
>> I am wanting to download a subset of the May 2014 file for air
>> temperature programmatically(either python or java) and I would like to
>> store this file as a netcdf file on my disk that will be read downstream
>> by a Fortran program. The URL Ryan gave me will give ASCII file but I am
>> wanting netcdf binary format. Regarding the URL Don gave me would it be
>> possible to construct an OpenDap constraint expression and then send it
>> to the RAMADDA server ?
>>
>
> The subset URL I sent will return a netCDF file. You can replace certain
> parameters in the url arguments for different levels and regions and times,
> or replace the entry id with that of another file on the server. You can
> use the pycurl module in Python to retrieve the file using that URL. The
> subset from that URL is attached.
>
> You could also use the netcdf-java library to make the subsets
> programmatically in Java, pointing to the OPeNDAP URL. You can do the
> subsets in real world space (e.g. lat/lon, time (with a stride), etc) and
> it will convert that to the appropriate index space constraints.
>
> You cannot use the constraint expressions with netcdf-java or through the
> RAMADDA url (which in the end calls the netcdf-java library). I'm not sure
> why you would need that when you can use a higher level interface.
>
> Don
>
> This is the only outstanding question.
>>
>> Regards,
>> Ashwin.
>>
>> On Fri, May 27, 2016 at 12:57 AM, Don Murray (NOAA Affiliate)
>> <don.murray@xxxxxxxx <mailto:don.murray@xxxxxxxx>> wrote:
>>
>> Ashwin-
>>
>> We do not expose the netCDF subset service (NCSS) on the PSD THREDDS
>> server, but the same files can be accessed through the PSD RAMADDA
>> server, starting from here:
>>
>>
>> http://www.esrl.noaa.gov/psd/repository/entry/show?entryid=e570c8f9-ec09-4e89-93b4-babd5651e7a9
>>
>> If you drill down to the 2014 file, you'll get to here:
>>
>>
>> http://www.esrl.noaa.gov/psd/repository/entry/show?entryid=synth%3Ae570c8f9-ec09-4e89-93b4-babd5651e7a9%3AL25jZXAucmVhbmFseXNpczIvcHJlc3N1cmUvYWlyLjIwMTQubmM%3D
>>
>> where you can download the entire file, or select the "Subset Grid"
>> option on the right side of the page. There you can subset by time,
>> level and region. (however, you can't pick specific hours, just a
>> range in this interface).
>>
>> For for month of May 2014, latitude between 5 N
>> and 40 N and longitude 65 E and 100 E for the pressure level
>> 850 hPa, the URL would be:
>>
>>
>> http://www.esrl.noaa.gov/psd/repository/entry/show/air.2014_subset.nc?submit=Subset&output=data.gridsubset&entryid=synth%3Ae570c8f9-ec09-4e89-93b4-babd5651e7a9%3AL25jZXAucmVhbmFseXNpczIvcHJlc3N1cmUvYWlyLjIwMTQubmM%3D&hstride=1&area_north=40&area_west=65&area_east=100&area_south=5&calendar=gregorian&fromdate=2014-05-01%2000%3A00%3A00%20UTC&todate=2014-05-31%2018%3A00%3A00%20UTC&addlatlon=true&level=2&variable=air
>>
>> (shown at the bottom of the Grid Subset form).
>>
>> Don Murray
>> --
>> Don Murray
>> NOAA/ESRL/PSD and CU-CIRES
>> 303-497-3596
>> http://www.esrl.noaa.gov/psd/people/don.murray/
>>
>> On 5/26/16 12:32 PM, Ryan May wrote:
>>
>> Regarding #2, the URL from thredds to just download the file
>> would be:
>>
>>
>> http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.ascii?air[480:603][2][20:34][26:40]
>>
>> You can find that in the catalog here:
>>
>>
>> http://www.esrl.noaa.gov/psd/thredds/catalog/Datasets/ncep.reanalysis2/pressure/catalog.html?dataset=Datasets/ncep.reanalysis2/pressure/air.2014.nc
>>
>> (The HTTPServer link.)
>>
>> Ryan
>>
>>
>> On Thu, May 26, 2016 at 11:10 AM, ashwinD12 .
>> <winash12@xxxxxxxxx <mailto:winash12@xxxxxxxxx>
>> <mailto:winash12@xxxxxxxxx <mailto:winash12@xxxxxxxxx>>> wrote:
>>
>> James,
>> What an OpenDap primer from the person who invented
>> OpenDap ! Thank you very much indeed. I absorbed all of that
>> information.
>>
>> I had three more follow up questions(maybe related)
>> 1) I presume this URL relates to air temperatures -
>>
>>
>> http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.ascii?air[480:603][2][20:34][26:40]
>>
>> Why are the values negative ?
>>
>> 2) I presume I can download this file as a netcdf file in
>> Python or
>> Java ?
>>
>> 3) Finally is there a way in the constraint expression that
>> I can
>> add what time of day I want(00,06,12,18) ?
>>
>> Regards,
>> Ashwin.
>>
>> On Thu, May 26, 2016 at 8:41 PM, James Gallagher
>> <jgallagher@xxxxxxxxxxx <mailto:jgallagher@xxxxxxxxxxx>
>> <mailto:jgallagher@xxxxxxxxxxx <mailto:jgallagher@xxxxxxxxxxx>>>
>> wrote:
>>
>>
>>
>> On May 26, 2016 at 07:06:38, ashwinD12 .
>> (winash12@xxxxxxxxx <mailto:winash12@xxxxxxxxx>
>> <mailto:winash12@xxxxxxxxx <mailto:winash12@xxxxxxxxx>>)
>>
>> wrote:
>>
>> Hello,
>> I am not sure whether I will get any help
>> for
>> asking such a basic question on OpenDAP constraint
>> expression
>> but if this is not the forum for getting
>> clarifications on
>> OpenDAP constraint expression please let me know the
>> appropriate forum.
>>
>>
>> I I have this URL
>> -
>>
>> http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.ascii?air[408:603][2][20:34][26:40]
>> '
>>
>>
>>
>> This is the output of a program that fetches data
>> from a
>> THREDDS server. I am wanting to replicate the
>> functionality by
>> coming up with that URL myself.
>>
>> Can somebody explain to me what the values in
>> parenthesis are ?
>>
>> This is the input I give
>>
>> variable='air',level=850,
>> months.minmax=c(5,5), years.minmax=c(2014,2014),
>> lat.southnorth=c(5, 40), lon.westeast=c(65, 100),
>>
>> I am wanting data for month of May 2014, latitude
>> between 5 N
>> and 40 N and longitude 65 E and 100 E for the
>> pressure level
>> 850 hPa.
>>
>> How does that input get translated to the above URL ?
>>
>>
>> Here’s how the URL breaks down: air.2014.nc
>> <http://air.2014.nc> .ascii ? air
>>
>> [408:603 ] [2] [20:34] [26:40]
>>
>> <
>> http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.ascii?air[408:603][2][20:34][26:40]
>> >
>>
>> The dataset (which is a file in this case) is
>> ‘air.2014.nc <http://air.2014.nc>
>> <http://air.2014.nc>’. You’re asking the server to
>> subset that
>>
>> dataset and return just the variable ‘air’ and to
>> translate the
>> result into ASCII (text). When the variable ‘air’ is
>> extracted
>> from the data set its a four-dimensional array and
>> you’re asking
>> to have those dimensions ‘sliced’ (or subset) so that,
>> for the
>> first dimension, you see only elements 408 to 603, for the
>> second dim only element 2, for the third elements 20 to
>> 34 and
>> elements 26 to 40 for the fourth.
>>
>> You can look at the dataset to see how your request
>> (level=850,
>> …) translated into those array indices by looking at the
>> datasets’s metadata. To do that, removed the .ascii and
>> replace
>> it with .dds, .das and/or .info and don’t include the
>> query
>> string (the part after the ‘?’). Like this:
>>
>>
>>
>> http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.dds
>>
>> Dataset {
>> Float32 level[level = 17];
>> Float32 lat[lat = 73];
>> Float32 lon[lon = 144];
>> Float64 time[time = 1460];
>> Grid {
>> ARRAY:
>> Int16 air[time = 1460][level = 17][lat = 73][lon
>> = 144];
>> MAPS:
>> Float64 time[time = 1460];
>> Float32 level[level = 17];
>> Float32 lat[lat = 73];
>> Float32 lon[lon = 144];
>> } air;
>> } Datasets/ncep.reanalysis2/pressure/air.2014.nc
>> <http://air.2014.nc> <http://air.2014.nc>;
>>
>>
>>
>> One thing you’ll see is that ‘air’ is a Grid - so it’s not
>> really a simple array, but rather a collection of
>> arrays: air,
>> time, level, lat and lon. The ‘air’ array holds the data
>> while
>> the other four hold what could be described as the
>> independent
>> variables (or dimensions).
>>
>> You can see more information about the variables by
>> looking at
>> ‘semantic metadata’ like this:
>>
>>
>>
>> http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.das
>>
>> Attributes {
>> level {
>> String units "millibar";
>> Float32 actual_range 1000.0, 10.0;
>> String long_name "Level";
>> String positive "down";
>>
>> …
>>
>> Hope this helps,
>> James
>>
>>
>>
>>
>> Regards,
>> Ashwin.
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained
>> email lists are
>> recorded in the Unidata inquiry tracking system and
>> made publicly
>> available through the web. Users who post to any of
>> the lists we
>> maintain are reminded to remove any personal
>> information that
>> they
>> do not want to be made public.
>>
>>
>> thredds mailing list
>> thredds@xxxxxxxxxxxxxxxx
>> <mailto:thredds@xxxxxxxxxxxxxxxx>
>> <mailto:thredds@xxxxxxxxxxxxxxxx
>> <mailto:thredds@xxxxxxxxxxxxxxxx>>
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>>
>>
>> --
>> James Gallagher
>> jgallagher@xxxxxxxxxxx <mailto:jgallagher@xxxxxxxxxxx>
>> <mailto:jgallagher@xxxxxxxxxxx <mailto:jgallagher@xxxxxxxxxxx>>
>>
>>
>>
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists
>> are
>> recorded in the Unidata inquiry tracking system and made
>> publicly
>> available through the web. Users who post to any of the
>> lists we
>> maintain are reminded to remove any personal information
>> that they
>> do not want to be made public.
>>
>>
>> thredds mailing list
>> thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
>> <mailto:thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx
>> >>
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>>
>>
>>
>>
>> --
>> Ryan May, Ph.D.
>> Software Engineer
>> UCAR/Unidata
>> Boulder, CO
>>
>>
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web. Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>>
>>
>> thredds mailing list
>> thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>>
>>
>>
>>
> --
> Don Murray
> NOAA/ESRL/PSD and CIRES
> 303-497-3596
> http://www.esrl.noaa.gov/psd/people/don.murray/
>
