One problem that you are up against is that
there is no defined conversion between opendap(dap2)
and netcdf-4. There is a defined conversion between
dap2 and netcdf-3 (classic).
So when you make a opendap(dap2) url request,
the resulting data is downloaded and is accessible (read-only)
thru the netcdf-3 (classic) API an appears largely the same
as any other netcdf-3 file.
If you need netcdf-4, then you will need to use DAP4.
=Dennis Heimbigner
Unidata
On 5/27/2016 9:38 AM, Nathan Potter wrote:
Ashwin,
I believe that’s correct. To be honest I am not well versed on what can/cannot be done with the Java and C
implementations of the NetCDF library once the data have been subset and retrieved. But given that
nccopy/ncdump/etc. are “client” programs that are built on the NetCDF lib to achieve their
functionality I imagine that source code for these programs will provide examples of how to save the response as a
NetCDF file on your local disk. And I suspect that the Java NetCDF library must have similar
“client” tools that could be reviewed to find the analogous examples in Java.
Antonio Cofino also makes the salient point (in an email that is now orphaned
from this response) that you should be thoughtful and subset/retrieve the
coordinate/map/shared_dimension variables associated with the core data so that
all of the relevant information is captured in the resulting NetCDF result.
Without this the downstream software may face additional challenges.
You might write to the operators of www.esrl.noaa.gov (esrl.psd.data@xxxxxxxx)
and ask them if they have enabled or disabled the TDS feature that will allow
you to make the subset request in such a way as to prompt the TDS to return a
NetCDF file rather than a DAP2 or ASCII encoded response.
Sincerely,
Nathan
On May 27, 2016, at 7:30 AM, ashwinD12 . <winash12@xxxxxxxxx> wrote:
Nathan,
That is a very important point you bring up. If I want to write my
own Java client for instance I have to convert the OpenDAP 2 response into a
netcdf 4 file on my disk. It is very likely that the THREDDS server does not
serve netcdf files but I can get the response which is OpenDAP and convert it
into a netcdf 4 file.
Correct ?
Regards,
Ashwin.
On Fri, May 27, 2016 at 7:53 PM, Nathan Potter <ndp@xxxxxxxxxxx
<mailto:ndp@xxxxxxxxxxx>> wrote:
Ashwin,
I’m pleased! I want to clarify what’s happen though - The data is held in a
NetCDF file on the server. However, what is transported is NOT a NetCDF file, but rather the
response is encoded as a DAP2 metadata/data response which is in turn loaded into the (client
side) NetCDF library and then written to local disk as a NetCDF file. DAP2 is the transport
protocol/encoding.
Some servers (Hyrax being my favorite example ;) are able to encode the response as a NetCDF file at the
server end. This behavior can be triggered (in Hyrax) by replacing the original “.ascii”
component of the path URL with “.nc”. For example:
These ASCII request URLs:
http://test.opendap.org:8080/opendap/data/nc/fnoc1.nc.ascii
<http://test.opendap.org:8080/opendap/data/nc/fnoc1.nc.ascii>
http://test.opendap.org:8080/opendap/data/hdf5/grid_1_2d.h5.ascii
<http://test.opendap.org:8080/opendap/data/hdf5/grid_1_2d.h5.ascii>
Become NetCDF request URLs:
http://test.opendap.org:8080/opendap/data/nc/fnoc1.nc.nc
<http://test.opendap.org:8080/opendap/data/nc/fnoc1.nc.nc>
http://test.opendap.org:8080/opendap/data/hdf5/grid_1_2d.h5.nc
<http://test.opendap.org:8080/opendap/data/hdf5/grid_1_2d.h5.nc>
I believe that the TDS (which is the server software in play in your example)
also supports a similar feature, albeit through a somewhat different URL
scheme. And I think that earlier in the thread someone mentioned that this
feature is disabled on the server in question.
Cheers,
Nathan
On May 27, 2016, at 7:05 AM, ashwinD12 . <winash12@xxxxxxxxx
<mailto:winash12@xxxxxxxxx>> wrote:
Nathan,
Beautiful ! That is exactly what I am looking for. I can come up
with my own version of nccopy in Java or Python if I so wish. So that means
DAP2 can download netCDF files using the PSD THREDDS server and I can pass
OpeNDAP constraint expressions.
Regards,
Ashwin.
On Fri, May 27, 2016 at 7:17 PM, Nathan Potter <ndp@xxxxxxxxxxx
<mailto:ndp@xxxxxxxxxxx>> wrote:
Ashwin,
The NetCDF library may be useful for your desired outcome. Because the NetCDF library supports access via
the DAP2 protocol it is a natural for your probelm. I was able to use the NetCDF command line tools to
accomplish your task. Note that be dropping the “.ascii” suffix from the URL path you return
to the core “DAP2 URL” which is an important step in the following example.
Retrieving the data subset and saving it as a NetCDF-4 file:
./nccopy -4
"http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[480:603][2][20:34][26:40]
<http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc?air[480:603][2][20:34][26:40]>”
foo.nc <http://foo.nc/>
Which was easily read by ncdump:
./ncdump -h foo.nc <http://foo.nc/>
netcdf foo {
dimensions:
time = UNLIMITED ; // (124 currently)
lat = 15 ;
level = 1 ;
lon = 15 ;
variables:
short air(time, level, lat, lon) ;
air:long_name = "6-hourly Air Temperature on Pressure Levels" ;
air:valid_range = -32765s, -10260s ;
air:unpacked_valid_range = 137.5f, 362.5f ;
air:actual_range = 180.6f, 325.8f ;
air:units = "degK" ;
air:add_offset = 465.15f ;
air:scale_factor = 0.01f ;
air:missing_value = 32766s ;
air:_FillValue = -32767s ;
air:precision = 2s ;
air:least_significant_digit = 1s ;
air:GRIB_id = 11s ;
air:GRIB_name = "TMP" ;
air:var_desc = "Air temperature" ;
air:dataset = "NCEP/DOE AMIP-II Reanalysis (Reanalysis-2)" ;
air:level_desc = "Pressure Levels" ;
air:statistic = "Individual Obs" ;
air:parent_stat = "Other" ;
air:standard_name = "air_temperature" ;
// global attributes:
:Conventions = "CF-1.0" ;
:title = "4x Daily NCEP/DOE Reanalysis 2" ;
:history = "created 2015/02 by NOAA/ESRL/PSD" ;
:comments = "Data is from \nNCEP/DOE AMIP-II Reanalysis
(Reanalysis-2)\n(4x/day). It consists of most variables interpolated to\npressure
surfaces from model (sigma) surfaces." ;
:platform = "Model" ;
:source = "NCEP/DOE AMIP-II Reanalysis (Reanalysis-2) Model" ;
:institution = "National Centers for Environmental Prediction" ;
:References =
"http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalysis2.html
<http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalysis2.html>" ;
:dataset_title = "NCEP-DOE AMIP-II Reanalysis" ;
:DODS_EXTRA.Unlimited_Dimension = "time" ;
}
I am sure that there are several other ways to achieve a similar result, but
this one does seem simple and amenable to programatic (well, at least script
level) use.
I believe that both the NetCDF C and Java APIs will also happily detect that a
supplied file name is a URL and then dereference and retrieve data from it.
Does this in any way address your problem?
Regards,
Nathan
On May 27, 2016, at 2:13 AM, ashwinD12 . <winash12@xxxxxxxxx
<mailto:winash12@xxxxxxxxx>> wrote:
Regarding the specifics of what you have written in relation to the programming
language of choice
1) If I use java netcdf(the latest version) I can use the OPeNDAP URL that James
explained to me and send it to the URL -
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.ascii?air[480:603][2][20:34][26:40]
<http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc.ascii?air[480:603][2][20:34][26:40]>
(as an example)
the data returned will be ascii but the java netcdf library will allow me to
generate a netcdf file containing that data precisely that can be written to
disk. Am I correct in this ?
2) The other option is to use the URL you gave me and generate the URL with the
required parameters and then use Python and pycurl and write out the netcdf
file to disk.
Your input is appreciated,
Regards,
Ashwin.
= = =
Nathan Potter ndp at opendap.org <http://opendap.org/>
OPeNDAP, Inc. +1.541.231.3317
= = =
Nathan Potter ndp at opendap.org <http://opendap.org/>
OPeNDAP, Inc. +1.541.231.3317
= = =
Nathan Potter ndp at opendap.org
OPeNDAP, Inc. +1.541.231.3317
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
