I've now uninstalled NetCDF 4.2.1 (as provided by the "Geo" repository on
openSUSE 12.1) and installed NetCDF 4.2.1.1, leaving Lynton's C++ libraries
intact.
Here is the output from *nc-config --all*
This netCDF 4.2.1.1 has been built with the following features:
--cc -> gcc
--cflags -> -I/usr/include
--libs -> -L/usr/lib -lnetcdf
--has-c++ -> no
--cxx ->
--has-c++4 -> yes
--cxx4 -> g++
--fc ->
--fflags ->
--flibs ->
--has-f90 -> no
--has-dap -> yes
--has-nc2 -> yes
--has-nc4 -> yes
--has-hdf5 -> yes
--has-hdf4 -> no
--has-pnetcdf-> no
--has-szlib ->
--prefix -> /usr
--includedir-> /usr/include
--version -> netCDF 4.2.1.1
...and from Dennis' request, the output from
ncdump '[log][show=fetch]
http://goldsmr2.sci.gsfc.nasa.gov/opendap/hyrax/MERRA/MAT1NXSLV.5.2.0/1990/01/MERRA100.prod.assim.tavg1_2d_slv_Nx.19900101.hdf?U50M_EOSGRID_Data_Fields[0:23][282:282][441:441]
'
Log:fetch:
http://goldsmr2.sci.gsfc.nasa.gov/opendap/hyrax/MERRA/MAT1NXSLV.5.2.0/1990/01/MERRA100.prod.assim.tavg1_2d_slv_Nx.19900101.hdf.dds
Log:fetch complete: 6.015 secs
Log:fetch:
http://goldsmr2.sci.gsfc.nasa.gov/opendap/hyrax/MERRA/MAT1NXSLV.5.2.0/1990/01/MERRA100.prod.assim.tavg1_2d_slv_Nx.19900101.hdf.das
Log:fetch complete: 0.538 secs
Log:fetch:
http://goldsmr2.sci.gsfc.nasa.gov/opendap/hyrax/MERRA/MAT1NXSLV.5.2.0/1990/01/MERRA100.prod.assim.tavg1_2d_slv_Nx.19900101.hdf.dds?U50M_EOSGRID_Data_Fields[0:23][282][441]
Log:fetch complete: 0.313 secs
Log:prefetch: U50M_EOSGRID_Data_Fields=24
Log:prefetch: U50M_EOSGRID_Data_Fields
Log:prefetch.final: U50M_EOSGRID_Data_Fields*[0:23/24][0:360/361][0:539/540]
Log:fetch:
http://goldsmr2.sci.gsfc.nasa.gov/opendap/hyrax/MERRA/MAT1NXSLV.5.2.0/1990/01/MERRA100.prod.assim.tavg1_2d_slv_Nx.19900101.hdf.dods?U50M_EOSGRID_Data_Fields
Log:fetch complete: 78.109 secs
Log:prefetch.complete
Log:prefetch.vars: U50M_EOSGRID_Data_Fields
netcdf MERRA100.prod.assim.tavg1_2d_slv_Nx.19900101 {
dimensions:
TIME_EOSGRID = 24 ;
XDim_EOSGRID = 1 ;
YDim_EOSGRID = 1 ;
variables:
float U50M_EOSGRID_Data_Fields(TIME_EOSGRID, YDim_EOSGRID,
XDim_EOSGRID) ;
U50M_EOSGRID_Data_Fields:_FillValue = 1.e+15f ;
U50M_EOSGRID_Data_Fields:long_name = "Eastward wind at 50 m
above surface" ;
U50M_EOSGRID_Data_Fields:standard_name =
"eastward_wind_at_50_meters" ;
U50M_EOSGRID_Data_Fields:units = "m/s" ;
U50M_EOSGRID_Data_Fields:scale_factor = 1.f ;
U50M_EOSGRID_Data_Fields:add_offset = 0.f ;
U50M_EOSGRID_Data_Fields:missing_value = 1.e+15f ;
U50M_EOSGRID_Data_Fields:fmissing_value = 1.e+15f ;
U50M_EOSGRID_Data_Fields:vmin = -1.e+30f ;
U50M_EOSGRID_Data_Fields:vmax = 1.e+30f ;
U50M_EOSGRID_Data_Fields:valid_range = -1.e+30f, 1.e+30f ;
U50M_EOSGRID_Data_Fields:coordinates = "TIME_EOSGRID
YDim_EOSGRID XDim_EOSGRID" ;
// global attributes:
:HDFEOSVersion = "HDFEOS_V2.14" ;
:missing_value = 1.e+15f ;
:Conventions = "CF-1.0" ;
:title = "MERRA reanalysis. GEOS-5.2.0" ;
:history = "File written by CFIO" ;
:institution = "Global Modeling and Assimilation Office,
NASA Goddard Space Flight Center, Greenbelt, MD 20771" ;
:source = "Global Modeling and Assimilation Office.
GEOSops_5_2_0" ;
:references = "http://gmao.gsfc.nasa.gov/research/merra/"; ;
:comment = "GEOS-5.2.0" ;
:contact = "http://gmao.gsfc.nasa.gov/"; ;
data:
U50M_EOSGRID_Data_Fields =
5.483779,
4.986389,
4.076741,
3.479807,
3.125561,
2.818876,
2.536896,
2.388737,
2.501732,
2.531001,
2.440533,
2.245501,
1.99101,
1.798543,
1.793866,
1.694912,
1.561274,
1.471124,
1.408249,
1.449105,
1.588835,
1.835351,
2.13877,
2.48236 ;
}
I notice no immediate change in the bandwidth usage, using the new 4.2.1.1
libraries.
The other suggestion (from Chris, and Benno on the OPeNDAP forums) was that
the subsetting should not be handled by the DAP URL, but from within the
functionality of the C++ libraries I'm using.
Perhaps someone can point me in the right direction: I've explored quite
thoroughly the C++ interface
guide<http://www.unidata.ucar.edu/software/netcdf/docs/cxx4/index.html>
and
associated
examples<http://www.unidata.ucar.edu/software/netcdf/docs/cxx4/examples.html>,
but have struggled to gain a foothold on file subsetting.
Any help is appreciated!
On Wed, Feb 20, 2013 at 12:52 PM, Taylor Binnington
<tbinnington@xxxxxxxxx>wrote:
> Dennis,
> It doesn't look like there is a version 4.2.1.1, at least not for the
> NetCDF-4 C++ libraries, according to this:
> http://www.unidata.ucar.edu/downloads/netcdf/netcdf-cxx/index.jsp
>
> It hasn't been updated since I installed it over a month ago.
> Taylor
>
>
>
> On Tue, Feb 19, 2013 at 2:20 PM, Taylor Binnington
> <tbinnington@xxxxxxxxx>wrote:
>
>> Thank you both.
>>
>> First, in reply to Christopher's suggestion:
>>
>> I've explored the documentation
>> here<http://www.unidata.ucar.edu/software/netcdf/docs/cxx4/index.html> as
>> well as the
>> test_var.cpp<http://www.unidata.ucar.edu/software/netcdf/docs/cxx4/test_var_8cpp-example.html#_a10>
>> example
>> program, in addition to a few other sources. I'm finding the documentation
>> quite opaque, regarding learning to extract only certain indices or an
>> array via the NetCDF API. However, one thing I've gleaned is that
>> subsetting parameters are specified as a NetCDF Group; however, this data
>> structure is not found in MERRA files, since Groups are only supported by
>> NetCDF-4 files.
>>
>> Second, in reply to Dennis' request. The output from
>>
>> ncdump '[log][show=fetch]
>> http://goldsmr2.sci.gsfc.nasa.gov/opendap/hyrax/MERRA/MAT1NXSLV.5.2.0/1990/01/MERRA100.prod.assim.tavg1_2d_slv_Nx.19900101.hdf?U50M_EOSGRID_Data_Fields[0:23][282:282][441:441]
>> '
>>
>> is the following:
>>
>> Log:fetch:
>> http://goldsmr2.sci.gsfc.nasa.gov/opendap/hyrax/MERRA/MAT1NXSLV.5.2.0/1990/01/MERRA100.prod.assim.tavg1_2d_slv_Nx.19900101.hdf.dds
>>
>> Log:fetch complete: 2.906 secs
>> Log:fetch:
>> http://goldsmr2.sci.gsfc.nasa.gov/opendap/hyrax/MERRA/MAT1NXSLV.5.2.0/1990/01/MERRA100.prod.assim.tavg1_2d_slv_Nx.19900101.hdf.das
>>
>> Log:fetch complete: 0.443 secs
>> Log:fetch:
>> http://goldsmr2.sci.gsfc.nasa.gov/opendap/hyrax/MERRA/MAT1NXSLV.5.2.0/1990/01/MERRA100.prod.assim.tavg1_2d_slv_Nx.19900101.hdf.dds?U50M_EOSGRID_Data_Fields[0:23][282][441]
>>
>> Log:fetch complete: 0.228 secs
>> Log:prefetch: U50M_EOSGRID_Data_Fields=24
>> Log:prefetch: U50M_EOSGRID_Data_Fields
>> Log:prefetch.final:
>> U50M_EOSGRID_Data_Fields*[0:23/24][0:360/361][0:539/540]
>> Log:fetch:
>> http://goldsmr2.sci.gsfc.nasa.gov/opendap/hyrax/MERRA/MAT1NXSLV.5.2.0/1990/01/MERRA100.prod.assim.tavg1_2d_slv_Nx.19900101.hdf.dods?U50M_EOSGRID_Data_Fields
>>
>> Log:fetch complete: 58.994 secs
>> Log:prefetch.complete
>> Log:prefetch.vars: U50M_EOSGRID_Data_Fields
>>
>> netcdf MERRA100.prod.assim.tavg1_2d_slv_Nx.19900101 {
>> dimensions:
>> TIME_EOSGRID = 24 ;
>> XDim_EOSGRID = 1 ;
>> YDim_EOSGRID = 1 ;
>> variables:
>> float U50M_EOSGRID_Data_Fields(TIME_EOSGRID, YDim_EOSGRID,
>> XDim_EOSGRID) ;
>> U50M_EOSGRID_Data_Fields:_FillValue = 1.e+15f ;
>> U50M_EOSGRID_Data_Fields:long_name = "Eastward wind at 50
>> m above surface" ;
>> U50M_EOSGRID_Data_Fields:standard_name =
>> "eastward_wind_at_50_meters" ;
>> U50M_EOSGRID_Data_Fields:units = "m/s" ;
>> U50M_EOSGRID_Data_Fields:scale_factor = 1.f ;
>> U50M_EOSGRID_Data_Fields:add_offset = 0.f ;
>> U50M_EOSGRID_Data_Fields:missing_value = 1.e+15f ;
>> U50M_EOSGRID_Data_Fields:fmissing_value = 1.e+15f ;
>> U50M_EOSGRID_Data_Fields:vmin = -1.e+30f ;
>> U50M_EOSGRID_Data_Fields:vmax = 1.e+30f ;
>> U50M_EOSGRID_Data_Fields:valid_range = -1.e+30f, 1.e+30f ;
>> U50M_EOSGRID_Data_Fields:coordinates = "TIME_EOSGRID
>> YDim_EOSGRID XDim_EOSGRID" ;
>>
>> // global attributes:
>> :HDFEOSVersion = "HDFEOS_V2.14" ;
>> :missing_value = 1.e+15f ;
>> :Conventions = "CF-1.0" ;
>> :title = "MERRA reanalysis. GEOS-5.2.0" ;
>> :history = "File written by CFIO" ;
>> :institution = "Global Modeling and Assimilation Office,
>> NASA Goddard Space Flight Center, Greenbelt, MD 20771" ;
>> :source = "Global Modeling and Assimilation Office.
>> GEOSops_5_2_0" ;
>> :references = "http://gmao.gsfc.nasa.gov/research/merra/";
>> ;
>> :comment = "GEOS-5.2.0" ;
>> :contact = "http://gmao.gsfc.nasa.gov/"; ;
>> data:
>>
>> U50M_EOSGRID_Data_Fields =
>> 5.483779,
>> 4.986389,
>> 4.076741,
>> 3.479807,
>> 3.125561,
>> 2.818876,
>> 2.536896,
>> 2.388737,
>> 2.501732,
>> 2.531001,
>> 2.440533,
>> 2.245501,
>> 1.99101,
>> 1.798543,
>> 1.793866,
>> 1.694912,
>> 1.561274,
>> 1.471124,
>> 1.408249,
>> 1.449105,
>> 1.588835,
>> 1.835351,
>> 2.13877,
>> 2.48236 ;
>> }
>>
>>
>>
>> On Tue, Feb 19, 2013 at 11:00 AM, Lynnes, Christopher S. (GSFC-6102) <
>> christopher.s.lynnes@xxxxxxxx> wrote:
>>
>>> On Feb 19, 2013, at 10:48 AM, Taylor Binnington <tbinnington@xxxxxxxxx>
>>> wrote:
>>>
>>> > Thank you Dennis.
>>> > I'm using version 4.2.1, netcdf-cxx4 by Lynton Appel, on openSUSE
>>> 12.1, so I think it's fairly up-to-date. The problem is not that I am
>>> unable to open the file (I, too, can print out the 24 values), but that
>>> doing so apparently uses ~19 mb of bandwidth, according to the iftop
>>> monitoring utility. This seems quite excessive.
>>> > Please let me know if there is specific information that I should
>>> provide here. Much appreciated,
>>> > Taylor
>>>
>>> Taylor,
>>> I think Benno might have put his finger on the issue over on the
>>> opendap-tech mailing list. That is, instead of passing the fully
>>> constrained URL to a read function, open the URL without the constraints
>>> and then use one of the methods that allows you to specify the start and
>>> length (or corner points) of the segment you want to retrieve. I know they
>>> must be there somewhere, because the C API certainly has it.
>>> >
>>> >
>>> > On Mon, Feb 18, 2013 at 5:57 PM, Dennis Heimbigner <
>>> dmh@xxxxxxxxxxxxxxxx> wrote:
>>> >
>>> > The opendap code in the netcdf-C libraries
>>> > (used by the C++ library) will in fact
>>> > properly handle the constraint you specified.
>>> > I just ran it and as expected, it is only pulling
>>> > in the 24 values you requested (took some 5 seconds to
>>> > obtain and print out the data).
>>> >
>>> > One possibility is that you are using a rather old version
>>> > of the netcdf-C library.
>>> >
>>> > =Dennis Heimbigner
>>> > Unidata
>>> >
>>> > Taylor Binnington wrote:
>>> > Hello,
>>> >
>>> > I have an 'understanding' type of question. I have omitted details of
>>> my
>>> > OS, NetCDF installation, and a minimal example since I do not think
>>> they
>>> > are important for my question, but I can provide these details if they
>>> are
>>> > important.
>>> >
>>> > When I attempt to open a MERRA file using the NetCDF C++ libraries,
>>> with a
>>> > line like the following, in my code:
>>> >
>>> > NcFile datafile("
>>> >
>>> http://goldsmr2.sci.gsfc.nasa.gov/opendap/hyrax/MERRA/MAT1NXSLV.5.2.0/1990/01/MERRA100.prod.assim.tavg1_2d_slv_Nx.19900101.hdf?U50M_EOSGRID_Data_Fields[0:23][282:282][441:441]
>>> > ",NcFile::read);
>>> >
>>> > ...I notice that quite a lot (nearly 20 MB) is being transferred from
>>> Hyrax
>>> > to my PC. However, all that I am interested in are the actual array
>>> values
>>> > (24 separate 32-bit real numbers) of the U50M_EOSGRID_Data_Fields. It
>>> has
>>> > been suggested that the entire array (of size 24*361*540 32-bit reals)
>>> is
>>> > being transferred by NetCDF before any subsetting takes place, is this
>>> > true? If so, is it then not possible to use OPeNDAP URLs to subset
>>> > on-the-fly through the NetCDF libraries?
>>> >
>>> > Thank you for your help!
>>> > Taylor
>>> >
>>> >
>>> >
>>> >
>>> ------------------------------------------------------------------------
>>> >
>>> > _______________________________________________
>>> > netcdfgroup mailing list
>>> > netcdfgroup@xxxxxxxxxxxxxxxx
>>> > For list information or to unsubscribe, visit:
>>> http://www.unidata.ucar.edu/mailing_lists/
>>> >
>>> >
>>> >
>>> > --
>>> > Taylor Binnington
>>> > e. tbinnington@xxxxxxxxx
>>> > c. 647 926 4144
>>> > _______________________________________________
>>> > netcdfgroup mailing list
>>> > netcdfgroup@xxxxxxxxxxxxxxxx
>>> > For list information or to unsubscribe, visit:
>>> http://www.unidata.ucar.edu/mailing_lists/
>>>
>>> --
>>> Dr. Christopher Lynnes, NASA/GSFC, ph: 301-614-5185
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Taylor Binnington
>> e. tbinnington@xxxxxxxxx
>>
>>
>
>
> --
> Taylor Binnington
> e. tbinnington@xxxxxxxxx
>
>
--
Taylor Binnington
e. tbinnington@xxxxxxxxx
