Hi Chris,
First off, let me just say that this is an absolutely fantastic bug report.
I wish I had better news for you, but the simple answer is that NetCDF-Java
doesn't support linked HDF files. Frankly, you're the first use that's even
mentioned them to us. Is there a particular reason that you prefer linked
files to normal files?
Regards,
Christian
On Tue, Jul 15, 2014 at 1:29 PM, Christopher Mueller <
cmueller@xxxxxxxxxxxxxx> wrote:
> *tl;dr* There appears to be a bug in NetCDF Java with respect to
> reading linked HDF4 files which results in data that is read from the
> linked file(s) to be erroneous.
> Resources
>
> - ToolsUI
> - HDFView
> - The files mentioned below can be retrieved directly from OceanColor
>
> <http://oceancolor.gsfc.nasa.gov/cgi/l3/A20021822013212.L3b_MC_RRS.main.bz2?sub=bin>
> (one
> at a time), or (for convenience) as one tar.gz file fromhere
>
> <https://drive.google.com/uc?id=0B6UT7Mn4GZQhMjdLNDBBMFE0TTA&export=download>
>
> Details
>
> I'm reading data from the Aqua MODIS L3 Binned products available from the
> NASA
> OceanColor <http://oceancolor.gsfc.nasa.gov/> website. It should be noted
> that these files are HDF4 (4.2.9 according to NetCDF Java - ncdump). Many
> of the products, such as chlorophyll, Particulate Inorganic Carbon, and Sea
> Surface Temperature, come as a single file. The NetCDF library reads these
> files without any difficulty.
>
> However, one of the datasets of interest is the Remote Sensing Reflectance
> data, which is NOT provided as a single file, but as a "main" file and a
> set of subordinate files which are read via the "main" file as needed (see
> here for more information
> <http://oceancolor.gsfc.nasa.gov/PRODUCTS/modis_binned.html>):
>
> - A20021822013212.L3b_MC_RRS.main
> - A20021822013212.L3b_MC_RRS.x00
> - A20021822013212.L3b_MC_RRS.x01
> - A20021822013212.L3b_MC_RRS.x02
> - A20021822013212.L3b_MC_RRS.x03
> - A20021822013212.L3b_MC_RRS.x04
> - A20021822013212.L3b_MC_RRS.x05
> - A20021822013212.L3b_MC_RRS.x06
> - A20021822013212.L3b_MC_RRS.x07
> - A20021822013212.L3b_MC_RRS.x08
> - A20021822013212.L3b_MC_RRS.x09
> - A20021822013212.L3b_MC_RRS.x10
> - A20021822013212.L3b_MC_RRS.x11
>
> NetCDF Java (via ToolsUI) loads the .main file without issue, and permits
> reading of data variables (i.e. Rrs_412) without raising any errors.
> However, the data returned is not accurate. Below is a comparison of the
> data returned by ToolsUI and the same data returned by HDFView (which uses
> the HDF-java JNI <http://www.hdfgroup.org/products/java/JNI/> library):
>
> Retrieving the first 10 values for variable "Rrs_412"
> HDFView
>
> Screen Capture <http://cl.ly/WZnD>
>
> Opening the .main file in HDFView and looking at the Rrs_412 dataset gives
> a very different set of data:
>
> 0.0055423053, 0.0106070135, 0.006894292, -0.0040368317, -0.0020879991,
> -0.0020279996, 0.009794002, 0.011879213, 0.010874448, 0.012330733
>
> ToolsUI
>
> Screen Capture <http://cl.ly/WZMW>
>
> Opening the .main file and performing an *Ncdump Data* of variable:
> "Level-3_Binned_Data/Rrs_412(0:10:1).Rrs_412_sum"
>
> Returns:
>
> float Rrs_412_sum;
>
> data:
>
> {1.86057E-40, 9.403955E-38, 6.4099753E-10, 2.6076459E-9, 1.0297978E21,
> 5.6431478E-11, 0.0, -2.9699963E36, 4.59183E-40, 3.67343E-40, 2.60329423E11}
>
> Also, in ToolsUI, *all of the other data variables* (i.e. angstrom,
> aot_869 & Rrs_*) all display very very similar (most are identical) values
> as the Rrs_412. This is not the case for HDFView.
>
> Incidentally, reading the data via OceanColor's SeaDas
> <http://seadas.gsfc.nasa.gov/> application (which uses NetCDF Java under
> the hood) results in the same data as ToolsUI.
> Wrap-up
>
> The evidence above appears to indicate that there is a bug in NetCDF Java
> related to linked HDF files which results in incorrect data reads from
> linked files.
>
> Does anyone have any idea:
>
> *a)* what could be causing the issue?
> *b)* how could it be addressed?
>
>
>
> Thanks in advance,
> Chris
>
> _______________________________________________
> netcdf-java mailing list
> netcdf-java@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
