Re: [netcdfgroup] [Hdf-forum] Detecting netCDF versus HDF5

Isn't netCDF utility "ncdump -k" to tell whether an HDF file has
all the required components understood by netcdf?

Wei-keng

On Mar 3, 2016, at 11:39 AM, Ed Hartnett wrote:

> This is a super interesting conversation for me! ;-)
> 
> I must admit that when putting netCDF-4 together I never considered that 
> someone might want to tell the difference between a "native" HDF5 file and a 
> netCDF-4/HDF5 file. Well, you can't think of everything.
> 
> I agree that it is not possible to canonically tell the difference. The 
> netCDF-4 API does use some special attributes to track named dimensions, and 
> to tell whether classic mode should be enforced. But it can easily produce 
> files without any named dimensions, etc. So I don't think there is any easy 
> way to tell.
> 
> However, you can attempt to open a HDF5 file with nc_create, and, if it 
> fails, use HDF5. (It should very rarely fail.)
> 
> Ed
> 
> 
> 
> On Wed, Mar 2, 2016 at 7:07 PM, Miller, Mark C. <miller86@xxxxxxxx> wrote:
> I like John's suggestion here.
> 
> But, any code you add to any applications now will work *only* for files that 
> were produced post-adoption of this convention.
> 
> There are probably a bazillion files out there at this point that don't 
> follow that convention and you probably still want your applications to be 
> able to read them.
> 
> In VisIt, we support >140 format readers. Over 20 of those are different 
> variants of HDF5 files (H5part, Xdmf, Pixie, Silo, Samrai, netCDF, Flash, 
> Enzo, Chombo, etc., etc.) When opening a file, how does VisIt figure out 
> which plugin to use? In particular, how do we avoid one poorly written reader 
> plugin (which may be the wrong one for a given file) from preventing the 
> correct one from being found. Its kinda a hard problem.
> 
> Some of our discussion is captured here. . .
> 
> http://www.visitusers.org/index.php?title=Database_Format_Detection
> 
> Mark
> 
> 
> From: Hdf-forum <hdf-forum-bounces@xxxxxxxxxxxxxxxxxx> on behalf of John 
> Shalf <jshalf@xxxxxxx>
> Reply-To: HDF Users Discussion List <hdf-forum@xxxxxxxxxxxxxxxxxx>
> Date: Wednesday, March 2, 2016 1:02 PM
> To: HDF Users Discussion List <hdf-forum@xxxxxxxxxxxxxxxxxx>
> Cc: "netcdfgroup@xxxxxxxxxxxxxxxx" <netcdfgroup@xxxxxxxxxxxxxxxx>, Ward 
> Fisher <wfisher@xxxxxxxx>
> Subject: Re: [Hdf-forum] Detecting netCDF versus HDF5
> 
> Perhaps NetCDF (and other higher-level APIs that are built on top of HDF5) 
> should include an attribute attached to the root group that identifies the 
> name and version of the API that created the file?  (adopt this as a 
> convention)
> 
> -john
> 
> On Mar 2, 2016, at 12:55 PM, Pedro Vicente <pedro.vicente@xxxxxxxxxxxxxxxxxx> 
> wrote:
> Hi Ward
> As you know, Data Explorer is going to be a general purpose data reader for 
> many formats, including HDF5 and netCDF.
> Here
> http://www.space-research.org/
> Regarding the handling of both HDF5 and netCDF, it seems there is a potential 
> issue, which is, how to tell if any HDF5 file was saved by the HDF5 API or by 
> the netCDF API?
> It seems to me that this is not possible. Is this correct?
> netCDF uses an internal function NC_check_file_type to examine the first few 
> bytes of a file, and for example for any HDF5 file the test is
> /* Look at the magic number */
>    /* Ignore the first byte for HDF */
>    if(magic[1] == 'H' && magic[2] == 'D' && magic[3] == 'F') {
>      *filetype = FT_HDF;
>      *version = 5;
> The problem is that this test works for any HDF5 file and for any netCDF 
> file, which makes it impossible to tell which is which.
> Which makes it impossible for any general purpose data reader to decide to 
> use the netCDF API or the HDF5 API.
> I have a possible solution for this , but before going any further, I would 
> just like to confirm that
> 1)      Is indeed not possible
> 2)      See if you have a solid workaround for this, excluding the dumb ones, 
> for example deciding on a extension .nc or .h5, or traversing the HDF5 file 
> to see if it's non netCDF conforming one. Yes, to further complicate things, 
> it is possible that the above test says OK for a HDF5 file, but then the read 
> by the netCDF API fails because the file is a HDF5 non netCDF conformant
> Thanks
> ----------------------
> Pedro Vicente
> pedro.vicente@xxxxxxxxxxxxxxxxxx
> http://www.space-research.org/
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@xxxxxxxxxxxxxxxxxx
> http://secure-web.cisco.com/1r-EJFFfg6rWlpQsvXstBNTjaHQaKT_NkYRN0Jj_f-Z3EK0-hs6IbYc8XUBRyPsH3mU3CS0iiY7_qnchCA0QxNzQt270d_2HikCwpAWFmuHdacin62eaODutktDSOULIJmVbVYqFVSKWPzoX7kdP0yN9wIzSFxZfTwfhU8ebsN409xRg1PsW_8cvNiWzxDNm9wv9yBf9yK6nkEm-bOx2S0kBLbg9WfIChWzZrkpE3AHU9I-c2ZRH_IN-UF4g_g0_Dh4qE1VETs7tZTfKd1ox1MtBmeyKf7EKUCd3ezR9EbI5tK4hCU5qW4v5WWOxOrD17e8yCVmob27xz84Lr3bCK5wIQdH5VzFRTtyaAhudpt9E/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5
> 
> 
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@xxxxxxxxxxxxxxxxxx
> http://secure-web.cisco.com/1r-EJFFfg6rWlpQsvXstBNTjaHQaKT_NkYRN0Jj_f-Z3EK0-hs6IbYc8XUBRyPsH3mU3CS0iiY7_qnchCA0QxNzQt270d_2HikCwpAWFmuHdacin62eaODutktDSOULIJmVbVYqFVSKWPzoX7kdP0yN9wIzSFxZfTwfhU8ebsN409xRg1PsW_8cvNiWzxDNm9wv9yBf9yK6nkEm-bOx2S0kBLbg9WfIChWzZrkpE3AHU9I-c2ZRH_IN-UF4g_g0_Dh4qE1VETs7tZTfKd1ox1MtBmeyKf7EKUCd3ezR9EbI5tK4hCU5qW4v5WWOxOrD17e8yCVmob27xz84Lr3bCK5wIQdH5VzFRTtyaAhudpt9E/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5
> 
> 
> 
> _______________________________________________
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit: 
> http://www.unidata.ucar.edu/mailing_lists/ 
> 
> _______________________________________________
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit: 
> http://www.unidata.ucar.edu/mailing_lists/