Hi Ed,
I can point you or anyone on the development team to sample files where
ncdump (and netcdf4-python) show nested subgroups that appear to be
empty, but which actually contain several compound datatype variables.
Also other files with compound datatype variables where if you try to
get the actual values, for string type members you only get the first
character.
Perhaps these are just isolated bugs and not an indication of less
than full support for HDF5. In which case I would be happy to file a
bug report and supply the file.
-dave
On Thu, Mar 3, 2016 at 12:26 PM, Ed Hartnett
<edwardjameshartnett@xxxxxxxxx> wrote:
> I think that it should be a bug if you can get a success code from nc_open,
> and then the netCDF API cannot read the file.
>
> When nc_open is call, all the file metadata is read. If there is a problem
> in the file, then it should fail.
>
> I believe the only thing that can't be read by netCDF are circular groups,
> and the archaic TIME type. Is there anything else that won't work?
>
> Thanks,
> Ed
>
>
>
> On Thu, Mar 3, 2016 at 1:29 PM, David Brown <dbrown@xxxxxxxx> wrote:
>>
>> > However, you can attempt to open a HDF5 file with nc_create, and, if it
>> > fails, use HDF5. (It should very rarely fail.)
>>
>> But Ed, this is precisely the problem. You can successfully open an
>> HDF5 file with the NetCDF4 interface, only to discover that there are
>> elements inside the file that NetCDF4 does not support.
>> -Dave Brown
>>
>> On Thu, Mar 3, 2016 at 10:39 AM, <netcdfgroup-request@xxxxxxxxxxxxxxxx>
>> wrote:
>> > Send netcdfgroup mailing list submissions to
>> > netcdfgroup@xxxxxxxxxxxxxxxx
>> >
>> > To subscribe or unsubscribe via the World Wide Web, visit
>> > http://mailman.unidata.ucar.edu/mailman/listinfo/netcdfgroup
>> > or, via email, send a message with subject or body 'help' to
>> > netcdfgroup-request@xxxxxxxxxxxxxxxx
>> >
>> > You can reach the person managing the list at
>> > netcdfgroup-owner@xxxxxxxxxxxxxxxx
>> >
>> > When replying, please edit your Subject line so it is more specific
>> > than "Re: Contents of netcdfgroup digest..."
>> >
>> >
>> > Today's Topics:
>> >
>> > 1. Re: [Hdf-forum] Detecting netCDF versus HDF5 (Miller, Mark C.)
>> > 2. Re: [Hdf-forum] Detecting netCDF versus HDF5 (Ed Hartnett)
>> >
>> >
>> > ----------------------------------------------------------------------
>> >
>> > Message: 1
>> > Date: Thu, 3 Mar 2016 00:07:03 +0000
>> > From: "Miller, Mark C." <miller86@xxxxxxxx>
>> > To: HDF Users Discussion List <hdf-forum@xxxxxxxxxxxxxxxxxx>
>> > Cc: "netcdfgroup@xxxxxxxxxxxxxxxx" <netcdfgroup@xxxxxxxxxxxxxxxx>,
>> > Ward Fisher <wfisher@xxxxxxxx>
>> > Subject: Re: [netcdfgroup] [Hdf-forum] Detecting netCDF versus HDF5
>> > Message-ID: <D2FCBD8C.44474%miller86@xxxxxxxx>
>> > Content-Type: text/plain; charset="us-ascii"
>> >
>> > I like John's suggestion here.
>> >
>> > But, any code you add to any applications now will work *only* for files
>> > that were produced post-adoption of this convention.
>> >
>> > There are probably a bazillion files out there at this point that don't
>> > follow that convention and you probably still want your applications to be
>> > able to read them.
>> >
>> > In VisIt, we support >140 format readers. Over 20 of those are different
>> > variants of HDF5 files (H5part, Xdmf, Pixie, Silo, Samrai, netCDF, Flash,
>> > Enzo, Chombo, etc., etc.) When opening a file, how does VisIt figure out
>> > which plugin to use? In particular, how do we avoid one poorly written
>> > reader plugin (which may be the wrong one for a given file) from preventing
>> > the correct one from being found. Its kinda a hard problem.
>> >
>> > Some of our discussion is captured here. . .
>> >
>> > http://www.visitusers.org/index.php?title=Database_Format_Detection
>> >
>> > Mark
>> >
>> >
>> > From: Hdf-forum
>> > <hdf-forum-bounces@xxxxxxxxxxxxxxxxxx<mailto:hdf-forum-bounces@xxxxxxxxxxxxxxxxxx>>
>> > on behalf of John Shalf <jshalf@xxxxxxx<mailto:jshalf@xxxxxxx>>
>> > Reply-To: HDF Users Discussion List
>> > <hdf-forum@xxxxxxxxxxxxxxxxxx<mailto:hdf-forum@xxxxxxxxxxxxxxxxxx>>
>> > Date: Wednesday, March 2, 2016 1:02 PM
>> > To: HDF Users Discussion List
>> > <hdf-forum@xxxxxxxxxxxxxxxxxx<mailto:hdf-forum@xxxxxxxxxxxxxxxxxx>>
>> > Cc: "netcdfgroup@xxxxxxxxxxxxxxxx<mailto:netcdfgroup@xxxxxxxxxxxxxxxx>"
>> > <netcdfgroup@xxxxxxxxxxxxxxxx<mailto:netcdfgroup@xxxxxxxxxxxxxxxx>>, Ward
>> > Fisher <wfisher@xxxxxxxx<mailto:wfisher@xxxxxxxx>>
>> > Subject: Re: [Hdf-forum] Detecting netCDF versus HDF5
>> >
>> > Perhaps NetCDF (and other higher-level APIs that are built on top of
>> > HDF5) should include an attribute attached to the root group that
>> > identifies
>> > the name and version of the API that created the file? (adopt this as a
>> > convention)
>> >
>> > -john
>> >
>> > On Mar 2, 2016, at 12:55 PM, Pedro Vicente
>> > <pedro.vicente@xxxxxxxxxxxxxxxxxx<mailto:pedro.vicente@xxxxxxxxxxxxxxxxxx>>
>> > wrote:
>> > Hi Ward
>> > As you know, Data Explorer is going to be a general purpose data reader
>> > for many formats, including HDF5 and netCDF.
>> > Here
>> > http://www.space-research.org/
>> > Regarding the handling of both HDF5 and netCDF, it seems there is a
>> > potential issue, which is, how to tell if any HDF5 file was saved by the
>> > HDF5 API or by the netCDF API?
>> > It seems to me that this is not possible. Is this correct?
>> > netCDF uses an internal function NC_check_file_type to examine the first
>> > few bytes of a file, and for example for any HDF5 file the test is
>> > /* Look at the magic number */
>> > /* Ignore the first byte for HDF */
>> > if(magic[1] == 'H' && magic[2] == 'D' && magic[3] == 'F') {
>> > *filetype = FT_HDF;
>> > *version = 5;
>> > The problem is that this test works for any HDF5 file and for any netCDF
>> > file, which makes it impossible to tell which is which.
>> > Which makes it impossible for any general purpose data reader to decide
>> > to use the netCDF API or the HDF5 API.
>> > I have a possible solution for this , but before going any further, I
>> > would just like to confirm that
>> > 1) Is indeed not possible
>> > 2) See if you have a solid workaround for this, excluding the dumb
>> > ones, for example deciding on a extension .nc or .h5, or traversing the
>> > HDF5
>> > file to see if it's non netCDF conforming one. Yes, to further complicate
>> > things, it is possible that the above test says OK for a HDF5 file, but
>> > then
>> > the read by the netCDF API fails because the file is a HDF5 non netCDF
>> > conformant
>> > Thanks
>> > ----------------------
>> > Pedro Vicente
>> >
>> > pedro.vicente@xxxxxxxxxxxxxxxxxx<mailto:pedro.vicente@xxxxxxxxxxxxxxxxxx>
>> > http://www.space-research.org/
>> > _______________________________________________
>> > Hdf-forum is for HDF software users discussion.
>> > Hdf-forum@xxxxxxxxxxxxxxxxxx<mailto:Hdf-forum@xxxxxxxxxxxxxxxxxx>
>> >
>> > http://secure-web.cisco.com/1r-EJFFfg6rWlpQsvXstBNTjaHQaKT_NkYRN0Jj_f-Z3EK0-hs6IbYc8XUBRyPsH3mU3CS0iiY7_qnchCA0QxNzQt270d_2HikCwpAWFmuHdacin62eaODutktDSOULIJmVbVYqFVSKWPzoX7kdP0yN9wIzSFxZfTwfhU8ebsN409xRg1PsW_8cvNiWzxDNm9wv9yBf9yK6nkEm-bOx2S0kBLbg9WfIChWzZrkpE3AHU9I-c2ZRH_IN-UF4g_g0_Dh4qE1VETs7tZTfKd1ox1MtBmeyKf7EKUCd3ezR9EbI5tK4hCU5qW4v5WWOxOrD17e8yCVmob27xz84Lr3bCK5wIQdH5VzFRTtyaAhudpt9E/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org
>> > Twitter: https://twitter.com/hdf5
>> >
>> >
>> > _______________________________________________
>> > Hdf-forum is for HDF software users discussion.
>> > Hdf-forum@xxxxxxxxxxxxxxxxxx<mailto:Hdf-forum@xxxxxxxxxxxxxxxxxx>
>> >
>> > http://secure-web.cisco.com/1r-EJFFfg6rWlpQsvXstBNTjaHQaKT_NkYRN0Jj_f-Z3EK0-hs6IbYc8XUBRyPsH3mU3CS0iiY7_qnchCA0QxNzQt270d_2HikCwpAWFmuHdacin62eaODutktDSOULIJmVbVYqFVSKWPzoX7kdP0yN9wIzSFxZfTwfhU8ebsN409xRg1PsW_8cvNiWzxDNm9wv9yBf9yK6nkEm-bOx2S0kBLbg9WfIChWzZrkpE3AHU9I-c2ZRH_IN-UF4g_g0_Dh4qE1VETs7tZTfKd1ox1MtBmeyKf7EKUCd3ezR9EbI5tK4hCU5qW4v5WWOxOrD17e8yCVmob27xz84Lr3bCK5wIQdH5VzFRTtyaAhudpt9E/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org
>> > Twitter: https://twitter.com/hdf5
>> >
>> >
>> > -------------- next part --------------
>> > An HTML attachment was scrubbed...
>> > URL:
>> > <http://mailman.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/attachments/20160303/107872be/attachment.html>
>> >
>> > ------------------------------
>> >
>> > Message: 2
>> > Date: Thu, 3 Mar 2016 12:39:04 -0500
>> > From: Ed Hartnett <edwardjameshartnett@xxxxxxxxx>
>> > To: "Miller, Mark C." <miller86@xxxxxxxx>
>> > Cc: "netcdfgroup@xxxxxxxxxxxxxxxx" <netcdfgroup@xxxxxxxxxxxxxxxx>,
>> > HDF
>> > Users Discussion List <hdf-forum@xxxxxxxxxxxxxxxxxx>, Ward
>> > Fisher
>> > <wfisher@xxxxxxxx>
>> > Subject: Re: [netcdfgroup] [Hdf-forum] Detecting netCDF versus HDF5
>> > Message-ID:
>> >
>> > <CADO9Cd3KY8t8Et2CJmOFBXx9geJaWPANC1wZ5WQh1N6aMiEMsA@xxxxxxxxxxxxxx>
>> > Content-Type: text/plain; charset="utf-8"
>> >
>> > This is a super interesting conversation for me! ;-)
>> >
>> > I must admit that when putting netCDF-4 together I never considered that
>> > someone might want to tell the difference between a "native" HDF5 file
>> > and
>> > a netCDF-4/HDF5 file. Well, you can't think of everything.
>> >
>> > I agree that it is not possible to canonically tell the difference. The
>> > netCDF-4 API does use some special attributes to track named dimensions,
>> > and to tell whether classic mode should be enforced. But it can easily
>> > produce files without any named dimensions, etc. So I don't think there
>> > is
>> > any easy way to tell.
>> >
>> > However, you can attempt to open a HDF5 file with nc_create, and, if it
>> > fails, use HDF5. (It should very rarely fail.)
>> >
>> > Ed
>> >
>> >
>> >
>> > On Wed, Mar 2, 2016 at 7:07 PM, Miller, Mark C. <miller86@xxxxxxxx>
>> > wrote:
>> >
>> >> I like John's suggestion here.
>> >>
>> >> But, any code you add to any applications now will work *only* for
>> >> files
>> >> that were produced post-adoption of this convention.
>> >>
>> >> There are probably a bazillion files out there at this point that don't
>> >> follow that convention and you probably still want your applications to
>> >> be
>> >> able to read them.
>> >>
>> >> In VisIt, we support >140 format readers. Over 20 of those are
>> >> different
>> >> variants of HDF5 files (H5part, Xdmf, Pixie, Silo, Samrai, netCDF,
>> >> Flash,
>> >> Enzo, Chombo, etc., etc.) When opening a file, how does VisIt figure
>> >> out
>> >> which plugin to use? In particular, how do we avoid one poorly written
>> >> reader plugin (which may be the wrong one for a given file) from
>> >> preventing
>> >> the correct one from being found. Its kinda a hard problem.
>> >>
>> >> Some of our discussion is captured here. . .
>> >>
>> >> http://www.visitusers.org/index.php?title=Database_Format_Detection
>> >>
>> >> Mark
>> >>
>> >>
>> >> From: Hdf-forum <hdf-forum-bounces@xxxxxxxxxxxxxxxxxx> on behalf of
>> >> John
>> >> Shalf <jshalf@xxxxxxx>
>> >> Reply-To: HDF Users Discussion List <hdf-forum@xxxxxxxxxxxxxxxxxx>
>> >> Date: Wednesday, March 2, 2016 1:02 PM
>> >> To: HDF Users Discussion List <hdf-forum@xxxxxxxxxxxxxxxxxx>
>> >> Cc: "netcdfgroup@xxxxxxxxxxxxxxxx" <netcdfgroup@xxxxxxxxxxxxxxxx>, Ward
>> >> Fisher <wfisher@xxxxxxxx>
>> >> Subject: Re: [Hdf-forum] Detecting netCDF versus HDF5
>> >>
>> >> Perhaps NetCDF (and other higher-level APIs that are built on top of
>> >> HDF5)
>> >> should include an attribute attached to the root group that identifies
>> >> the
>> >> name and version of the API that created the file? (adopt this as a
>> >> convention)
>> >>
>> >> -john
>> >>
>> >> On Mar 2, 2016, at 12:55 PM, Pedro Vicente <
>> >> pedro.vicente@xxxxxxxxxxxxxxxxxx> wrote:
>> >> Hi Ward
>> >> As you know, Data Explorer is going to be a general purpose data reader
>> >> for many formats, including HDF5 and netCDF.
>> >> Here
>> >> http://www.space-research.org/
>> >> Regarding the handling of both HDF5 and netCDF, it seems there is a
>> >> potential issue, which is, how to tell if any HDF5 file was saved by
>> >> the
>> >> HDF5 API or by the netCDF API?
>> >> It seems to me that this is not possible. Is this correct?
>> >> netCDF uses an internal function NC_check_file_type to examine the
>> >> first
>> >> few bytes of a file, and for example for any HDF5 file the test is
>> >> /* Look at the magic number */
>> >> /* Ignore the first byte for HDF */
>> >> if(magic[1] == 'H' && magic[2] == 'D' && magic[3] == 'F') {
>> >> *filetype = FT_HDF;
>> >> *version = 5;
>> >> The problem is that this test works for any HDF5 file and for any
>> >> netCDF
>> >> file, which makes it impossible to tell which is which.
>> >> Which makes it impossible for any general purpose data reader to decide
>> >> to
>> >> use the netCDF API or the HDF5 API.
>> >> I have a possible solution for this , but before going any further, I
>> >> would just like to confirm that
>> >> 1) Is indeed not possible
>> >> 2) See if you have a solid workaround for this, excluding the dumb
>> >> ones, for example deciding on a extension .nc or .h5, or traversing the
>> >> HDF5 file to see if it's non netCDF conforming one. Yes, to further
>> >> complicate things, it is possible that the above test says OK for a
>> >> HDF5
>> >> file, but then the read by the netCDF API fails because the file is a
>> >> HDF5
>> >> non netCDF conformant
>> >> Thanks
>> >> ----------------------
>> >> Pedro Vicente
>> >> pedro.vicente@xxxxxxxxxxxxxxxxxx
>> >> http://www.space-research.org/
>> >> _______________________________________________
>> >> Hdf-forum is for HDF software users discussion.
>> >> Hdf-forum@xxxxxxxxxxxxxxxxxx
>> >>
>> >>
>> >> http://secure-web.cisco.com/1r-EJFFfg6rWlpQsvXstBNTjaHQaKT_NkYRN0Jj_f-Z3EK0-hs6IbYc8XUBRyPsH3mU3CS0iiY7_qnchCA0QxNzQt270d_2HikCwpAWFmuHdacin62eaODutktDSOULIJmVbVYqFVSKWPzoX7kdP0yN9wIzSFxZfTwfhU8ebsN409xRg1PsW_8cvNiWzxDNm9wv9yBf9yK6nkEm-bOx2S0kBLbg9WfIChWzZrkpE3AHU9I-c2ZRH_IN-UF4g_g0_Dh4qE1VETs7tZTfKd1ox1MtBmeyKf7EKUCd3ezR9EbI5tK4hCU5qW4v5WWOxOrD17e8yCVmob27xz84Lr3bCK5wIQdH5VzFRTtyaAhudpt9E/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org
>> >> Twitter: https://twitter.com/hdf5
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> Hdf-forum is for HDF software users discussion.
>> >> Hdf-forum@xxxxxxxxxxxxxxxxxx
>> >>
>> >>
>> >> http://secure-web.cisco.com/1r-EJFFfg6rWlpQsvXstBNTjaHQaKT_NkYRN0Jj_f-Z3EK0-hs6IbYc8XUBRyPsH3mU3CS0iiY7_qnchCA0QxNzQt270d_2HikCwpAWFmuHdacin62eaODutktDSOULIJmVbVYqFVSKWPzoX7kdP0yN9wIzSFxZfTwfhU8ebsN409xRg1PsW_8cvNiWzxDNm9wv9yBf9yK6nkEm-bOx2S0kBLbg9WfIChWzZrkpE3AHU9I-c2ZRH_IN-UF4g_g0_Dh4qE1VETs7tZTfKd1ox1MtBmeyKf7EKUCd3ezR9EbI5tK4hCU5qW4v5WWOxOrD17e8yCVmob27xz84Lr3bCK5wIQdH5VzFRTtyaAhudpt9E/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org
>> >> Twitter: https://twitter.com/hdf5
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> netcdfgroup mailing list
>> >> netcdfgroup@xxxxxxxxxxxxxxxx
>> >> For list information or to unsubscribe, visit:
>> >> http://www.unidata.ucar.edu/mailing_lists/
>> >>
>> > -------------- next part --------------
>> > An HTML attachment was scrubbed...
>> > URL:
>> > <http://mailman.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/attachments/20160303/7e288bd3/attachment.html>
>> >
>> > ------------------------------
>> >
>> > _______________________________________________
>> > netcdfgroup mailing list
>> > netcdfgroup@xxxxxxxxxxxxxxxx
>> > For list information or to unsubscribe, visit:
>> > http://www.unidata.ucar.edu/mailing_lists/
>> >
>> > End of netcdfgroup Digest, Vol 1110, Issue 2
>> > ********************************************
>>
>> _______________________________________________
>> netcdfgroup mailing list
>> netcdfgroup@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>
>
