I think you should certainly file a bug report for such cases.
Ed
On Thu, Mar 3, 2016 at 4:17 PM, David Brown <dbrown@xxxxxxxx> wrote:
> Hi Ed,
> I can point you or anyone on the development team to sample files where
> ncdump (and netcdf4-python) show nested subgroups that appear to be
> empty, but which actually contain several compound datatype variables.
> Also other files with compound datatype variables where if you try to
> get the actual values, for string type members you only get the first
> character.
>
> Perhaps these are just isolated bugs and not an indication of less
> than full support for HDF5. In which case I would be happy to file a
> bug report and supply the file.
> -dave
>
>
> On Thu, Mar 3, 2016 at 12:26 PM, Ed Hartnett
> <edwardjameshartnett@xxxxxxxxx> wrote:
> > I think that it should be a bug if you can get a success code from
> nc_open,
> > and then the netCDF API cannot read the file.
> >
> > When nc_open is call, all the file metadata is read. If there is a
> problem
> > in the file, then it should fail.
> >
> > I believe the only thing that can't be read by netCDF are circular
> groups,
> > and the archaic TIME type. Is there anything else that won't work?
> >
> > Thanks,
> > Ed
> >
> >
> >
> > On Thu, Mar 3, 2016 at 1:29 PM, David Brown <dbrown@xxxxxxxx> wrote:
> >>
> >> > However, you can attempt to open a HDF5 file with nc_create, and, if
> it
> >> > fails, use HDF5. (It should very rarely fail.)
> >>
> >> But Ed, this is precisely the problem. You can successfully open an
> >> HDF5 file with the NetCDF4 interface, only to discover that there are
> >> elements inside the file that NetCDF4 does not support.
> >> -Dave Brown
> >>
> >> On Thu, Mar 3, 2016 at 10:39 AM, <netcdfgroup-request@xxxxxxxxxxxxxxxx
> >
> >> wrote:
> >> > Send netcdfgroup mailing list submissions to
> >> > netcdfgroup@xxxxxxxxxxxxxxxx
> >> >
> >> > To subscribe or unsubscribe via the World Wide Web, visit
> >> > http://mailman.unidata.ucar.edu/mailman/listinfo/netcdfgroup
> >> > or, via email, send a message with subject or body 'help' to
> >> > netcdfgroup-request@xxxxxxxxxxxxxxxx
> >> >
> >> > You can reach the person managing the list at
> >> > netcdfgroup-owner@xxxxxxxxxxxxxxxx
> >> >
> >> > When replying, please edit your Subject line so it is more specific
> >> > than "Re: Contents of netcdfgroup digest..."
> >> >
> >> >
> >> > Today's Topics:
> >> >
> >> > 1. Re: [Hdf-forum] Detecting netCDF versus HDF5 (Miller, Mark C.)
> >> > 2. Re: [Hdf-forum] Detecting netCDF versus HDF5 (Ed Hartnett)
> >> >
> >> >
> >> > ----------------------------------------------------------------------
> >> >
> >> > Message: 1
> >> > Date: Thu, 3 Mar 2016 00:07:03 +0000
> >> > From: "Miller, Mark C." <miller86@xxxxxxxx>
> >> > To: HDF Users Discussion List <hdf-forum@xxxxxxxxxxxxxxxxxx>
> >> > Cc: "netcdfgroup@xxxxxxxxxxxxxxxx" <netcdfgroup@xxxxxxxxxxxxxxxx>,
> >> > Ward Fisher <wfisher@xxxxxxxx>
> >> > Subject: Re: [netcdfgroup] [Hdf-forum] Detecting netCDF versus HDF5
> >> > Message-ID: <D2FCBD8C.44474%miller86@xxxxxxxx>
> >> > Content-Type: text/plain; charset="us-ascii"
> >> >
> >> > I like John's suggestion here.
> >> >
> >> > But, any code you add to any applications now will work *only* for
> files
> >> > that were produced post-adoption of this convention.
> >> >
> >> > There are probably a bazillion files out there at this point that
> don't
> >> > follow that convention and you probably still want your applications
> to be
> >> > able to read them.
> >> >
> >> > In VisIt, we support >140 format readers. Over 20 of those are
> different
> >> > variants of HDF5 files (H5part, Xdmf, Pixie, Silo, Samrai, netCDF,
> Flash,
> >> > Enzo, Chombo, etc., etc.) When opening a file, how does VisIt figure
> out
> >> > which plugin to use? In particular, how do we avoid one poorly written
> >> > reader plugin (which may be the wrong one for a given file) from
> preventing
> >> > the correct one from being found. Its kinda a hard problem.
> >> >
> >> > Some of our discussion is captured here. . .
> >> >
> >> > http://www.visitusers.org/index.php?title=Database_Format_Detection
> >> >
> >> > Mark
> >> >
> >> >
> >> > From: Hdf-forum
> >> > <hdf-forum-bounces@xxxxxxxxxxxxxxxxxx<mailto:
> hdf-forum-bounces@xxxxxxxxxxxxxxxxxx>>
> >> > on behalf of John Shalf <jshalf@xxxxxxx<mailto:jshalf@xxxxxxx>>
> >> > Reply-To: HDF Users Discussion List
> >> > <hdf-forum@xxxxxxxxxxxxxxxxxx<mailto:hdf-forum@xxxxxxxxxxxxxxxxxx>>
> >> > Date: Wednesday, March 2, 2016 1:02 PM
> >> > To: HDF Users Discussion List
> >> > <hdf-forum@xxxxxxxxxxxxxxxxxx<mailto:hdf-forum@xxxxxxxxxxxxxxxxxx>>
> >> > Cc: "netcdfgroup@xxxxxxxxxxxxxxxx<mailto:netcdfgroup@xxxxxxxxxxxxxxxx
> >"
> >> > <netcdfgroup@xxxxxxxxxxxxxxxx<mailto:netcdfgroup@xxxxxxxxxxxxxxxx>>,
> Ward
> >> > Fisher <wfisher@xxxxxxxx<mailto:wfisher@xxxxxxxx>>
> >> > Subject: Re: [Hdf-forum] Detecting netCDF versus HDF5
> >> >
> >> > Perhaps NetCDF (and other higher-level APIs that are built on top of
> >> > HDF5) should include an attribute attached to the root group that
> identifies
> >> > the name and version of the API that created the file? (adopt this
> as a
> >> > convention)
> >> >
> >> > -john
> >> >
> >> > On Mar 2, 2016, at 12:55 PM, Pedro Vicente
> >> > <pedro.vicente@xxxxxxxxxxxxxxxxxx<mailto:
> pedro.vicente@xxxxxxxxxxxxxxxxxx>>
> >> > wrote:
> >> > Hi Ward
> >> > As you know, Data Explorer is going to be a general purpose data
> reader
> >> > for many formats, including HDF5 and netCDF.
> >> > Here
> >> > http://www.space-research.org/
> >> > Regarding the handling of both HDF5 and netCDF, it seems there is a
> >> > potential issue, which is, how to tell if any HDF5 file was saved by
> the
> >> > HDF5 API or by the netCDF API?
> >> > It seems to me that this is not possible. Is this correct?
> >> > netCDF uses an internal function NC_check_file_type to examine the
> first
> >> > few bytes of a file, and for example for any HDF5 file the test is
> >> > /* Look at the magic number */
> >> > /* Ignore the first byte for HDF */
> >> > if(magic[1] == 'H' && magic[2] == 'D' && magic[3] == 'F') {
> >> > *filetype = FT_HDF;
> >> > *version = 5;
> >> > The problem is that this test works for any HDF5 file and for any
> netCDF
> >> > file, which makes it impossible to tell which is which.
> >> > Which makes it impossible for any general purpose data reader to
> decide
> >> > to use the netCDF API or the HDF5 API.
> >> > I have a possible solution for this , but before going any further, I
> >> > would just like to confirm that
> >> > 1) Is indeed not possible
> >> > 2) See if you have a solid workaround for this, excluding the
> dumb
> >> > ones, for example deciding on a extension .nc or .h5, or traversing
> the HDF5
> >> > file to see if it's non netCDF conforming one. Yes, to further
> complicate
> >> > things, it is possible that the above test says OK for a HDF5 file,
> but then
> >> > the read by the netCDF API fails because the file is a HDF5 non netCDF
> >> > conformant
> >> > Thanks
> >> > ----------------------
> >> > Pedro Vicente
> >> >
> >> > pedro.vicente@xxxxxxxxxxxxxxxxxx<mailto:
> pedro.vicente@xxxxxxxxxxxxxxxxxx>
> >> > http://www.space-research.org/
> >> > _______________________________________________
> >> > Hdf-forum is for HDF software users discussion.
> >> > Hdf-forum@xxxxxxxxxxxxxxxxxx<mailto:Hdf-forum@xxxxxxxxxxxxxxxxxx>
> >> >
> >> >
> http://secure-web.cisco.com/1r-EJFFfg6rWlpQsvXstBNTjaHQaKT_NkYRN0Jj_f-Z3EK0-hs6IbYc8XUBRyPsH3mU3CS0iiY7_qnchCA0QxNzQt270d_2HikCwpAWFmuHdacin62eaODutktDSOULIJmVbVYqFVSKWPzoX7kdP0yN9wIzSFxZfTwfhU8ebsN409xRg1PsW_8cvNiWzxDNm9wv9yBf9yK6nkEm-bOx2S0kBLbg9WfIChWzZrkpE3AHU9I-c2ZRH_IN-UF4g_g0_Dh4qE1VETs7tZTfKd1ox1MtBmeyKf7EKUCd3ezR9EbI5tK4hCU5qW4v5WWOxOrD17e8yCVmob27xz84Lr3bCK5wIQdH5VzFRTtyaAhudpt9E/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org
> >> > Twitter: https://twitter.com/hdf5
> >> >
> >> >
> >> > _______________________________________________
> >> > Hdf-forum is for HDF software users discussion.
> >> > Hdf-forum@xxxxxxxxxxxxxxxxxx<mailto:Hdf-forum@xxxxxxxxxxxxxxxxxx>
> >> >
> >> >
> http://secure-web.cisco.com/1r-EJFFfg6rWlpQsvXstBNTjaHQaKT_NkYRN0Jj_f-Z3EK0-hs6IbYc8XUBRyPsH3mU3CS0iiY7_qnchCA0QxNzQt270d_2HikCwpAWFmuHdacin62eaODutktDSOULIJmVbVYqFVSKWPzoX7kdP0yN9wIzSFxZfTwfhU8ebsN409xRg1PsW_8cvNiWzxDNm9wv9yBf9yK6nkEm-bOx2S0kBLbg9WfIChWzZrkpE3AHU9I-c2ZRH_IN-UF4g_g0_Dh4qE1VETs7tZTfKd1ox1MtBmeyKf7EKUCd3ezR9EbI5tK4hCU5qW4v5WWOxOrD17e8yCVmob27xz84Lr3bCK5wIQdH5VzFRTtyaAhudpt9E/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org
> >> > Twitter: https://twitter.com/hdf5
> >> >
> >> >
> >> > -------------- next part --------------
> >> > An HTML attachment was scrubbed...
> >> > URL:
> >> > <
> http://mailman.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/attachments/20160303/107872be/attachment.html
> >
> >> >
> >> > ------------------------------
> >> >
> >> > Message: 2
> >> > Date: Thu, 3 Mar 2016 12:39:04 -0500
> >> > From: Ed Hartnett <edwardjameshartnett@xxxxxxxxx>
> >> > To: "Miller, Mark C." <miller86@xxxxxxxx>
> >> > Cc: "netcdfgroup@xxxxxxxxxxxxxxxx" <netcdfgroup@xxxxxxxxxxxxxxxx>,
> >> > HDF
> >> > Users Discussion List <hdf-forum@xxxxxxxxxxxxxxxxxx>, Ward
> >> > Fisher
> >> > <wfisher@xxxxxxxx>
> >> > Subject: Re: [netcdfgroup] [Hdf-forum] Detecting netCDF versus HDF5
> >> > Message-ID:
> >> >
> >> > <CADO9Cd3KY8t8Et2CJmOFBXx9geJaWPANC1wZ5WQh1N6aMiEMsA@xxxxxxxxxxxxxx>
> >> > Content-Type: text/plain; charset="utf-8"
> >> >
> >> > This is a super interesting conversation for me! ;-)
> >> >
> >> > I must admit that when putting netCDF-4 together I never considered
> that
> >> > someone might want to tell the difference between a "native" HDF5 file
> >> > and
> >> > a netCDF-4/HDF5 file. Well, you can't think of everything.
> >> >
> >> > I agree that it is not possible to canonically tell the difference.
> The
> >> > netCDF-4 API does use some special attributes to track named
> dimensions,
> >> > and to tell whether classic mode should be enforced. But it can easily
> >> > produce files without any named dimensions, etc. So I don't think
> there
> >> > is
> >> > any easy way to tell.
> >> >
> >> > However, you can attempt to open a HDF5 file with nc_create, and, if
> it
> >> > fails, use HDF5. (It should very rarely fail.)
> >> >
> >> > Ed
> >> >
> >> >
> >> >
> >> > On Wed, Mar 2, 2016 at 7:07 PM, Miller, Mark C. <miller86@xxxxxxxx>
> >> > wrote:
> >> >
> >> >> I like John's suggestion here.
> >> >>
> >> >> But, any code you add to any applications now will work *only* for
> >> >> files
> >> >> that were produced post-adoption of this convention.
> >> >>
> >> >> There are probably a bazillion files out there at this point that
> don't
> >> >> follow that convention and you probably still want your applications
> to
> >> >> be
> >> >> able to read them.
> >> >>
> >> >> In VisIt, we support >140 format readers. Over 20 of those are
> >> >> different
> >> >> variants of HDF5 files (H5part, Xdmf, Pixie, Silo, Samrai, netCDF,
> >> >> Flash,
> >> >> Enzo, Chombo, etc., etc.) When opening a file, how does VisIt figure
> >> >> out
> >> >> which plugin to use? In particular, how do we avoid one poorly
> written
> >> >> reader plugin (which may be the wrong one for a given file) from
> >> >> preventing
> >> >> the correct one from being found. Its kinda a hard problem.
> >> >>
> >> >> Some of our discussion is captured here. . .
> >> >>
> >> >> http://www.visitusers.org/index.php?title=Database_Format_Detection
> >> >>
> >> >> Mark
> >> >>
> >> >>
> >> >> From: Hdf-forum <hdf-forum-bounces@xxxxxxxxxxxxxxxxxx> on behalf of
> >> >> John
> >> >> Shalf <jshalf@xxxxxxx>
> >> >> Reply-To: HDF Users Discussion List <hdf-forum@xxxxxxxxxxxxxxxxxx>
> >> >> Date: Wednesday, March 2, 2016 1:02 PM
> >> >> To: HDF Users Discussion List <hdf-forum@xxxxxxxxxxxxxxxxxx>
> >> >> Cc: "netcdfgroup@xxxxxxxxxxxxxxxx" <netcdfgroup@xxxxxxxxxxxxxxxx>,
> Ward
> >> >> Fisher <wfisher@xxxxxxxx>
> >> >> Subject: Re: [Hdf-forum] Detecting netCDF versus HDF5
> >> >>
> >> >> Perhaps NetCDF (and other higher-level APIs that are built on top of
> >> >> HDF5)
> >> >> should include an attribute attached to the root group that
> identifies
> >> >> the
> >> >> name and version of the API that created the file? (adopt this as a
> >> >> convention)
> >> >>
> >> >> -john
> >> >>
> >> >> On Mar 2, 2016, at 12:55 PM, Pedro Vicente <
> >> >> pedro.vicente@xxxxxxxxxxxxxxxxxx> wrote:
> >> >> Hi Ward
> >> >> As you know, Data Explorer is going to be a general purpose data
> reader
> >> >> for many formats, including HDF5 and netCDF.
> >> >> Here
> >> >> http://www.space-research.org/
> >> >> Regarding the handling of both HDF5 and netCDF, it seems there is a
> >> >> potential issue, which is, how to tell if any HDF5 file was saved by
> >> >> the
> >> >> HDF5 API or by the netCDF API?
> >> >> It seems to me that this is not possible. Is this correct?
> >> >> netCDF uses an internal function NC_check_file_type to examine the
> >> >> first
> >> >> few bytes of a file, and for example for any HDF5 file the test is
> >> >> /* Look at the magic number */
> >> >> /* Ignore the first byte for HDF */
> >> >> if(magic[1] == 'H' && magic[2] == 'D' && magic[3] == 'F') {
> >> >> *filetype = FT_HDF;
> >> >> *version = 5;
> >> >> The problem is that this test works for any HDF5 file and for any
> >> >> netCDF
> >> >> file, which makes it impossible to tell which is which.
> >> >> Which makes it impossible for any general purpose data reader to
> decide
> >> >> to
> >> >> use the netCDF API or the HDF5 API.
> >> >> I have a possible solution for this , but before going any further, I
> >> >> would just like to confirm that
> >> >> 1) Is indeed not possible
> >> >> 2) See if you have a solid workaround for this, excluding the
> dumb
> >> >> ones, for example deciding on a extension .nc or .h5, or traversing
> the
> >> >> HDF5 file to see if it's non netCDF conforming one. Yes, to further
> >> >> complicate things, it is possible that the above test says OK for a
> >> >> HDF5
> >> >> file, but then the read by the netCDF API fails because the file is a
> >> >> HDF5
> >> >> non netCDF conformant
> >> >> Thanks
> >> >> ----------------------
> >> >> Pedro Vicente
> >> >> pedro.vicente@xxxxxxxxxxxxxxxxxx
> >> >> http://www.space-research.org/
> >> >> _______________________________________________
> >> >> Hdf-forum is for HDF software users discussion.
> >> >> Hdf-forum@xxxxxxxxxxxxxxxxxx
> >> >>
> >> >>
> >> >>
> http://secure-web.cisco.com/1r-EJFFfg6rWlpQsvXstBNTjaHQaKT_NkYRN0Jj_f-Z3EK0-hs6IbYc8XUBRyPsH3mU3CS0iiY7_qnchCA0QxNzQt270d_2HikCwpAWFmuHdacin62eaODutktDSOULIJmVbVYqFVSKWPzoX7kdP0yN9wIzSFxZfTwfhU8ebsN409xRg1PsW_8cvNiWzxDNm9wv9yBf9yK6nkEm-bOx2S0kBLbg9WfIChWzZrkpE3AHU9I-c2ZRH_IN-UF4g_g0_Dh4qE1VETs7tZTfKd1ox1MtBmeyKf7EKUCd3ezR9EbI5tK4hCU5qW4v5WWOxOrD17e8yCVmob27xz84Lr3bCK5wIQdH5VzFRTtyaAhudpt9E/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org
> >> >> Twitter: https://twitter.com/hdf5
> >> >>
> >> >>
> >> >>
> >> >> _______________________________________________
> >> >> Hdf-forum is for HDF software users discussion.
> >> >> Hdf-forum@xxxxxxxxxxxxxxxxxx
> >> >>
> >> >>
> >> >>
> http://secure-web.cisco.com/1r-EJFFfg6rWlpQsvXstBNTjaHQaKT_NkYRN0Jj_f-Z3EK0-hs6IbYc8XUBRyPsH3mU3CS0iiY7_qnchCA0QxNzQt270d_2HikCwpAWFmuHdacin62eaODutktDSOULIJmVbVYqFVSKWPzoX7kdP0yN9wIzSFxZfTwfhU8ebsN409xRg1PsW_8cvNiWzxDNm9wv9yBf9yK6nkEm-bOx2S0kBLbg9WfIChWzZrkpE3AHU9I-c2ZRH_IN-UF4g_g0_Dh4qE1VETs7tZTfKd1ox1MtBmeyKf7EKUCd3ezR9EbI5tK4hCU5qW4v5WWOxOrD17e8yCVmob27xz84Lr3bCK5wIQdH5VzFRTtyaAhudpt9E/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org
> >> >> Twitter: https://twitter.com/hdf5
> >> >>
> >> >>
> >> >>
> >> >> _______________________________________________
> >> >> netcdfgroup mailing list
> >> >> netcdfgroup@xxxxxxxxxxxxxxxx
> >> >> For list information or to unsubscribe, visit:
> >> >> http://www.unidata.ucar.edu/mailing_lists/
> >> >>
> >> > -------------- next part --------------
> >> > An HTML attachment was scrubbed...
> >> > URL:
> >> > <
> http://mailman.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/attachments/20160303/7e288bd3/attachment.html
> >
> >> >
> >> > ------------------------------
> >> >
> >> > _______________________________________________
> >> > netcdfgroup mailing list
> >> > netcdfgroup@xxxxxxxxxxxxxxxx
> >> > For list information or to unsubscribe, visit:
> >> > http://www.unidata.ucar.edu/mailing_lists/
> >> >
> >> > End of netcdfgroup Digest, Vol 1110, Issue 2
> >> > ********************************************
> >>
> >> _______________________________________________
> >> netcdfgroup mailing list
> >> netcdfgroup@xxxxxxxxxxxxxxxx
> >> For list information or to unsubscribe, visit:
> >> http://www.unidata.ucar.edu/mailing_lists/
> >
> >
>
