Andrew
If you are writing a new format, my guess is that you are writing an API for
it, and since I have been writing these High-Level APIs for years (starting
with the very own HDF5 High-Level API), and I am writing one now, here's my
advice for it :-)
1) Do not use groups as a means to access your data (more on this later).
2) Encapsulate all HDF5 IDs.
For my case, I am using a model where datasets are accessed for read/write
using their full path name, e.g "/path/to/dataset".
So, "/path/to/" is the group name and "dataset" is the dataset relative name.
API usage for read/write is
write( "/path/to/dataset", void * data buffer)
so, the model to read/write is just the full dataset name as a string and a
buffer with data.
The group part is implicit in the string, as you can see no HDF5 group IDs or
group paths to read/write (that is what I mean with
"not use groups as a means to access your data ")
How can you accomplish this?
Here's an example of functions part of a C++ class , regarding this part
void create_group(const std::string& group_name);
void create_dataset(const std::string& group_path, const std::string &
dataset_name, other parameters here like dataset size or chunk size
void write(const std::string& path, const void* buf);
Let's start with the create_group function
Here just create the group using the HDF5 C API (I use the C API even in a C++
program, the HDF5 C++ API adds nothing of use here)
gid = H5Gcreate2(this->m_fid, group_name.c_str(), other parameters
H5Gclose(gid)
As can be seen the group is created and closed immediately.
This means, you will have no groups open at all during your read/write calls
(typically the core part of a program execution).
Since groups are closed immediately, you can even exceed the number of maximum
open groups Russ mentioned.
For the dataset creation
create_dataset(const std::string& group_path, const std::string& dataset_name,
concatenate the group path with the dataset name and store it somehow in your
C++ class
std::string absolute_dataset_name = group_path + "/" + dataset_name;
In my case , I am using a map with dataset ID/ full path of dataset, because I
want to keep the datasets open (in memory), but any other way to store the path
will do
//map with dataset name (full path) / HDF5 dataset ID
std::map<std::string, hid_t> m_map_datasets;
here is the HDF5 create call
did = H5Dcreate2(this->m_fid, absolute_dataset_name.c_str()
The trick here is to use the file ID (that must be stored in the class) and the
full path. This takes group IDs out of the equation.
For the write/read call use simply
write(const std::string& path, const void* buf)
get the dataset ID from full path
//get dataset ID from map <path/ID>
hid_t did = this->m_map_datasets[path];
use it in the write call
H5Dwrite(did, other parameters
Hope this helps and ask any further questions if needed.
btw, I downloaded your file with only groups, it has a size of 37MB, lots of
group metadata here.
-Pedro
----------------------
Pedro Vicente
pedro.vicente@xxxxxxxxxxxxxxxxxx
http://www.space-research.org/
----- Original Message -----
From: Russ Rew
To: Andrew Dowsey
Cc: netcdfgroup@xxxxxxxxxxxxxxxx
Sent: Monday, November 17, 2014 2:11 PM
Subject: Re: [netcdfgroup] Problem with nc_inq_grps and ncdump reading
HDF5file
Hi Andrew,
You've run across a limitation in the number of simultaneously open groups
permitted in netCDF-4. The groups documentation says
... Encoding both the open file id and group id in a single integer
currently limits the number of groups per netCDF-4 file to no more than 32767.
Similarly, the number of simultaneously open netCDF-4 files in one program
context is limited to 32767.
I think both those limits should actually be 65535 (== 2**16 - 1), but in any
case, your HDF5 file has 119254 groups, which is too many for netCDF-4 to
handle.
The only workaround I can think of would be to close some groups if you don't
need to have them all open simultaneously.
--Russ
On Mon, Nov 17, 2014 at 8:14 AM, Andrew Dowsey
<andrew.dowsey@xxxxxxxxxxxxxxxx> wrote:
Hi,
I’m trying to create HDF5 files that can be read by NetCDF4 and I’ve run
into a problem in that nc_inq_grps seems to report some bad ids. ncdump bails
with this error too. h5dump works fine. The problem is deterministic but I
haven’t been able to figure out what causes it because slightly different HDF5
files work fine. I have created a test file that has this problem, which
contains nothing but groups. It can be downloaded from
http://personalpages.manchester.ac.uk/staff/andrew.dowsey/test.h5
I am creating a new format for a type of instrument data we use, and for
flexibility I would like it to be writeable/readable both by HDF5 and netCDF4
libraries.
Any insight would be greatly appreciated!
Kind regards,
Andy
Andrew Dowsey PhD
Lecturer and CADET Bioinformatics Research Lead
Institute of Human Development, The University of Manchester
t: +44 161 701 0244
f: +44 161 701 0242
http://www.manchester.ac.uk/research/andrew.dowsey
Centre for Advanced Discovery and Experimental Therapeutics (CADET)
Central Manchester University Hospitals NHS Foundation Trust
Oxford Road
Manchester M13 9WL
UK
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
------------------------------------------------------------------------------
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
