Re: 1.8 file format

NOTE: The netcdf-hdf mailing list is no longer active. The list archives are made available for historical reasons.

Hi John,

On Jul 24, 2007, at 1:35 PM, John Caron wrote:


Hi Quincey: comments are inline:

Quincey Koziol wrote:

Hi John,
On Jul 24, 2007, at 12:22 PM, John Caron wrote:

Meanwhile yet another question on a different topic:
Im trying to figure out what the NPOESS data files are doing with "Reference Types". 

Case 1 is a Reference with type = 0 (Object Reference)

long VIIRS-AF-EDR_Aggr(5);
   :AggregateBeginningDate = "2003125";
   :AggregateBeginningGranuleID = "NPP001212126088";
   :AggregateBeginningOrbitNumber = 9; // int
   :AggregateBeginningTime = "10109.840960z";
   :AggregateCreationDate = "2003125";
   :AggregateEndingDate = "2003125";
   :AggregateEndingGranuleID = "NPP001212126088";
   :AggregateEndingOrbitNumber = 9; // int
   :AggregateEndingTime = "101038.325248z";
   :_LastModified = "2005-08-29T15:54:58Z";

data:

{2928, 3528, 3800, 4072, 4344}
The 5 values are indeed object references to 5 other datasets in the file. Any clues on how this is used, or is it an internal structure that should be left alone?
An object reference is simply the offset in the file of the object header for the object referenced. Users shouldn't modify them directly, but use the H5R* API routines for working with them. 

I can understand a reference to another object as an alias.
Bur what does an array of length 5 of references to 5 other objects mean?
The datatype is "object reference" and the datatype is "5", so that means the dataset's raw data is an array of 5 elements, each of which is an object reference (i.e. object ID) to another object in the file. 


h5dump is:

DATASET "VIIRS-AF-EDR_Aggr" {
           DATATYPE  H5T_REFERENCE
           DATASPACE  SIMPLE { ( 5 ) / ( 5 ) }
           ATTRIBUTE "AggregateBeginningDate" {
              DATATYPE  H5T_STRING {
                    STRSIZE 7;
                    STRPAD H5T_STR_NULLTERM;
                    CSET H5T_CSET_ASCII;
                    CTYPE H5T_C_S1;
                 }
              DATASPACE  SIMPLE { ( 1, 1 ) / ( 1, 1 ) }
           }
           ATTRIBUTE "AggregateBeginningGranuleID" {
              DATATYPE  H5T_STRING {
                    STRSIZE 15;
                    STRPAD H5T_STR_NULLTERM;
                    CSET H5T_CSET_ASCII;
                    CTYPE H5T_C_S1;
                 }
              DATASPACE  SIMPLE { ( 1, 1 ) / ( 1, 1 ) }
           }
           ....



Case 2 is a Reference, with type = 1 (Dataset Region Reference).

long VIIRS-AF-EDR_Gran_0(5);

data:
{3299824, 14172636162555905, 8589934592, 3299824, 14172636162555907} 

h5dump is

DATASET "VIIRS-AF-EDR_Gran_0" {
           DATATYPE  H5T_REFERENCE
           DATASPACE  SIMPLE { ( 5 ) / ( 5 ) }
The docs on this are pretty sketchy, i wonder if i could get an expanded description of what the Dataset Region Reference structure looks like? Heres whats there (last page of 1.6.5 doc):
"Dataset region references are stored as a heap-ID which points to the following information within the file-heap: an offset of the object pointed to, number-type information (same format as header message), dimensionality information (same format as header message), sub-set start and end information (i.e. a coordinate location for each), and field start and end names (i.e. a [pointer to the] string indicating the first field included and a [pointer to the] string name for the last field)." 

specifically:
 "an offset of the object pointed to" = object id ?
"number-type information (same format as header message)" datatype message? "dimensionality information (same format as header message)" = dataspace message? 

and then im even more lost as to what the fields are....
Hmm, looks like the docs are out of date and/or wrong for region references. Because region references are variable in length, the actual reference information is stored in the global heap and a (fixed length) heap ID is stored in the dataset's raw data - that part is right, although too briefly described. The format of the data in the heap is wrong however. The information actually stored in the heap is as follows: - The offset of the object header of the object (ie. dataset) pointed to (yes, an object ID) - A serialized form of a dataspace _selection_ of elements (in the dataset pointed to). I don't have a formal description of this information now, but it's encoded in the H5S_<foo>_serialize () routines in src/H5S<foo>.c, where foo = {all, hyper, point, none}. There is _no_ datatype information stored for these sort of selections currently. I'll file a bug report in our bugzilla to make certain this description gets fixed in the file format spec. 

ok thanks, any further docs as you develop them would be appreciated.


        Definitely. :-)
BTW, it doesnt seem like h5dump distinguishes between type 0 and type 1 references.
Hmm, what were your command line options to h5dump? (And what version of the library were you using?) 

        Quincey

Attachment: smime.p7s
Description: S/MIME cryptographic signature

  • 2007 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-hdf archives: