On Mon, May 11, 2015 at 4:04 PM, Dave Allured - NOAA Affiliate <
dave.allured@xxxxxxxx> wrote:
> I appreciate your interest in the details. Netcdf-3 does not include
> support for "jagged arrays", so all data storage is in rectangular prisms.
> However, there is some support for jagged arrays in netcdf-4.
>
Is there? could you point me to some docs about that?
I found this:
"""
In netCDF-4/HDF5 format files, the introduction of the compound type allows
the creation of complex data types, involving any combination of types. The
VLEN type allows efficient storage of ragged arrays, and the introduction
of hierarchical groups allows users new ways to organize data.
"""
here:
https://www.unidata.ucar.edu/software/netcdf/docs/netcdf/Limitations.html
but haven't been able to see how to do it.
-CHB
> --Dave
>
>
> On Mon, May 11, 2015 at 3:13 PM, Michael Powell <mwpowellhtx@xxxxxxxxx>
> wrote:
>
>> On Mon, May 11, 2015 at 3:28 PM, Dave Allured - NOAA Affiliate
>> <dave.allured@xxxxxxxx> wrote:
>> > Michael,
>> >
>> > On Sun, May 10, 2015 at 2:35 PM, Michael Powell <mwpowellhtx@xxxxxxxxx>
>> > wrote:
>> >>
>> >> It's been a few years since I've looked at netCDF at all, so this is a
>> >> refresher. Also, this has no doubt been covered a million times, but I
>> >> am confused about the nature of the data block.
>> >
>> >
>> > You are inquiring specifically about netcdf-3 format (classic and 64-bit
>> > offset), not netcdf-4 (HDF5), right?
>>
>> For now, yessir. Thanks for the follow up.
>>
>> > Unless you have some very good reason for attempting to access netcdf-3
>> > format directly, please use the netcdf libraries (C, fortran, etc.) to
>> read
>> > and write. Rough knowledge of the low-level format is helpful for
>> > understanding performance issues.
>>
>> It's good to know. Mostly, morbid curiosity, I enjoy working with file
>> formats like this, but I did take note that there are C/C++ and other
>> API readily available. I like to push my architectural skills over
>> areas like this.
>>
>> >> If I understand the header/data parts correctly (which, I may not),
>> >> non-record can be 'infinite'?
>> >
>> >
>> > Not infinite, rather about 8 exabytes. But that is probably what you
>> meant.
>>
>> Practically infinite, at any rate.
>>
>> > This is shown in a good summary table in the older netcdf documentation:
>> >
>> http://www.unidata.ucar.edu/software/netcdf/docs/netcdf/Large-File-Support.html
>> >
>> >>
>> >> Rather, the non-record, fixed-length
>> >> parts should occur prior to the record data?
>> >
>> >
>> > Yes. I think the following overview is a little less confusing than in
>> the
>> > BNF spec. Think of all netcdf-3 files as having five possible parts.
>> All
>> > parts except the header are optional. The two pads are a minor detail,
>> > don't worry about them:
>> >
>> > * Header
>> > * Header pad
>> > * Fixed length variables
>> > * Data pad
>> > * Record data
>> >
>> >>
>> >> How does one tell what is record versus non-record?
>> >
>> >
>> > Many applications don't bother to look. Basic read access works the
>> same
>> > either way, except for performance issues with large files.
>> >
>> > If you care, the manual method is to look for a dimension labeled
>> > "unlimited" in ncdump -h output. Any variables using this dimension are
>> > record variables. All others are fixed length variables.
>> >
>> > There are inquiry functions which can do this under program control.
>> >
>> >>
>> >> I think this is as a function of the var nelems, dimid, etc, but it's
>> >> not especially clear from the BNF grammar, or associated verbiage. For
>> >> example, what is meant by 'interleaved'? As a cross section of the
>> >> specified variables?
>> >
>> >
>> > Only record variables are interleaved. The sub-arrays ("cross
>> sections")
>> > for record subscript #0 of all record variables are combined together
>> at the
>> > start of the variable length section. The combined block for subscript
>> #0
>> > is often referred to casually as one "record". The combined block for
>> > subscript #1 is the next record. And so on.
>> >
>> > Suppose you have precip(time, lat, lon) and time(time), and time(time).
>> > Then the first record is precip(0,*,*) and time(0) together, and so on.
>> >
>> >> Also taking into account spiked variable array dimensions?
>> >
>> >
>> > What do you mean by "spiked"?
>>
>> Jagged arrays: something like this (turned 90 degrees):
>>
>> ++
>> ++++
>> +
>> ++++++
>> ++++++
>> ++
>>
>> And so on.
>>
>> >> I'm sure I'm barely scratching the surface here... Insights are
>> welcome.
>> >>
>> >> Thank you...
>> >>
>> >> Regards,
>> >>
>> >> Michael
>> >
>> >
>> > --Dave
>>
>
>
> _______________________________________________
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@xxxxxxxx
