Howdy Torben!
Certainly you should store these data in the netCDF file, not in a separate
text file! That's the whole point of netCDF.
You can store them as attributes. You can store them as a 1-D array of
values, or as a char array full of KEY=VALUE pairs, or you can create an
attribute for each key. You could also create one or more compound types to
organize your metadata into structs.
You could also store the exact same data in variables. If you are going to
change them, you should do that.
Don't try and game netcdf WRT to loading data into memory. True, metadata
are loaded into memory, but pages of data are also loaded into memory, and
if you fill up your memory, the OS will page memory off to disk without
notification. So best to store the data in the most convenient
representation and let netcdf and the OS read it as best they can.
(NetCDF-4 provides some advances caching control as well.)
Ed Hartnett
On Thu, Nov 30, 2017 at 9:57 AM, Constantine Khrulev <c.khroulev@xxxxxxxxx>
wrote:
> Torben,
>
> PISM (www.pism-docs.org) uses NetCDF attributes to store the model
> configuration[1]. This is not a great solution, but it served us well so
> far. We store about 300 parameters, which, together with their metadata (I
> had to come up with a homespun convention using attribute name suffixes for
> this) gives around 1300 NetCDF variable attributes.
>
> When using the NetCDF-3 file format you should pad the file header so that
> all your attributes "fit" (see nc__enddef; note the double underscore). In
> most cases this does not matter, but it could reduce the time spent
> creating a NetCDF-3 file (especially if the file is large).
>
> When using the HDF5-based NetCDF-4 file format the file header size is not
> an issue, but *modifying* attributes is: see issue 350 on GitHub [2]. It
> does not look like this issue will be fixed any time soon, so I would
> advise against using attributes to store data that you may need to update
> after the file is created.
>
> As far as I know (hopefully NetCDF developers here will correct me if I'm
> wrong) all file metadata is stored in RAM when a file is opened, so
> accessing attributes is cheap *if* you keep the file open, but increasing
> the number of attributes should increase the cost of re-opening files.
> (This additional cost is not likely to be significant, though.)
>
> [1]: https://github.com/pism/pism/blob/master/src/pism_config.cdl
> [2]: https://github.com/Unidata/netcdf-c/issues/350
>
>
> On 11/30/17 7:06 AM, Helsloot, T. wrote:
>
>> Hello,
>>
>> How bad of an idea is it to store data in attributes?
>> I want to store settings (metadata) containing 600+ key value pairs.
>>
>> These settings may change in time, though infrequently and they don’t
>> need to be read often.
>>
>> I could store them as attributes with a key and a 1D array of values,
>> which has the advantage that I have all the metadata within the .nc file .
>>
>> In an extreme case the array will grow to a thousand entries.
>>
>> Alternatively I would store the settings history in a separate text file .
>>
>> Any thoughts on this are appreciated,
>>
>> Regards,
>>
>> Torben
>>
>>
>>
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web. Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>>
>>
>> netcdfgroup mailing list
>> netcdfgroup@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>>
>>
> --
> Constantine
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
