Hi Magnus,
I once set up a system to store real-time meteorological data in NetCDF
files. One of the difficulties is that you can never be sure of the
temporal frequency of the data; some stations report hourly, others
three hourly, and others report irregularly. To get around this issue, I
used the unlimited dimension for time, and simply "grew" the dimension
as new data became available. However, "growing" the unlimited dimension
seemed to take a lot of computer time compared to slotting the data into
an array where all the dimensions were fixed length. It may be that I
wrote my software inefficiently, but I suspect the slowness of "growing"
the unlimited dimension is partly due to the structure of a NetCDF file.
If it is possible, it may be more efficient to predefine the length of
your time dimension (making it large enough to hold all the data you
receive), rather than using an unlimited dimension. The downside of this
approach is that you are likely to be left with some "empty cells" in
your data array.
Also, the option of using a text file, as suggested by someone else, may
be better. You can trivially append new data to a text file using the
shell's ">>" feature.
Tim Hume
Bureau of Meteorology Research Centre
Melbourne
Australia
-----Original Message-----
From: owner-netcdfgroup@xxxxxxxxxxxxxxxx on behalf of Magnus Hagdorn
Sent: Fri 2007-01-26 05:44
To: netcdfgroup@xxxxxxxxxxxxxxxx
Subject: using netCDF for storing data from realtime source
Hello world,
we are currently considering to use the netCDF library to store data
coming from our instruments. The data represents multiple variables
together with a timestamp of the measurement. Some variables are updated
more often than others. Sample rates are of the order of 100Hz to
1000Hz. I was thinking of storing the data as a table with 3 columns,
timestamp, ID and value together with some appropriate header. Do you
think this is a reasonable idea, are there other more suitable standard
formats?
Regards
magnus
=============================================================================To
unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
=============================================================================
