Re: using netCDF for storing data from realtime source

Depending on resources, I'd consider this application for a relational database like PostgreSQL, MySQL or Oracle, depending on your preferences and hexadecimal religion. A simple insert without reindexing is pretty quick, and if you're certain you don't have duplications it can be blindingly fast, in my experience.

We do this for surface observations coming from a variety of sources with a variety of frequencies of update, although no one source at 1000 Hz...

gerry

Timothy Hume wrote:
Hi Magnus,

I once set up a system to store real-time meteorological data in NetCDF files. One of the difficulties is that you can never be sure of the temporal frequency of the data; some stations report hourly, others three hourly, and others report irregularly. To get around this issue, I used the unlimited dimension for time, and simply "grew" the dimension as new data became available. However, "growing" the unlimited dimension seemed to take a lot of computer time compared to slotting the data into an array where all the dimensions were fixed length. It may be that I wrote my software inefficiently, but I suspect the slowness of "growing" the unlimited dimension is partly due to the structure of a NetCDF file. If it is possible, it may be more efficient to predefine the length of your time dimension (making it large enough to hold all the data you receive), rather than using an unlimited dimension. The downside of this approach is that you are likely to be left with some "empty cells" in your data array.

Also, the option of using a text file, as suggested by someone else, may be better. You can trivially append new data to a text file using the shell's ">>" feature.

Tim Hume
Bureau of Meteorology Research Centre
Melbourne
Australia

-----Original Message-----
From: owner-netcdfgroup@xxxxxxxxxxxxxxxx on behalf of Magnus Hagdorn
Sent: Fri 2007-01-26 05:44
To: netcdfgroup@xxxxxxxxxxxxxxxx
Subject: using netCDF for storing data from realtime source

Hello world,
we are currently considering to use the netCDF library to store data
coming from our instruments. The data represents multiple variables
together with a timestamp of the measurement. Some variables are updated
more often than others. Sample rates are of the order of 100Hz to
1000Hz. I was thinking of storing the data as a table with 3 columns,
timestamp, ID and value together with some appropriate header. Do you
think this is a reasonable idea, are there other more suitable standard
formats?
Regards
magnus

==============================================================================
To unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================


--
Gerry Creager -- gerry.creager@xxxxxxxx
Texas Mesonet -- AATLT, Texas A&M University        
Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983
Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843

==============================================================================
To unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================


  • 2007 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: