[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: NetCDF structures and Profile/in situ data

Roy Mendelssohn wrote:


Hi John:
On your web pages, you have some very interesting suggestions on how to use structures and linked lists within netcdf to store profile or in situ data. I am wondering if this is being used by anyone on real data sets, and there experience with access times etc. etc. 

Any info appreciated.

Regards,

-Roy M.


Hi Roy:

The latest draft of that is at

http://my.unidata.ucar.edu/content/software/netcdf-java/formats/UnidataObsConvention.html
we still havent finalized it, however we are using it in the IDD metar decoder, a sample file (50 Mb) can be downloaded at 

http://motherlode.ucar.edu:8088/thredds/fileServer/station/metar/20050802_metar.nc

in this catalog:

http://motherlode.ucar.edu:8088/thredds/catalog.html
we have not yet done any timing on this, but we know that for our current purposes (implementing StationObsDataset in CDM) it has good response time when reading from local files (not so good for remote files dure to latency). The queries we optimize for are "get all obs at a station", and "get all stations in a bounding box". The metar file above has 25 - 50 observations for each of 4000 stations. Each obs probably requires a disk access / network round trip. If you are writing these files with complete control over the layout, id use the contiguous list layout. (That assumes you want to optimise the "get all obs at a station".)
we havent yet written or tested files for trajectory or profile data yet, but we will.
let me know what plans you have, perhaps we can work together to finalize this and get some tests run. 


John