Re: Need advice on storing lat-lon data

Cj Beegle-Krause wrote:

Hi Matthew,

We at NOAA had the same problem - rectangular grids are easy to deal with, but curvilinear and triangular grids get very messy. Thankfully, netCDF is very flexible. We at NOAA (CSDL, CO-OPS, PMEL and HAZMAT) are in the process of standardizing spill response and other relevant data exchange among nowcast/forecast models for rectangular, curvilinear and triangular grids. The idea is that there are many nowcast/forecast models around the US, and in order for a spill trajectory modeler like me to use this data quickly it had to be very standardized for relevant data fields. Our (NOAA/HAZMAT) response time from initial incident notification to descriptive products is 30 minutes and model products is 2 hours on a 24 x 7 basis. If HAZMAT modelers have to worry about converting formats at 3AM on Christmas Day, we will use our inhouse modeling capability to make the response deadline and save looking at outside model products for later.

Creating these standards also allows the creation of consistent graphical and analysis wraparound for nowcast/forecast models, so that if you can find the data you want from one forecast model, another will have the same look and feel. Also, researchers that develop nowcast/forecast models can plug into an operational system that covers the user needs without significant programming. We (CSDL, CO-OPS, HAZMAT) are working with several of the regional OOS that are interested in either having their models certified as operational and/or making their model circulation fields available for spill response. One of our goals is to connect these models via DODS/OPeNDAP to the Live Access Server (LAS) following the IOOS/DMAC conventions with constraints for emergency response requirements. Our own HAZMAT LAS can subset these models and provide the data to us formatted for our trajectory model (GNOME). Our prototype system for providing nowcast/forecast model data allows HAZMAT trajectory modelers to (1) select a nowcast/forecast model in the system (2) create custom graphics of data fields for QA/QC purposes and (3) to download custom subsetted data in GNOME format in less than 2 minutes total, which fits well with our response time.

I've copied examples below for curvilinear and triangular grids. You mentioned that you already have a standard for rectangular grids, but if you are interested in ours, just let me know. Also let me know if you are interested in finding out more about the NOAA operational model standardization, and I can direct you to the relevant people at NOAA/CO-OPS and NOAA/CSDL.

Best Regards,
CJ

--
CJ Beegle-Krause, Ph.D.
NOAA/NOS/ORR/Hazmat
7600 Sand Point Way NE
Seattle, WA 98115
voice: (206) 526-6961
fax: (206) 526-6329

<:}}}}}><
<:}}}}}><

\!/ \!/ \!/ >^<**>^< \!/ \!/ \!/


_Curvilinear Grid netCDF_
Below is an example of the curvilinear format for netCDF files (this is the same as the LAS server output format). This still may be tweaked to include a land mask, and global attributes (e.g. grid_type). Currently the _FillValue attribute is used to identify land points and a boundary map is created. The dimension names only need to start with X, Y or LAT, LON to be recognized. The first depth value is used. Time is always assumed to be the unlimited dimension. The GNOME topology can be saved out from GNOME the first time and reloaded.

netCDF MacintoshHD:Desktop Folder:Tampa2 {
dimensions:
    XPOS29_41 = 13 ;
    YPOS28_43 = 16 ;
    ZPOS1_1 = 1 ;
    TIME = UNLIMITED ; // (1 currently)
variables:
    double XPOS29_41(XPOS29_41) ;
        XPOS29_41:point_spacing = "even" ;
    double YPOS28_43(YPOS28_43) ;
        YPOS28_43:point_spacing = "even" ;
    double ZPOS1_1(ZPOS1_1) ;
        ZPOS1_1:point_spacing = "even" ;
    double TIME(TIME) ;
        TIME:units = "days since 1999-01-01 00:00:00" ;
        TIME:time_origin = "01-JAN-1999 00:00:00" ;
    float U(TIME, ZPOS1_1, YPOS28_43, XPOS29_41) ;
        U:missing_value = -1.e+34f ;
        U:_FillValue = -1.e+34f ;
        U:long_name = "U East-West" ;
        U:history = "From ecom_fc" ;
        U:units = "m/s" ;
    float V(TIME, ZPOS1_1, YPOS28_43, XPOS29_41) ;
        V:missing_value = -1.e+34f ;
        V:_FillValue = -1.e+34f ;
        V:long_name = "V North-South" ;
        V:history = "From ecom_fc" ;
        V:units = "m/s" ;
    float LONGITUDE(YPOS28_43, XPOS29_41) ;
        LONGITUDE:missing_value = -1.e+34f ;
        LONGITUDE:_FillValue = -1.e+34f ;
        LONGITUDE:long_name = "Longitude" ;
        LONGITUDE:units = "degrees east" ;
    float LATITUDE(YPOS28_43, XPOS29_41) ;
        LATITUDE:missing_value = -1.e+34f ;
        LATITUDE:_FillValue = -1.e+34f ;
        LATITUDE:long_name = "Latitude" ;
        LATITUDE:units = "degrees north" ;

// global attributes:
        :history = "FERRET V5.41   13-Nov-02" ;
data:

 XPOS29_41 = 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41 ;

YPOS28_43 = 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43 ;

 ZPOS1_1 = 1 ;

 TIME = 1412.41662597656 ;
}

_Triangular Grid netCDF_
Below is an example of the triangular grid format for netCDF files. The global attribute grid_type = TRIANGULAR is required (the default is CURVILINEAR). Time is always assumed to be the unlimited dimension. The first depth value is used. Time is always assumed to be the unlimited dimension. A map will be created using the boundary data. The GNOME topology can be saved out from GNOME the first time and reloaded.

The netCDF header description for finite element model:
netCDF MacintoshHD:Desktop Folder:testFile {
dimensions:
    node = 7258 ;
    nele = 13044 ;
    nbnd = 1476 ;
    nbi = 4 ;
    sigma = 11 ;
    time = UNLIMITED ; // (12 currently)
variables:
    short bnd(nbnd, nbi) ;
        bnd:long_name = "Boundary Segment Node List" ;
        bnd:units = "nondimensional" ;
    float time(time) ;
        time:long_name = "Time" ;
        time:units = "days since 2003-01-00  0:00:00 00:00" ;
        time:base_date = 2003, 1, 0, 0 ;
    float lon(node) ;
        lon:long_name = "Longitude" ;
        lon:units = "degrees_east" ;
    float lat(node) ;
        lat:long_name = "Latitude" ;
        lat:units = "degrees_north" ;
    float sigma(sigma) ;
        sigma:long_name = "Stretched Vertical Coordinate" ;
        sigma:units = "sigma_level" ;
        sigma:positive = "down" ;
    float u(time, sigma, node) ;
        u:long_name = "Eastward Water Velocity" ;
        u:units = "m/s" ;
        u:missing_value = -99999.f ;
    float v(time, sigma, node) ;
        v:long_name = "Northward Water Velocity" ;
        v:units = "m/s" ;
        v:missing_value = -99999.f ;

// global attributes:
        :file_type = "FEM" ;
        :Conventions = "COARDS" ;
        :grid_type = "Triangular" ;
data:

time = 26.95833, 27, 27.04167, 27.08333, 27.125, 27.16667, 27.20833, 27.25,
    27.29167, 27.33333, 27.375, 27.41667 ;

 sigma = 1, 0.9807215, 0.9306101, 0.83061, 0.6807215, 0.5, 0.3192785,
    0.1693899, 0.06938996, 0.01927857, 0 ;
}

Notes :
1. The boundary list is an array of dimension bnd(nbnd, 4)
It is node numbers of the line segments, with a digit to
indicate which land or island the segment is a part of, and
a digit to indicate whether a boundary is land or water:
 node1  node2  island  land/water (0/1)
  1      2      0       (0 is usually the continent and outer water BC)
  2      5      0
  5      23     0
....
  3568   1      0         The last segment joins up with the first
  551    552    1         next island
  552    567    1
...
  677    551    1
789 388 2 .. next island, etc....

Hi CJ:

I have been studying netcdf data usage as part of the THREDDS project, and I'm interested in your standardization efforts. You might be interested in the CF Conventions, which I *think* would handle both cases above:

 http://www.cgd.ucar.edu/cms/eaton/cf-metadata/

If you decide to use a different set of conventions, you should add a global attribute

       :Conventions = "MyStandard";

and send me a link to it, and ill put it on this page:

http://www.unidata.ucar.edu/packages/netcdf/conventions.html

Also I think the second case is probably not COARDS compliant, so shouldnt have that Convention attribute.

In any case, I would like to hear more about the NOAA operational model standardization, so any additional info would be helpful.


  • 2003 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: