[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20050329:Is it possible to archive streaming data using NetCDF?

Unidata Support wrote:


------- Forwarded Message


To: address@hidden
From: "Peyush Jain" <address@hidden>
Subject: netCDF Java - Newbie Question: Is it possible to archive streaming 
data using NetCDF?
Organization: UCAR/Unidata
Keywords: 200503291623.j2TGNUiv002216


Institution: NASA
Package Version: Version 2.1
Operating System: Windows XP Pro
Hardware Information: P4, 3.0GHz, 2GB RAM
Inquiry: Hello,

I was wondering if it is possible to archive streaming data using netCDF. If it 
is, can you please point me to an example? A
no problem with streaming data - its just like any other. the main thing is to make sure your outer dimension is the unlimited dimension, so that you can continue to expand the file. 

example below



ssuming the data coming in has 5 one dimensional arrays (of different types), I 
was able to create 5 variables and save them to a netCDF file. Everytime I 
receive this data, it overwrites the previous data in the file. I am sure that 
it is not appending the data because the file size never increases.

I am using the following function to write to netCDF file:
ncfile.write(name, ArrayAbstract.factory(doubleArray));

problem is you are using this call, where origin assumed to be 0
/** Write data to the named variable, origin assumed to be 0. Must not be in define mode. * @param varName name of variable. IllegalArgumentException if variable name does not exist. 
  * @param values write this array; must be same type and rank as Variable
  * @throws IOException
  */
public void write(String varName, Array values) throws java.io.IOException, InvalidRangeException;
you should use this one, and create an origin array; then increment the outer dimension each time you write (leave the others zero): 

 /** Write data to the named variable. Must not be in define mode.
* @param varName name of variable. IllegalArgumentException if variable name does not exist. 
  * @param origin offset within the variable to start writing.
  * @param values write this array; must be same type and rank as Variable
  * @throws IOException
  */
public void write(String varName, int [] origin, Array values) throws java.io.IOException, InvalidRangeException { 


 public void testNC3WriteWithRecord() throws IOException {
NetcdfFileWriteable ncfile = new NetcdfFileWriteable("C:/temp/writeRecordExample.nc", false); 

   // define dimensions, including unlimited
   Dimension latDim = ncfile.addDimension("lat", 64);
   Dimension lonDim = ncfile.addDimension("lon", 128);
   Dimension timeDim = ncfile.addDimension("time", -1);

   // define Variables
   Dimension[] dim3 = new Dimension[3];
   dim3[0] = timeDim;
   dim3[1] = latDim;
   dim3[2] = lonDim;

   // double T(time, lat, lon) ;
   //   T:long_name="surface temperature" ;
   //   T:units = "degC" ;
   ncfile.addVariable("T", DataType.DOUBLE, dim3);
   ncfile.addVariableAttribute("T", "long_name", "surface temperature");
   ncfile.addVariableAttribute("T", "units", "degC");


   // float lat(lat) ;
   //   lat:units = "degrees_north" ;
   ncfile.addVariable("lat", DataType.FLOAT, new Dimension[] {latDim});
   ncfile.addVariableAttribute("lat", "units", "degrees_north");

   // float lon(lon) ;
   // lon:units = "degrees_east" ;
   ncfile.addVariable("lon", DataType.FLOAT, new Dimension[] {lonDim});
   ncfile.addVariableAttribute("lon", "units", "degrees_east");

   // int time(time) ;
   //   time:units = "hours" ;
   ncfile.addVariable("time", DataType.INT, new Dimension[] {timeDim});
   ncfile.addVariableAttribute("time", "units", "hours");

   //  :title = "Example Data" ;
   ncfile.addGlobalAttribute("title", "Example Data");

   // create the file
   try {
     ncfile.create();
   }  catch (IOException e) {
     System.err.println("ERROR creating file");
     e.printStackTrace();
     return;
   }
   System.out.println( "ncfile = "+ ncfile);

   // now write one record at a time
   Variable v = ncfile.findTopVariable("T");
ArrayDouble data = new ArrayDouble.D3(1, latDim.getLength(), lonDim.getLength()); 
   ArrayInt timeData = new ArrayInt.D1(1);
   int[] origin = new int[v.getRank()];
   int[] timeOrigin = new int[1];

   for (int time=0; time<100; time++) {
     // fill the data array
     Index ima = data.getIndex();
     for (int j=0; j<latDim.getLength(); j++)
       for (int k=0; k<lonDim.getLength(); k++)
         data.setDouble(ima.set(0,j,k), (double) time*j*k);
     timeData.setInt( timeData.getIndex(), time);

     // write to file
     origin[0] = time;
     timeOrigin[0] = time;
     try {
       ncfile.write("T", origin, data);
       ncfile.write("time", timeOrigin, timeData);
     } catch (IOException e) {
       System.err.println("ERROR writing file");
     } catch (InvalidRangeException e) {
       e.printStackTrace();
     }
   }

   // all done
   try {
     ncfile.close();
   } catch (IOException e) {
     System.err.println("ERROR writing file");
   }

   System.out.println( "**** TestWriteRecord done");
 }