Re: [netcdfgroup] questions about netCDf file size.

you are using netcdf-4 format i think. this has much more complex storage 
objects than what you are used to with netcdf-3 format.

LB wrote:
Hi,

I'm very new to netCDF and I was surprised by the size of files generated by netCDF.
Here is the header of the netCDF file :

ncdump -h cycle.nc <http://cycle.nc>
netcdf cycle {
dimensions:
        Regime = 3 ;

group: Regimes {
  variables:
        float T41(Regime) ;
        float TTR(Regime) ;
        float N1(Regime) ;

  } // group Regimes

group: Cycles {
  dimensions:
        Cycle = 5 ;
        SousCycle = 11 ;
  variables:
        byte Type(Cycle) ;
        short Nb_lignes(Cycle) ;
        short Indice(SousCycle) ;
        short Repeat(SousCycle) ;
        float Duree(SousCycle) ;

  } // group Cycles

group: Simulation {
  dimensions:
        Temps = UNLIMITED ; // (19168 currently)
  variables:
        float Temps(Temps) ;
        byte Regime(Temps) ;
        float T41(Temps) ;
        float e_p1(Temps) ;
        float Temps_Touche ;
  } // group Simulation
}

The generated file is 3.1 Mo and so 50% heavier than I had expected. Here is how I calculated the expected file size :

name Dimension cell size Nb cell variable size group: Regimes { float T41 Regime 32 3 96 float TTR Regime 32 3 96 float N1 Regime 32 3 96 group: Cycles {
        
                
        
byte Type Cycle 8 5 40 short Nb_lignes Cycle 16 5 80 short Indice SousCycle 16 11 176 short Repeat SousCycle 16 11 176 float Duree SousCycle 32 11 352 group: Simulation {
        
                
        
float Temps Temps 32 19168 613376 byte Regime Temps 8 19168 153344 float T41 Temps 32 19168 613376 float e_p1 Temps 32 19168 613376 float Temps_Touche 32 1 32
        
        
                
        
        
        
                total size      1994616         Bits
                                        soit    1.99    Mo



Is there anything I forgot ?

Besides, I would like to know how to if there is any efficient way to reduce this size:

- I've tried to use deflate_level=1 to 9 in nf90_def_var for the variables in the simulation group but it didn't change the file size. Is there anything else to do to to let netcdf compress data ? gzip seems to compress the resulting file quite well.

- Is there a way to store 16bits float to compress data (with loss) in order to reduce the size of data where we have a limited resolution ?

Best regards,

--
Loic





------------------------------------------------------------------------

_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/



  • 2010 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: