[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Happy Friday!



Hi all:

1. The CFSR dataset is known to have grib encoding issues, so its possible we are seeing some of that and/or bugs in the software. in particular, the translation of grib1 to grib2 had bugs in it. ive been unable to get authoritative info on how to fix this, but i do have machinery to do so if we can figure out what to do.

2. The 4.3 software is completely different from 4.2. We are only going to fix 4.3 grib problems. 4.2 doesnt handle time interval coordinates correctly.

3. In this file:

ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/cfs/prod/cfs/cfs.20120927/00/monthly_grib_01/flxf.01.2012092700.201305.avrg.grib.grb2

I am seeing records like the one I have dumped below.

statistic type is "Average"

reference time is 2012-09-27T00:00:00.000Z

the "end of overall time interval" is  == 2012/09/27T0:0:0

forecast time is 8 months.

we are interpreting the time interval coordinate as:

 (0.000000, 1.000000) == (2012-08-27T00:00:00.000Z, 2012-09-27T00:00:00.000Z)

which seems wrong. ill have to look at this closer.

4. Note that this is an interval coordinate, meaning the "valid time" is a time interval. From this comment:

"The forecasts from that model run are for the mean conditions over the months of May 2013"

Im guessing the correct interval should be:

 (0.000000, 1.000000) == (2013-05-01T00:00:00.000Z, 2013-06-01T00:00:00.000Z)

so:
  - we do have a bug
  - the dates are encoded wrong
  - have to figure out how to fix systematically. if theres a way to derive the correct answer from the info in the PDS below, can you see it? It will be unfortunate if we have to rely on the filename.



---------

File=0 E:/data/work/ansari/flxf.01.2012092700.201305.avrg.grib.grb2
Header=""

Grib2IndicatorSection
 Discipline = (0) Meteorological products
 Length     = 64370

Grib2IdentificationSection
 Center        = (7) US National Weather Service, National Centres for Environmental Prediction (NCEP)
 SubCenter     = (0) null
 Master Table  = 2
 Local Table   = 1
 RefTimeSignif = 1 (Start of forecast)
 RefTime       = 2012-09-27T00:00:00.000Z
 RefTime Fields = 2012-9-27 0:0:0
 ProductionStatus      = 0 (Operational products)
 TypeOfProcessedData   = 1 (Forecast products)

Grib2GridDefinitionSection hash=-1824839391 crc=4229097007
 Length             = 72
 Source  (3.0)      = 0 (Specified in Code table 3.1)
 Npts               = 72960
 Template (3.1)     = 40

(3.40) Grid definition template 3.40 - Gaussian latitude/longitude
  1:                                                                                 GDS length == 72
  5:                                                                                    Section == 3
  6:                                             Source of Grid Definition (see code table 3.0) == 0 (table 3.0: Specified in Code table 3.1)
  7:                                                                      Number of data points == 72960
 11:                                             Number of octects for optional list of numbers == 0
 12:                                                          Interpretation of list of numbers == 0 (table 3.11: There is no appended list)
 13:                                                            Grid Definition Template Number == 40
 15:                                                                         Shape of the Earth == 6 (table 3.2: Earth assumed spherical with radius of 6 371 229.0 m)
 16:                                                  Scale factor of radius of spherical Earth == 0
 17:                                                  Scaled value of radius of spherical Earth == 0
 21:                                        Scale factor of major axis of oblate spheroid Earth == 0
 22:                                        Scaled value of major axis of oblate spheroid Earth == 0
 26:                                        Scale factor of minor axis of oblate spheroid Earth == 0
 27:                                        Scaled value of minor axis of oblate spheroid Earth == 0
 31:                                                     Ni - number of points along a parallel == 384
 35:                                                     Nj - number of points along a meridian == 190
 39:                                               Basic angle of the initial production domain == 0
 43: Subdivisions of basic angle used to define extreme longitudes and latitudes, and direction increments == 0
 47:                                                         La1 - latitude of first grid point == 89277000
 51:                                                        Lo1 - longitude of first grid point == 0
 55:                                                             Resolution and component flags == 48
 56:                                                          La2 - latitude of last grid point == -89277000
 60:                                                         Lo2 - longitude of last grid point == 359062000
 64:                                                                 Di - i direction increment == 938000
 68:                                     N - number of parallels between a pole and the Equator == 95
 72:                                                                              Scanning mode == 0
 73:                                  List of number of points along each meridian or parallel. == -9999

Grib2ProductDefinitionSection
 Interval     = (2012-08-27T00:00:00.000Z,2012-09-27T00:00:00.000Z)

(4.8) Product definition template 4.8 - average, accumulation and/or extreme values or other statistically-processed values at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval
  1:                                                                                 PDS length == 58
  5:                                                                                    Section == 4
  6:                                                Number of coordinates values after Template == 0
  8:                                                         Product Definition Template Number == 8
 10:                                                                         Parameter category == 2
 11:                                                                           Parameter number == 17
 12:                                                                 Type of generating process == 2 (table 4.3: Forecast)
 13:                   Background generating process identifier (defined by originating centre) == 0 (table ProcessId: Table ProcessId code 0 not found)
 14:         Analysis or forecast generating process identifier (defined by originating centre) == 98 (table ProcessId: Table ProcessId code 98 not found)
 15:                                                  Hours after reference time of data cutoff == 0
 17:                                                Minutes after reference time of data cutoff == 0
 18:                                                            Indicator of unit of time range == 3 (table 4.4: Month)
 19:                                                 Forecast time in units defined by octet 18 == 8
 23:                                                                Type of first fixed surface == 1 (table 4.5: Ground or water surface)
 24:                                                        Scale factor of first fixed surface == 0
 25:                                                        Scaled value of first fixed surface == 0
 29:                                                               Type of second fixed surface == 255 (table 4.5: Missing)
 30:                                                       Scale factor of second fixed surface == 0
 31:                                                       Scaled value of second fixed surface == 0
 35:                                                Year - time of end of overall time interval == 2012
 37:                                               Month - time of end of overall time interval == 9
 38:                                                 Day - time of end of overall time interval == 27
 39:                                                Hour - time of end of overall time interval == 0
 40:                                              Minute - time of end of overall time interval == 0
 41:                                              Second - time of end of overall time interval == 0
 42: n - number of time range specifications describing the time intervals used to calculate the statistically-processed field == 1
 43:                                 Total number of data values missing in statistical process == 0
 47: Statistical process used to calculate the processed field from the field at each time increment during the time range == 0 (table 4.10: Average)
 48:        Type of time increment between successive fields used in the statistical processing == 2 (table 4.11: Successive times processed have same start time of forecast, forecast time is incremented)
 49:         Indicator of unit of time for time range over which statistical processing is done == 3 (table 4.4: Month)
 50: Length of the time range over which statistical processing is done, in units defined by the previous octet == 1
 54:             Indicator of unit of time for the increment between the successive fields used == 255 (table 4.4: Missing)
 55:           Time increment between successive fields, in units defined by the previous octet == 0
 59:                                      As octets 47 to 58, next innermost step of processing == -9999
 71: Additional time range specifications, included in accordance with the value of n. Contents as octets 47 to 58, repeated as necessary == -9999

Grib2SectionDataRepresentation
  Template           = 40 (Grid point data - JPEG 2000 code stream format)
  NPoints            = 72960

Grib2SectionData
  Starting Pos       = 196
  Data Length        = 64170


On 9/28/2012 11:45 AM, Steve Ansari wrote:
Hi Jack,

Thanks for bringing this up.  It appears that the GRIB decoders in the NetCDF for Java library are not detecting the time dimension correctly.  20120927 should be the runtime and 201305 should be the time, but the runtime date is showing up as the time dimension and the valid time of the model is not present.

I'm cc-ing John Caron and Robb at Unidata who are the GRIB masters at Unidata.

I'm using NCJ 4.2 in the Weather and Climate Toolkit, but I tested several files with NCJ 4.3 and got the same results.
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/cfs/prod/cfs/cfs.20120927/00/monthly_grib_01

John, Robb, thanks for your help!

Thanks!
Steve




On Fri, Sep 28, 2012 at 1:03 PM, Jack Settelmaier <address@hidden> wrote:
I'm poking around with displaying output from the climate (CFS) model, and it's forecasts of mean monthly conditions.

So, in the below files, one gets the CFS model output from the Sep 26, 2012 18Z model run.  The forecasts from that model run are for the mean conditions over the months of May and June, 2013, respectively.
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/cfs/prod/cfs/cfs.20120927/00/monthly_grib_01/flxf.01.2012092700.201305.avrg.grib.grb2
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/cfs/prod/cfs/cfs.20120927/00/monthly_grib_01/flxf.01.2012092700.201306.avrg.grib.grb2

When I Save a single KMZ after viewing each dataset, your software apparently assigns a "time stamp" based on first date in the filename, or from pulling the model initial time, NOT the valid time as I'd like.   So what I end up with in these two files
http://www.srh.noaa.gov/rtimages/srh/stsd/Jack/CFS_500HAnom4.kmz
http://www.srh.noaa.gov/rtimages/srh/stsd/Jack/CFS_500HAnom5.kmz

are two images that have the same "time" in Google, even though their valid time is a month apart.  Maybe if I made a loop over the 2 grib files it would have inserted a "time step" that would allow for some sort of animation in Google, but I'm still not sure it would help show me the valid time.

This is pretty specific to the CFS files and their nuanced forecasts and such, but is there something that could be done to better the result?