Hi all:The short answer is that version 4.3 makes everything ok. The two files look identical, because duplicates are ignored. 4.2 is broken and we cannot support it anymore for GRIB files.
As to why the ldm is getting duplicates, likely the pqact is the problem. John On 2/23/2013 6:46 AM, Signell, Richard wrote:
John, Welcome back! Wow, right from a great vacation into GRIB2 issues? Could there be anything worse? 1. This GRIB2 downloaded from the Unidata TDS can be accessed through OPeNDAP: <http://geoport.whoi.edu/thredds/catalog/usgs/data0/ldm/data/pub/native/grid/NCEP/catalog.html?dataset=usgs/data0/ldm/data/pub/native/grid/NCEP/GFS_Global_0p5deg_20130222_0000.grib2> 2. This GRIB2 from our LDM can NOT be accessed through OPeNDAP: <http://geoport.whoi.edu/thredds/catalog/usgs/data0/ldm/data/pub/native/grid/NCEP/catalog.html?dataset=usgs/data0/ldm/data/pub/native/grid/NCEP/ldm_GFS_Global_0p5deg_20130222_0000.grib2> If I look at the first file with "wgrib2", eveything looks good: $ wgrib2 ./tds/GFS_Global_0p5deg_20130222_0000.grib2 | grep 'UGRD:10 m' 221.1:40897348:d=2013022200:UGRD:10 m above ground:anl: 511.1:93268362:d=2013022200:UGRD:10 m above ground:3 hour fcst: 891.1:163581923:d=2013022200:UGRD:10 m above ground:6 hour fcst: 1210.1:221082087:d=2013022200:UGRD:10 m above ground:9 hour fcst: 1529.1:279345269:d=2013022200:UGRD:10 m above ground:12 hour fcst: ... 19886.1:3633573232:d=2013022200:UGRD:10 m above ground:174 hour fcst: 20222.1:3695640054:d=2013022200:UGRD:10 m above ground:189 hour fcst: 20669.1:3775293846:d=2013022200:UGRD:10 m above ground:192 hour fcst: But the 2nd one retrieved by LDM from the IDD doesn't proceed nicely from 0 to 192 hours -- there are a bunch of duplicated records inserted throughout the file, causing the time stamp to not proceed monotonically, and resulting in a much bigger file. This one can't be read via the opendap link on TDS (and therefore produces a zero length grib index file). on 4.2.9 $ wgrib2 ./ldm/GFS_Global_0p5deg_20130222_0000.grib2 | grep 'UGRD:10 m ' 202.1:36706943:d=2013022200:UGRD:10 m above ground:anl: 511.1:93268362:d=2013022200:UGRD:10 m above ground:3 hour fcst: 891.1:163581923:d=2013022200:UGRD:10 m above ground:6 hour fcst: 1210.1:221082087:d=2013022200:UGRD:10 m above ground:9 hour fcst: 1515.1:276755666:d=2013022200:UGRD:10 m above ground:12 hour fcst: ... 9779.1:1793637231:d=2013022200:UGRD:10 m above ground:90 hour fcst: 10059.1:1844502447:d=2013022200:UGRD:10 m above ground:93 hour fcst: 10470.1:1920009884:d=2013022200:UGRD:10 m above ground:96 hour fcst: 10686.1:1959836180:d=2013022200:UGRD:10 m above ground:anl: <== 10995.1:2016397599:d=2013022200:UGRD:10 m above ground:3 hour fcst: 11527.1:2114541141:d=2013022200:UGRD:10 m above ground:6 hour fcst: 11801.1:2163994314:d=2013022200:UGRD:10 m above ground:99 hour fcst: 12013.1:2202639475:d=2013022200:UGRD:10 m above ground:9 hour fcst: <== 12332.1:2260902657:d=2013022200:UGRD:10 m above ground:12 hour fcst: ... 34575.1:6322545271:d=2013022200:UGRD:10 m above ground:180 hour fcst: 34951.1:6390316958:d=2013022200:UGRD:10 m above ground:183 hour fcst: 35240.1:6443194232:d=2013022200:UGRD:10 m above ground:186 hour fcst: 35447.1:6479653290:d=2013022200:UGRD:10 m above ground:174 hour fcst: <== 35780.1:6541465044:d=2013022200:UGRD:10 m above ground:189 hour fcst: 36227.1:6621118836:d=2013022200:UGRD:10 m above ground:192 hour fcst: Thanks for looking into this, Rich On Fri, Feb 22, 2013 at 7:14 PM, John Caron <address@hidden> wrote:Hi Rich, can you make the zero length problem file available to us and/or the 6.2 vs 3.8 Gb GRIB file? PS, cant tell who the original questioner is. John On 2/22/2013 4:49 PM, Unidata IDD Support wrote:Rich, I'm afraid I know very little about GRIB2 files. So I've forwarded this reply to some people who know much more than I.Guys, The LDM grib file is 6.2GB, while if I download the grib2 file from the TDS: wget http://motherlode.ucar.edu:8080/thredds/fileServer/fmrc/NCEP/GFS/Global_0p5deg/files/GFS_Global_0p5deg_20130222_0000.grib2 it is only 3.8GB. What the heck? I guess I need "wgrib" or something to check these out? -Rich On Fri, Feb 22, 2013 at 6:31 PM, Unidata IDD Support <address@hidden> wrote:Rich,In the past 9 months we have had 22 files NAM files and 33 GFS files that the TDS was unable to read (produces 0 length GRIB index files). When 0 length GRIB index files get created, the TDS times out on these datasets, and our workflow is broken.Understandable, that is not OK!This is an error rate of about 10%.Well, closer to 2% (4 runs per day), unless you are only requesting one synoptic time per day, but even 2% is far greater than one would expect, we see generally 99.98 reliability..if not better.TCP unreliability is much, much less than 2%. It's far more likely that there's something wrong with the decoder or the GRIB file. Do you have something that can check the validity of a GRIB file? Regards, Steve Emmerson Ticket Details =================== Ticket ID: ZGC-437176 Department: Support IDD Priority: Normal Status: Closed-- Dr. Richard P. Signell (508) 457-2229 USGS, 384 Woods Hole Rd. Woods Hole, MA 02543-1598Regards, Steve Emmerson Ticket Details =================== Ticket ID: ZGC-437176 Department: Support IDD Priority: Normal Status: Closed