[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [conduit] CONDUIT change dates for GRIB2 NAM files

Mike,

The GRIB2 NAM 40km data files are approximately 6 MB in size. For the 29
forecast hours available, that multiplies out to 174 MB. Thats the
beauty of using GRIB2
versus GRIB1! We'll save bandwidth, and disk space when storing using
the native
GRIB2 packing.

If you were decoding both GRIB1 and GRIB2 to the same file, then the
file would expand to the size of the GRIB1 data products (about 465 MB)
since that would
be the space allocated for each parameter.

Sounds like you have all your expected data. I'll look
for any missing parameters in the tables, but mostly they end up being 
cloud level parameters or the such, so your plots probably won't be
showing any
common missing parameters,

Steve Chiswell
Unidata User Support





On Wed, 2007-10-24 at 10:28 -0700, Mike Voss wrote:
> Steve,
> Thanks for your comments. My mistake.....I had been looking at 
> "pact.conf" instead of my "pqact.gempak_decoders" so I was fooled on 
> which actions were doing the decoding.
> 
> I have installed GEMPAK 5.10.4 and I'm decoding the grib2 portion of 
> nam212 with the following entry:
> ------------------------
> # NAM #212 40km grid CONUS
> CONDUIT prod/nam.*awip3d.*grib2
>          PIPE    decoders/dcgrib2 -d data/logs/dcgrib2_NAM212x.log
>                  -e GEMTBL=/usr/local/gempak/GEMPAK5.10.4/gempak/tables
> ---------------------------
> Here are the files from 06Z and 12Z  run today, they appear to be 
> short on size (typically ~450 MB):
> ---------------------------
> -rw-rw-r-- 1 ldm ldm 176100864 Oct 24 15:02 2007102412_nam212.gem
> -rw-rw-r-- 1 ldm ldm 175341568 Oct 24 08:59 2007102406_nam212.gem
> -----------------------------
> 
> And looking at the log file "dcgrib2_NAM212x.log" I can see there are 
> parameters not being decoded:
> http://www.met.sjsu.edu/files/gempak/dcgrib2_NAM212x.log
> 
> However, when I do GDINFO on the file it looks like everything is 
> there. And when I plot data, it appears to be all there.
> 
> Could you provide some insight on this? Am I really missing some 
> stuff? Just want to be sure before the cut over.
> 
> Thanks,
> 
> -Mike
> 
> 
> 
> 
> At 08:35 AM 10/18/2007, Steve Chiswell wrote:
> >On Wed, 2007-10-17 at 13:40 -0700, Mike Voss wrote:
> > > Steve,
> > >
> > > Thanks, sorry for the confusion. I should have asked this question:
> > > Will the general "pqact" entry for all NAM output still work?
> > >
> > > -----------------------------
> > > HDS|CONDUIT|NGRID      (/mETA|/mNAM|MT.nam|^[LM]..... KWBE)
> > >         PIPE    decoders/dcgrib2 -d data/logs/dcgrib2_ETA.log
> > >                 -e GEMTBL=/usr/local/gempak/GEMPAK5.10.2/gempak/tables
> > > -----------------------------
> > >
> > > And related.... if I am now receiving both GRIB1 and GRIB2 versions
> > > of NAM212 40KM via CONDUIT, is it decoding both? I only see one file:
> > > ------------------------------------
> > > -rw-rw-r-- 1 ldm ldm 460025856 Oct 17 15:08 2007101712_nam212.gem
> > > ------------------------------------
> > >
> > > thanks again,
> > >
> > > -Mike
> >
> >Mike,
> >
> >Your pqact action above is NOT decoding any CONDUIT NAM  data.
> >The MT.nam pattern was used to match NAM data when the CONDUIT source
> >was the NWS ftp servers, not the NCEP ftp servers.
> >
> >The general NAM product file naming used on CONDUIT is shown here:
> >http://www.unidata.ucar.edu/data/conduit/ldm_idd/nam_files.html
> >
> >A commented out "catch all" pattern distributed with GEMPAK 5.10.4 in
> >$NAWIPS/ldm/etc/templates/pqact.gempak_decoders.in is:
> >
> ># If you want to decode all NAM data currently available, use this action
> ># instead of the individual decoder lines shown below.
> >#HDS|CONDUIT|NGRID      (/mNAM|nccf/com/nam|^[LM]..... KWBE)
> >#       PIPE    decoders/dcgrib2 -d data/gempak/logs/dcgrib2_NAM.log
> >#               -e GEMTBL=@GEMTBL@
> >
> >I do not use a catch all pattern here due to the large number of
> >different grids that the
> >NAM is being transmitted on in the 3 different IDD data streams. Using
> >the
> >individual default patterns distributes the decoder load to different
> >processes.
> >
> >One point of clarification is that the GRIB2 data does not define the
> >grid number 212
> >within the data products! That means that the @@@ file name template
> >cannot
> >be determined solely from the GRIB2 data itself, which the above pattern
> >will need.
> >The dcgrib2 program in GEMPAK 5.10.4 will use the
> >$GEMTBL/grid/grdnav.tbl
> >to figure out the grid number 212, given the navigation within the GRIB2
> >data, so
> >the above pattern will work with 5.10.4. Older versions will use 255 as
> >the missing
> >identifier for the @@@ template, so the "catch all" pattern is not the
> >best solution
> >for older installations.
> >
> >The following pattern distributed with GEMPAK 5.10.4 will store
> >all the NAM #212 into a single file determined by the gribkey.tbl
> >template file:
> >#
> ># NAM #212 40km grid CONUS (5.10.4 and later)
> >HDS|CONDUIT     ((/mNAM|/mNMM).*#212|prod/nam.*awip3d)
> >         PIPE    decoders/dcgrib2 -d data/gempak/logs/dcgrib2_NAM212.log
> >                 -e GEMTBL=@GEMTBL@
> >
> >For older distributions where the grdnav.tbl isn't used to determine the
> >grid number for @@@ templates in GRIB2 decoding, you should provide the
> >output file name without using @@@ such as:
> >
> >#
> ># NAM #212 40km grid CONUS (pre 5.10.4)
> >HDS|CONDUIT     ((/mNAM|/mNMM).*#212|prod/nam.*awip3d)
> >         PIPE    decoders/dcgrib2 -d data/gempak/logs/dcgrib2_NAM212.log
> >                 -e GEMTBL=@GEMTBL@
> >                 data/gempak/model/nam/YYYYMMDDHH_nam212.gem
> >
> >The best solution is to have all 40km NAM data being written to a single
> >decoded
> >file regardless of data stream source so that the analysis and display
> >program users
> >don't have to know where the data originated.
> >
> >Steve Chiswell
> >Unidata User Support
> >
> >
> >
> > >
> > >
> > >
> > > At 01:17 PM 10/17/2007, Steve Chiswell wrote:
> > > >Mike,
> > > >
> > > >The changes announced for the CONDUIT stream do not affect the NOAAport
> > > >NGRID data feed. The pattern you have below is used for the GRIB2 format
> > > >12km data broadcast on the NWSTG2 channel of NOAAport which is provided
> > > >in the
> > > >NGRID feed of the IDD. Note that data in NGRID is already GRIB2 format.
> > > >
> > > >The CONDUIT awip3d files are 40km grids. CONDUIT also provides
> > > >the awip12 12km grids in GRIB2 format as well, and is not
> > > >affected by the changes from GRIB1 to GRIB2 for the awipak, grbgrd,
> > > >and awip3d grids this month.
> > > >
> > > >Steve Chiswell
> > > >Unidata User Support
> > > >
> > > >
> > > >
> > > >On Wed, 2007-10-17 at 13:10 -0700, Mike Voss wrote:
> > > > > Steve,
> > > > > Could you clarify something for me (maybe others have the same
> > > > > question). I am currently decoding GRIB1 NAM awip3d files with
> > > > > "dcgrib2". Am I correct to assume the GRIB2 NAM awip3d files will be
> > > > > decoded properly with the same "pqact" entry"
> > > > > ---------------
> > > > > # ETA/NAM #218 12km grid CONUS
> > > > > NGRID   ^[LM].B... KWBE
> > > > >          PIPE    decoders/dcgrib2 -d data/logs/dcgrib2_ETA218.log
> > > > >                  -e 
> > > > > GEMTBL=/usr/local/gempak/GEMPAK5.10.2/gempak/tables
> > > > > ---------------------------
> > > > >
> > > > > thanks,
> > > > >
> > > > > -Mike
> > > > >
> > > > >
> > > > >
> > > > > At 12:51 PM 10/17/2007, Steve Chiswell wrote:
> > > > > >CONDUIT Data users,
> > > > > >
> > > > > >On Nov 6, 2007, GRIB2 versions of GFS 1.0 and 2.5 degree data will be
> > > > > >added to CONDUIT. A two week overlap with the existing GRIB1 
> > data files
> > > > > >will be maintained.
> > > > > >
> > > > > >On Nov 20, 2007, the GRIB1 versions of GFS 1.0 and 2.5 
> > degree data will
> > > > > >be
> > > > > >removed from CONDUIT.
> > > > > >
> > > > > >The data files affected are the GFS files gfs.tHHz.pgrbf*  where "*"
> > > > > >represents:
> > > > > >
> > > > > >anl, 00 through 180: 1.0 degree grid #003
> > > > > >192 through 384: 2.5 degree grid #002
> > > > > >
> > > > > >The GRIB2 versions of these files are available on the NCEP ftp 
> > > > > >server
> > > > > >and have the
> > > > > >same files names with the ".grib2" appended to the above.
> > > > > >
> > > > > >This change does not affect the 0.5 degree grid #004 data already
> > > > > >available on CONDUIT in GRIB2 format ( eg the "pgrb2f" files 
> > rather than
> > > > > >the "pgrbf" as above).
> > > > > >
> > > > > >Reminder: The NAM awip3d files are currently being sent in both GRIB1
> > > > > >and GRIB2 format. The duplicate GRIB1 data will be removed on Oct 30,
> > > > > >2007.
> > > > > >
> > > > > >Please refer any questions to address@hidden,
> > > > > >
> > > > > >Steve Chiswell
> > > > > >Unidata User Support
> > > > > >
> > > > > >
> > > > > >--
> > > > > >Steve Chiswell <address@hidden>
> > > > > >Unidata
> > > > > >_______________________________________________
> > > > > >conduit mailing list
> > > > > >address@hidden
> > > > > >For list information or to unsubscribe, visit:
> > > > > >http://www.unidata.ucar.edu/mailing_lists/
> > > > >
> > > > > --------------------------
> > > > > Mike Voss
> > > > > Department of Meteorology
> > > > > San Jose State University
> > > > > One Washington Square
> > > > > San Jose, CA 95192-0104
> > > > >
> > > > > 408.924.5204 voice
> > > > > 408.924.5191 fax
> > > >--
> > > >Steve Chiswell <address@hidden>
> > > >Unidata
> > >
> > > --------------------------
> > > Mike Voss
> > > Department of Meteorology
> > > San Jose State University
> > > One Washington Square
> > > San Jose, CA 95192-0104
> > >
> > > 408.924.5204 voice
> > > 408.924.5191 fax
> > >
> > > _______________________________________________
> > > conduit mailing list
> > > address@hidden
> > > For list information or to unsubscribe, visit: 
> > http://www.unidata.ucar.edu/mailing_lists/
> >--
> >Steve Chiswell <address@hidden>
> >Unidata
> 
> --------------------------
> Mike Voss
> Department of Meteorology
> San Jose State University
> One Washington Square
> San Jose, CA 95192-0104
> 
> 408.924.5204 voice
> 408.924.5191 fax    
-- 
Steve Chiswell <address@hidden>
Unidata