[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

19990517: Conduit AVN data feed


William,

The dostats command will produce a ldmstats.upc file and mail that off
here. It will keep ths last 48 hours of raw files around- so you should
be seeing it scour off those by now. The stats files are very small-
so 48 of them should not cause space prablems. Their purpose is
to monitor the data arrival and determine how long it takes
for the data to arrive.

In regards to your EXEC problem; you can have as many actions that match
a pattern as you wish- order is not important. The problem with your
EXEC line is likely that you are trying to impose a shell level
redirection ">" rather than expecting pqact to use an execvp type
of process execution. You could create a simple shell script
that simply accepts as command line input the date string you wish
and have that script echo the data to an output file. An alternative
is to exec "touch" to touch a file named appropriately with
the date.

In general, the program that injects the model data into the IDD stream
finishes by creating the .status message that conttains the file
size and date that the process finished sending the data. The
.status file should arrive "near" the end of the data, however,
since the data uses multiple simultaneous pipes, you cannot guarantee
that the grib pieces will not all be received prior to the .status
file. I know this seems a little awkward- but T1 out of ncep is
the smallest pipe so 3 processes are splitting up the load of the
transmission out, so the .status can come across before the
last piece of data - though you can use the .status to exec a process
which might wait until the output file hasn't been written to
for say ~5 minutes before moving it behind the fire wall.

As for scouring, since the scour script is run from cron, you can edit
the scout.conf at any time and the next time it is run from cron,
your changes will take effect.

Since the pqact program is a daemon process, if you edit the pqact.conf
file, you have to send the pqact process a "kill -HUP" signal to
tell the program to reread the configuration file.

If you make changes to the ldmd.conf file which is only used at startup,
then you would have to shut down the running ldm processes and restart.


The ruc2 data is 0-3 hour forecasts every hour, and every three hours
will include the 6,9,12 hour forecast fields, so as you mention, it
is a good choice to use as a test data set to learn how to manipulate
the data system.

Steve Chiswell
Unidata User Support

On Fri, 14 May 1999, William Smith wrote:

> Steve,
> 
> 
> > I verified that you are receiving this mornings AVN data.
> 
> Thank you for your help, and Rob too.  
> 
> Here's the latest and some more questions.
> 
> 
> I was able to get the 00Z and the 12Z AVN data.  I might have
> had it the nite before, but there was a problem with the crontabs
> using HST and the ldm user seeing everything in GMT.  That meant
> the scour and other crontabs ran 10 hours different than what I
> expected.  They're all synced up now.
> 
> I used the watch command and saw the last of the 12Z run come
> in this morning.  
> 
> Leading to a question: the dostats is set to run at XXhour:35.
> I assume it does so, but don't know how to check it.  It doesn't
> seem to remove any of the .stats files from the data/logs directory.
> Is that supposed to happen automagically?  If not when do people
> usually remove them?  If so how do I set that up?
> 
> I've been running some checks using RUC data in small amounts
> for a specific data type and only the F06 forecast hour.  Since
> that comes in more frequently and in small doses I've used it to
> discover more about how this all works.  It's very impressive.
> 
> One of the things I'm trying to do is figure out when the data
> for a run has all been received.  I've tried to sync on the
> .status. record but without success.
> 
> This is the last two records from the 12Z AVN run this morning:
> 
> May 14 17:27:54 pqutil:     8228 19990514162822.294    NMC2 256 
> /u/ftp/gateway/ncepb/avn/avn.990514/gblav.T12Z.PGrbF84 
> !grib/ncep/SPEC_126_AVN/
> #003/199905141200/F084/LAND/sfc! 000256
> 
> May 14 17:27:54 pqutil:      109 19990514162822.801    NMC2 280 
> .status./u/ftp/gateway/ncepb/avn/avn.990514/gblav.T12Z.PGrbF84 000280
> 
> The .status. record was the last thing in.  Here's what I tried to do
> using the pqact.conf file.
> 
>       NMC2    avn/avn.(......)/gblav.T00Z
>               FILE    data/avn/19\1\2.grib
>       NMC2    \.status\..*avn/avn.(......)/gblav.T00Z
>               EXEC    echo 19\1\2 > data/currentdate
> 
> I'm trying to make a file called currentdate that contains the date and
> time of the AVN run.  Based on the .status. record I saw this morning
> pqact should have seen it and done something.  Did the FILE action
> gobble the record so it didn't make it to the EXEC action?
> 
> Will it work if I switch the two commands?
> 
> I'd like to know when the file is complete so I can ftp it behind
> our firewall.
> 
> If I make a change to pqact.conf or scour.conf do I need to stop and
> restart the LDM?  I know I have to do that for a change to the ldm.conf.
> 
> 
> Thanks again to you and Rob for all the help in getting me this far.
> 
> 
> 
> 
> William
> 
> _____________________________________________________________________________
> William Smith                             address@hidden
> Maui High Performance Computing Center    (808) 879-5077  
> 550 Lipoa Parkway                         (808) 879-5018 (fax)
> Kihei, HI  96753                          WWW: http://www.mhpcc.edu
> _____________________________________________________________________________
> 
>