[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20040722: 20040720: 20040719: 20040719: sending EXP data to another LDM server

Ying,

The pqinsert suceeded in entering the data into the queue. The sequence
number is 000. The file is 7MB. It will take a few seconds for 7MB to get to 
your downstream. Your test.exp file would have its timestamp changed if
the data arrived and was processed.

You did not show the notifyme command, but you must use the -o option, unless it
is already connected before you run pqinsert.

On spring.scs you would run "notifyme -vxl - -o 3600 -f EXP -h 
unidata.scs.gmu.edu"
to see the data on unidata.scsjjjjjjjj. 

Check you ldmd.log file on unidata.scs amd verify that you see the connection
from spring.scs when the ldm is started up (remember, you have to restart your
ldm on the machine if you edit its ldmd.conf before changes will take effect). 

Check your ldmd.conf file on spring to see if it has any trouble connecting
to the upstream.

You can also run "ldmadmin watch -f EXP" on spring to watch data arriving
in your queue.

Also, if you edit your pqact.conf file to add the FILE action, you have to
run "ldmadmin pqactHUP" to reread theconf file into a running pqact process.

Steve Chiswell
Unidata User Support





>From: address@hidden
>Organization: UCAR/Unidata
>Keywords: 200407221704.i6MH45aW000266

>Steve,
>
>I'm sorry I still can not get through.
>On my machine (upstream site)
>I inserted the data.  Then I use notifyme, it returns:
>pqinsert:  7226077 20040722161021.934     EXP 000  /usr/local/ldm/data/doqq/38
> 077g35.sid
>-- what does the 000 mean ( right after EXP)? 
>
>Immediately I check the downstrem site (spring.scs.gmu.edu) by notifyme, but t
> here is no result returning
>
>In the ldmd.conf on my machine, I added:
>ALLOW   EXP     spring.scs.gmu.edu
>
>In the ldmd.conf on spring.scs.gmu.edu (downstream site), I added:
>request EXP ".*" unidata.scs.gmu.edu
>
>And in pqact.conf I wrote:
>EXP     ^.*
>        FILE    -overwrite
>        /export/home0/ldm/data/EXP/test.exp
>
>I doubt if the data product arrived on spring.scs.gmu.edu. Really appreciate y
> our help!
>
>And as to the the sequence of library calls to insert the product into the que
> ue, I hope I can get it if possible. Thanks
>
>regards,
>
>Ying
>
>
>----- Original Message -----
>From: Unidata Support <address@hidden>
>Date: Wednesday, July 21, 2004 11:42 am
>Subject: 20040720: 20040719: 20040719: sending EXP data to another LDM server
>
>> 
>> Ying,
>> 
>> When you start notifyme, it asks for information from the current 
>> time.
>> Since you had already inserted the data into your queue, you would 
>> want to 
>> use the "-o" option to tell notifyme to start at an offset time, 
>> such as:
>> notifyme -vxl - -f EXP -o 3600
>> The -o 3600 would tell notifyme to ask for information on products 
>> since 2600 seconds ago.
>> 
>> For placing the data into the upstream queue, you could either
>> use a script as you mention, or you could use the LDM library if 
>> you have
>> access to the code that reads the data from your satellite feed.
>> At the point where you have a file to write to disk now, you
>> could use the LDM library calls to calculate the MD5 checksum and 
>> insertthe product into the LDM queue from within your program that 
>> reads the data from
>> your dish.
>> 
>> Pqing is an example of a program that can read from a stream and 
>> insert the data 
>> into the queue.  If you choose to use the LDM library from within your
>> program, let us now and we will provide you with the sequence of
>> library cals to insert the product into the queue.
>> 
>> Steve Chiswell
>> Unidata User Support
>> 
>> 
>> 
>> >From: address@hidden
>> >Organization: UCAR/Unidata
>> >Keywords: 200407202006.i6KK63aW006462
>> 
>> >Steve,
>> >
>> >I used pqinsert for three files
>> >[ldm@unidata ~/bin]$ ./pqinsert -v -l - -f EXP 
>> ~/data/doqq/38077g35.sdw>pqinsert:      186 20040720193223.205     
>> EXP 000  /usr/local/ldm/data/doqq/38
>> > 077g35.sdw
>> >[ldm@unidata ~/bin]$ ./pqinsert -v -l - -f EXP 
>> ~/data/doqq/38077g35.sid>pqinsert:  7226077 20040720193229.998     
>> EXP 000  /usr/local/ldm/data/doqq/38
>> > 077g35.sid
>> >[ldm@unidata ~/bin]$ ./pqinsert -v -l - -f EXP 
>> ~/data/doqq/38077g35.hdr>pqinsert:    18696 20040720193237.931     
>> EXP 000  /usr/local/ldm/data/doqq/38
>> > 077g35.hdr
>> >
>> >but when I used notifyme
>> >
>> >[ldm@unidata ~/bin]$ ./notifyme -vxl- -f EXP -h localhost
>> >Jul 20 19:33:06 notifyme[388]: Starting Up: localhost: 
>> 20040720193306.642 TS_E
>> > NDT {{EXP,  ".*"}}
>> >Jul 20 19:33:06 notifyme[388]: Connected to upstream LDM-5
>> >        NOTIFYME(localhost) returns OK
>> >Jul 20 19:33:06 notifyme[388]: NOTIFYME(localhost): OK
>> >
>> >I can not see the info about the data product. Are they in the 
>> product queue?
>> >
>> >Also I want to know how does upstream site do when they get new 
>> data from i.e.
>> >  antenna. They have some scripts to insert these data to product 
>> queue automa
>> > tically so that downstream sites are able to get the real time 
>> data, right? C
>> > an I get the script, if there is? Coz we want to share some 
>> satellite data wh
>> > ich is obtained directly from antenna.
>> >
>> >Thanks!
>> >
>> >Ying
>> >
>> >
>> >----- Original Message -----
>> >From: Unidata Support <address@hidden>
>> >Date: Monday, July 19, 2004 6:45 pm
>> >Subject: 20040719: 20040719: sending EXP data to another LDM server
>> >
>> >> 
>> >> Ying,
>> >> 
>> >> Sorry, my typo. Don't use pqing (which is meant to read WMO 
>> format 
>> >> bulletins).Instead, use pqinsert which will insert a file into 
>> the 
>> >> queue.
>> >> 
>> >> Steve Chiswell
>> >> Unidata User Support
>> >> 
>> >> 
>> >> 
>> >> 
>> >> 
>> >> 
>> >> >From: address@hidden
>> >> >Organization: UCAR/Unidata
>> >> >Keywords: 200407192318.i6JNIDaW001884
>> >> 
>> >> >Hi Steve,
>> >> >
>> >> >Thanks a lot !
>> >> >I store the files38077g35.hdr, 38077g35.sdw, 38077g35.sid  at 
>> >> ~/data/doqq/, an
>> >> > d use the pqing as below
>> >> >for file 38077g35.sid
>> >> >[ldm@unidata doqq]$ ~/bin/pqing -v -l - -f EXP 
>> >> ~/data/doqq/38077g35.sid>It returns:
>> >> >Jul 19 22:32:32 pqing[31751]: Starting Up
>> >> >Jul 19 22:32:32 pqing[31751]: FILE 
>> >> "/usr/local/ldm/data/doqq/38077g35.sid">Jul 19 22:32:32 
>> >> pqing[31751]: Lone SOH error                          3
>> >> >Jul 19 22:32:32 pqing[31751]: scan_wmo: length 5 too short
>> >> >Jul 19 22:32:32 pqing[31751]: Lone ETX error                   
>>   
>> >>  108       
>> >> >   -01   X Q00      192232
>> >> >Jul 19 22:32:32 pqing[31751]: Lone SOH error                   
>>   
>> >>   33       
>> >> >   -01      00      192232
>> >> >Jul 19 22:32:32 pqing[31751]: Lone SOH error                   
>>   
>> >>   25       
>> >> >   -01      00      192232
>> >> >Jul 19 22:32:32 pqing[31751]: Lone SOH error                   
>>   
>> >>    5       
>> >> >   -01      00      192232
>> >> >Jul 19 22:32:32 pqing[31751]: Lone SOH error                   
>>   
>> >>    5       
>> >> >   -01      00      192232
>> >> >Jul 19 22:32:32 pqing[31751]: Lone SOH error                   
>>   
>> >>    2
>> >> >Jul 19 22:32:32 pqing[31751]: Lone SOH error                   
>>   
>> >>    3
>> >> >Jul 19 22:32:32 pqing[31751]: scan_wmo: length 3 too short
>> >> >Jul 19 22:32:32 pqing[31751]: Lone SOH error                   
>>   
>> >>    2
>> >> >Jul 19 22:32:32 pqing[31751]: Lone SOH error                   
>>   
>> >>    7       
>> >> >   -01      00      192232
>> >> >Jul 19 22:32:32 pqing[31751]: Lone SOH error                   
>>   
>> >>    5       
>> >> >   -01      00      192232
>> >> >Jul 19 22:32:32 pqing[31751]: Lone SOH error                   
>>   
>> >>    6       
>> >> >   -01      00      192232
>> >> >Jul 19 22:32:32 pqing[31751]: scan_wmo: length 16 too short
>> >> >Jul 19 22:32:32 pqing[31751]: Lone SOH error                   
>>   
>> >>    3
>> >> >Jul 19 22:32:32 pqing[31751]: scan_wmo: length 14 too short
>> >> >...
>> >> >...
>> >> >
>> >> >for file 38077g35.hdr, it returns
>> >> >[ldm@unidata doqq]$ ~/bin/pqing -v -l - -f EXP 
>> >> ~/data/doqq/38077g35.hdr>Jul 19 22:47:42 pqing[31754]: Starting Up
>> >> >Jul 19 22:47:42 pqing[31754]: FILE 
>> >> "/usr/local/ldm/data/doqq/38077g35.hdr">Jul 19 22:47:42 
>> >> pqing[31754]: Expanding input buffer size to 32768
>> >> >Jul 19 22:47:42 pqing[31754]: Exiting
>> >> >Jul 19 22:47:42 pqing[31754]:   Queue usage (bytes):1048576000
>> >> >Jul 19 22:47:42 pqing[31754]:            (nregions):  182025
>> >> >Jul 19 22:47:42 pqing[31754]:   Duplicates rejected:       0
>> >> >Jul 19 22:47:42 pqing[31754]:   WMO Messages seen:         0
>> >> >Jul 19 22:47:42 pqing[31754]:   SOH/ETX missing  :         0
>> >> >Jul 19 22:47:42 pqing[31754]:   parity/chksum err:         0
>> >> >Jul 19 22:47:42 pqing[31754]:   WMO format errors:         0
>> >> >Jul 19 22:47:42 pqing[31754]:   FILE Bytes read:       18696
>> >> >
>> >> >for file 38077g35.swd, similar to the previous result.
>> >> >
>> >> >when I type notifyme, it is:
>> >> >[ldm@unidata doqq]$ ~/bin/notifyme -vxl- -f EXP -h 
>> >> unidata.scs.gmu.edu>Jul 19 22:48:42 notifyme[31755]: Starting 
>> Up: 
>> >> unidata.scs.gmu.edu: 20040719224
>> >> > 842.487 TS_ENDT {{EXP,  ".*"}}
>> >> >Jul 19 22:48:42 notifyme[31755]: Connected to upstream LDM-5
>> >> >        NOTIFYME(unidata.scs.gmu.edu) returns OK
>> >> >Jul 19 22:48:42 notifyme[31755]: 
>> NOTIFYME(unidata.scs.gmu.edu): OK
>> >> >
>> >> >Did I successfully pqing the files?  Anything wrong?
>> >> >
>> >> >Another question is,  does it mean everytime I get a new file 
>> >> from i.e. antenn
>> >> > a, I need to pqing it to product queue manually? Is  there 
>> any 
>> >> way to insert 
>> >> > it automatically once I put it under a certain folder?
>> >> >
>> >> >Sorry for so many questions and thank you very much for you help!
>> >> >
>> >> >Ying
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >----- Original Message -----
>> >> >From: Unidata Support <address@hidden>
>> >> >Date: Monday, July 19, 2004 2:37 pm
>> >> >Subject: 20040719: sending EXP data to another LDM server
>> >> >
>> >> >> 
>> >> >> Ying,
>> >> >> 
>> >> >> You can use a single "pqinsert" command to insetyour 3 files 
>> in 
>> >> your 
>> >> >> list.
>> >> >> The LDM product name of these files will be file name (uless you
>> >> >> specify anothr name). If you specify a full path to the files,
>> >> >> then that would be part of the name as well. You can use the
>> >> >> notifyme command to seee that the files are in your local queue.
>> >> >> 
>> >> >> To receive the files at another LDM, you should:
>> >> >> 
>> >> >> 1) Makefile sure your ~ldm/etc/ldmd.conf file has an allow line
>> >> >> for the other LDM host. Ensure that EXP is part of the feedtype
>> >> >> lit that you are allowing that host to receive.
>> >> >> 
>> >> >> 2) On the other LDM, make sure the ~ldm/etc/ldmd.conf
>> >> >> has a "request" line for the EXP feedtype from your ldm.
>> >> >> You can either use the ".*" pattern to match all EXP products,
>> >> >> or a pattern of file names to request. 
>> >> >> 
>> >> >> 3) On this LDM, you can have pqact.conf actions to automatically
>> >> >> process the files as they arrive.
>> >> >> 
>> >> >> Thats it.....
>> >> >> 
>> >> >> 
>> >> >> Steve Chiswell
>> >> >> Unidata User Support
>> >> >> 
>> >> >> 
>> >> >> >From: address@hidden
>> >> >> >Organization: UCAR/Unidata
>> >> >> >Keywords: 200407191635.i6JGZ3aW023824
>> >> >> 
>> >> >> >Hello,
>> >> >> >
>> >> >> >I've got LDM running well and am trying to send our own 
>> data 
>> >> >> (DOQQ) to another
>> >> >> >  LDM server as EXP data. But I'm not sure about the whole 
>> >> >> process. 
>> >> >> >For example, the data is in three related files:  
>> >> >> >38077g35.hdr, 
>> >> >> >38077g35.sdw, 
>> >> >> >38077g35.sid
>> >> >> >Do I need to write three "pqinsert" commands to insert them 
>> to 
>> >> >> product queue?
>> >> >> >
>> >> >> >How can I make the data automatically arrive at another LDM 
>> >> >> server once I put 
>> >> >> > new data into corresponding directory on my server? 
>> Anything 
>> >> I 
>> >> >> need to add in
>> >> >> >  pqact.conf, etc?
>> >> >> >
>> >> >> >Thank you in advance for your help!
>> >> >> >
>> >> >> >Ying 
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> --
>> >> >> 
>> >> 
>> ****************************************************************************
>> >> >> Unidata User Support                                    UCAR 
>> >> >> (303)497-8643                                                
>>  
>> >> >> address@hidden                                   
>> >> >> -------------------------------------------------------------
>> ---
>> >> ---
>> >> >> Unidata WWW Service              
>> >> >> -------------------------------------------------------------
>> ---
>> >> ---
>> >> >> NOTE: All email exchanges with Unidata User Support are 
>> >> recorded 
>> >> >> in the
>> >> >> Unidata inquiry tracking system and then made publically 
>> available>> >> through the web.  If you do not want to have your 
>> interactions made
>> >> >> available in this way, you must let us know in each email 
>> you 
>> >> send 
>> >> >> to us.
>> >> >> 
>> >> >
>> >> --
>> >> 
>> ****************************************************************************
>> >> Unidata User Support                                    UCAR 
>> >> (303)497-8643                                                  
>> >> address@hidden                                   
>> >> ----------------------------------------------------------------
>> ---
>> >> Unidata WWW Service              
>> >> ----------------------------------------------------------------
>> ---
>> >> NOTE: All email exchanges with Unidata User Support are 
>> recorded 
>> >> in the
>> >> Unidata inquiry tracking system and then made publically available
>> >> through the web.  If you do not want to have your interactions made
>> >> available in this way, you must let us know in each email you 
>> send 
>> >> to us.
>> >> 
>> >
>> --
>> ****************************************************************************
>> Unidata User Support                                    UCAR 
>> (303)497-8643                                                  
>> address@hidden                                   
>> -------------------------------------------------------------------
>> Unidata WWW Service              
>> -------------------------------------------------------------------
>> NOTE: All email exchanges with Unidata User Support are recorded 
>> in the
>> Unidata inquiry tracking system and then made publically available
>> through the web.  If you do not want to have your interactions made
>> available in this way, you must let us know in each email you send 
>> to us.
>> 
>
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publically available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.