[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Sequence number use in pqact.conf??

On Mon, 18 Sep 2000, Michael W Dross wrote:

> 
> 
> Thanks for the info...  I will look into other methods than using a sequence
> number since that seems to be a legacy type field from the pre NOAAPORT days.
> I am interested in the -5 flag. I have 2 NOAAPORT ingest systems and feed them
> into seperate LDM's, and they come up with different MD5 checksums
> which makes it difficult to crossfeed each other for backup purposes.  If the

Mike,

The -5 flag is available now if you have control over your ingest systems
and they use pqing.  If your ingest systems are not using pqing, the code
change skip the first part of the product containing the sequence
number:

Here's the section of pqing

/*
 * Determine if a product starts with the string
 * "^A\r\r\n<sequnceNumber>\r\r\n".  If it doesn't, return.  If it does,
 * return a pointer to the start of the product, skipping over those
 * leading control chars.
 *
 * A sequence number is expected to be any string of at most
MAX_SEQ_NUM_LEN
 * digits with possibly leading or trailing blanks included in that count.
 * However, the only check done here is to see that the sequence number
 * consists of MAX_SEQ_NUM_LEN or fewer characters. No other checks are
 * performed.
 */
char *
wmo_prod(const char *prod)
{
#define PART1_SIZE 4
#define PART2_SIZE 4
#define MAX_SEQ_NUM_LEN 4

  char part1[PART1_SIZE] = {'^A', '\r', '\r', '\n'};
  char part2[PART2_SIZE] = {'\r', '\r', '\n', '\0'}; /* '\0' is for strstr
*/
  char *startPart2;
  int seqNumLength;

  /*
   * If part1 is not at start of product, return
   */
  if ((strncmp(part1, prod, PART1_SIZE)) != 0)
    return 0;

  /*
   * If part2 doesn't occur somewhere after part1, return
   */
  if ((startPart2 = strstr (prod+PART1_SIZE, part2)) == 0)
    return 0;

  /*
   * Pick out substring between part1 and part2 that contains the
   * sequence number
   */
  seqNumLength = startPart2 - (prod+PART1_SIZE);

  /*
   * Sanity check: if the length of the sequence number string is
   * too big, return
   */
  if (seqNumLength > MAX_SEQ_NUM_LEN)
    return 0;

  /*
   * If we got here, we've classified it as a wmo product.
   * Return a pointer to the beginning of the product.
   */
  return startPart2 + PART2_SIZE - 1;  /* exclude trailing '\0'  */
}


> MD5 is not using the Sequence number then it must be using just the
> WMO header, file size and system time?


It uses the WMO header and file size, the system time is not included in
the md5 checksum.

Robb...


> 
> As always, thanks alot for your help!
> 
> Mike
> 
> 
> |--------+-------------------------->
> |        |          Robb Kambic     |
> |        |          <rkambic@unidata|
> |        |          .ucar.edu>      |
> |        |                          |
> |        |          09/18/2000 12:07|
> |        |          PM              |
> |        |                          |
> |--------+-------------------------->
>   
> >----------------------------------------------------------------------------|
>   |                                                                           
>  |
>   |       To:     Michael W Dross/Cust/DukePower@DukePower                    
>  |
>   |       cc:     address@hidden, support-ldm                      |
>   |       <address@hidden>                                       |
>   |       bcc:                                                                
>  |
>   |       Subject:     Re: Sequence  number use in pqact.conf??               
>  |
>   
> >----------------------------------------------------------------------------|
> 
> 
> 
> 
> 
> 
> On Sun, 17 Sep 2000, Michael W Dross wrote:
> 
> >
> >
> > I have searched and could not find the answer... So I turn to you guys  :-)
> >
> > Is there a way in defining a pqact.conf entry that will use the sequence
> > number?? I have tried with no luck. Consider the following example.
> >
> >
> > DDPLUS  ^WHXX04 (....) ([0-3][0-9])([0-2][0-9])([0-5][0-9])
> >         FILE    -overwrite 
> > data/tropical/GFDL_Hurricane_Guidance.(\2:mm)\2\3\4
> >
> Mike,
> 
> I looked at the pqact code about the feedtype and pattern matches, here's
> a comment that states the sequence number is NOT included in the match.
> 
> * If the feedtype matches AND ((the product ID matches the
> * regular expression) OR (the pattern is "_ELSE_" AND nothing
> * has been done to this product yet AND the first char of
> * the ident isn't '_'))
> 
> This is a problem because products coming from different NOAAport ingest
> systems will look differnt because of the different sequence number.  In
> the ldm-5.1.2 release we added a -5 flag to pqing to NOT include the
> sequence number in the md5 checksum.  Therefore, the checksum will just be
> on the data section of the product and it can designate these type of
> products as duplicates.  At this time, we have not implemented this
> because it needs some synchronization between all the NOAAport ingest
> source site.  I suspect it will happen in the next couple of months.
> 
> Robb...
> 
> 
> 
> >
> > Occasionally multiple bulletins with the same time/date heading are sent and
> it
> > gets appended to the file (-overwrite doesn't prevent it
> > for some reason) the only difference is the sequence number. I thought this
> > might work, but doesn't.
> >
> > DDPLUS  ^(...)  WHXX04 (....) ([0-3][0-9])([0-2][0-9])([0-5][0-9])
> >         FILE    -overwrite
> data/tropical/GFDL_Hurricane_Guidance.\1(\3:mm)\3\4\5
> >
> >
> > It seems the entry starts with WHXX04 and the sequence number cannot be
> > referenced??
> >
> > Any ideas/suggestions would be greatly appreciated!
> >
> > Mike Dross
> >
> >
> 
> ===============================================================================
> Robb Kambic                      Unidata Program Center
> Software Engineer III                 Univ. Corp for Atmospheric Research
> address@hidden         WWW: http://www.unidata.ucar.edu/
> ===============================================================================
> 
> 
> 
> 
> 

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================