[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

(Fwd) Re: 19990429: Product Duplicates/Checksum calculation

--- Forwarded mail from Paul Hamer <address@hidden>

Date: Fri, 30 Apr 1999 21:28:17 +0000
From: Paul Hamer <address@hidden>
To: "Glenn P. Davis" <address@hidden>
CC: Paul Hamer <address@hidden>
Subject: Re: 19990429: Product Duplicates/Checksum calculation


Glenn,

Thanks, as always, for your advice. We had already implemented the
configuration solution, i.e. making sure that we insert the subset
into another LDM.

I took a look at the other suggestions (updating the compare to include
the signature and something else, modify pqsend). The first appeals
because the existing interface doesn't change. But it would seem difficult
to implement. The pqsend option would not be appropriate since the filtering
we do requires examination of the data itself and not just the ID.

I was hoping to be able to send you some modified code with the comparision
change implemented but I don't think I'll be able to find the time to
do that for a while. Fingers crossed that you guys will implement something
along those lines for some future release? :-)

Thanks again. It's always fun (for me at least) to look closely at LDM.

Paul.



"Glenn P. Davis" wrote:
>
> > >To: address@hidden
> > >cc: Paul Hamer <address@hidden>
> > >From: Paul Hamer <address@hidden>
> > >Subject: Product Duplicates
> > >Organization: .
> > >Keywords: 199904292059.OAA29086
> >
> >
> > Hi there,
> >
> > We're running LDM on our platforms here at FSL and have just set up
> > a configuration that has highlighted a small problem for us.
> >
> > We receive a feed from an external agency that we filter internally
> > and re-identify a subset of these data with a new feedtype and indent.
> > The MD5 checksum only works over the actual data part which means that
> > the signiture remains the same. So now we can't now insert the newly
> > identified data back the queue for distribution to someone else because
> > LDM has it down as a duplicate.
> >
> > What would seem sensible would be to calc the checksum over the
> > entire product structure but since that's not done do you have
> > any suggestions?
> >
> > Thanks,
> >
> > Paul.
> >
> >
> > --
> > Paul Hamer
> > Email: address@hidden
> > Phone: 303.497.6342
>
> Paul:
>
> I think I would run the ldm which gets data from the
> "external agency" in isolation. (You are probably doing this
> already.) I would modify 'pqsend' to do the subsetting and
> re-identifying, and send the re-identified subset to
> another machine. The main idea is avoid sticking the re-ident
> product back in the same queue.
>
> > What would seem sensible would be to calc the checksum over the
> > entire product structure
>
> It is a matter of semantics. It is supposed to be a checksum of
> the cargo. I think I would leave it that way and change the
> "equality" test from "cargo the same" to "cargo and ..."
> The reason we do it this way is historical. Now that we have
> NOAAPort/AWIPS, we have reliable sequence numbers so we can
> do a lot of things differently.
>
> Others have been bitten by this as well.
>
> -glenn

--
Paul Hamer
Email: address@hidden
Phone: 303.497.6342


---End of forwarded mail from Paul Hamer <address@hidden>