[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20000913: Multiple feed types, dupilicate data, and downstreamclients (fwd)


===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================

---------- Forwarded message ----------
Date: Fri, 15 Sep 2000 14:24:44 -0600 (MDT)
From: Robb Kambic <address@hidden>
To: Steven Danz <address@hidden>
Subject: Re: 20000913: Multiple feed types, dupilicate data,
     and   downstreamclients

On Fri, 15 Sep 2000, Steven Danz wrote:

> Ok, I've taken the approach to have both upstream sources of data
> use feed types of WMO.  The question I have is that the intermediate
> server .stats files show IDS|HDS|DDPLUS for one server and WMO
> for the other.  Is there a problem requesting the same feedtype from
> more than one server, or is this something 'special' with using WMO?
> 
> Thanks!
> 
> Steven

Steven,

The feedtype is designate at the product inject time, so if some products
are injected as DDPLUS that will be the feedtype through the whole system.
I suspect that one ingestor is using feedtype WMO and the other is using
feedtypes IDS|DDPLUS|HDS.  The WMO feedtype is really an alias for
feedtypes IDS|DDPLUS|HDS.  The SSEC NOAAport ingest splits the data stream
into 2 types of feeds (text | binary), ie, IDS|DDPLUS and the other HDS.
This also helps to make the pqact entries more efficient by eliminating
the needs to check all the products in WMO. ie,

IDS|DDPLUS  <pattern> <action> <params>

HDS  <pattern> <action> <params>

Robb...

> 
> Robb Kambic wrote:
> 
> > On Wed, 13 Sep 2000, Steven Danz wrote:
> >
> > > Robb Kambic wrote:
> > >
> > > > On Wed, 13 Sep 2000, Unidata Support wrote:
> > > > ...
> > >
> > > > Yes, that's how the top source sites are configured for the IDD.  The 
> > > > top
> > > > sites receive the data from a satellite ingestor and also from one other
> > > > source site using the same feedtype. That way the downstream node can
> > > > request only one feedtype even though the products are coming from the
> > > > different sources.  The LDM stats files show the product source, it's
> > > > field 8.  In the example below, 634 HDS products came from UCAR 
> > > > satellite
> > > > noaaport.unidata.ucar.edu and 276 HDS products came from
> > > > sunshine.ssec.wisc.edu for hour 2000091319.
> > >
> > > Right, but if in the grand IDD scheme of things, if site A feeds B and B 
> > > feeds
> > > C, then site B adds a new experimental feed of say NOAAPort data to their
> > > ldm, the site C will start missing any piece of data that is a duplicate 
> > > between
> > > the 'original' feed and the newly added NOAAPort data.  Right?
> >
> > Steven,
> >
> > That's correct. But the products would have to be exactly the same or
> > there is 2 ** 64 chance (not positive on this number) of 2 different
> > products having the same MD5 checksum.
> >
> > >
> > > If I wanted to do something goofy like weather data and baseball scores
> > > on different feedtypes within the same ldm server, and a baseball score
> > > happened to have the same signature as a weather product, someone
> > > downstream from me is either going to
> >  miss a weather product or a
> > > baseball score.  Doesn't feel right... It seems like if I'm site C and I 
> > > ask
> > > for the WMO feedtype, then I should get all the WMO feedtype from
> > > site B (that I've requested), no matter what feedtype it came in from at
> > > site B.
> >
> > I understand your point but at this time that's the structure of the LDM.
> >
> > Robb...
> >
> > >
> > > Just a thought.
> > >
> > > Thanks again!
> > >
> > > Steven
> > >
> >
> > ===============================================================================
> > Robb Kambic                                Unidata Program Center
> > Software Engineer III                      Univ. Corp for Atmospheric 
> > Research
> > address@hidden                   WWW: http://www.unidata.ucar.edu/
> > ===============================================================================
> 

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================