[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Datastream #IJN-200350]: 20060708: EAST and WEST datasets not updating on unidata2

Jerry and Tom,

I've configured the gvarldm interface now (and for startup), a gvarldm 
user, and LDM 6.4.5 with the config files close to what they might need 
to look like.  I've restarted the operational LDM bound only to unidata2's
IP address and created a 300MB queue for gvarldm (may need to be larger 
or smaller?).  

I think all the rest is better done by someone who knows what they're 
doing with the actual data stream...

mike

> Jerry,
> 
> Please forgive my delay getting started on this.  I'll go ahead and configure 
> the virtual
> interface, create a gvarldm user and home directory, change the config of the 
> current LDM
> to bind only to the unidata2 address, and a few other incidental tasks.
> 
> mike
> 
> > Tom, and Mike,
> >
> > Unidata Datastream Support wrote:
> > > Hi Jerry,
> > >
> > > re:
> > >> The ldm relay of gvar data has been running since Friday ... so far, 
> > >> after the permission problem
> > >> got fixed Friday morning, it seems to be working well.
> > >
> > > I agree.
> > >
> > >> Have you had a chance to evaluate the
> > >> impact on unidata2.ssec.wisc.edu?  Should the queue remain the current 
> > >> size?
> > >
> > > Mike Schmidt and I chatted about the impact on unidata2's LDM, which is a 
> > > shorter set of realtime
> > > data available in the queue.  We discussed two approaches for mitigating 
> > > the impact:
> > >
> > > - increasing the size of the queue from 2 GB to 3 GB
> > >
> > > - assigning a second, virtual IP address to unidata2 and running a second 
> > > LDM that listens
> > >   only to this address
> >
> >
> >
> > I have an additional name/IP address to put on unidata2.ssec.wisc.edu
> >
> > 128.104.110.24  gvarldm.ssec.wisc.edu
> >
> > Mike, did you want me to set that up on hme0:1 or were you going to handle 
> > it?
> > I'm not sure how to set up a second ldm, so maybe Tom can help with that.
> >
> >
> >
> >
> > >
> > > The first option is quick and easy to implement, but I wanted to review 
> > > the processing
> > > you wrote for the GOES data to make sure that resends of the same data 
> > > would not
> > > harm the ADDE serving of EAST/WEST data.
> > >
> > > Last night, Mike suggested the second approach as being more attractive:
> > >
> > >   Date: Mon, 24 Jul 2006 18:34:33 -0600)
> > >   To: address@hidden
> > >   Subject: moving goes data to unidata2
> > >
> > >   Tom,
> > >
> > >   After very little thought, I think we should consider moving GOES data
> > >   to unidata2 in the fashion we were discussing for motherlode (via a
> > >   virtual IP and binding a second LDM to that address).  This would avoid
> > >   over-committing memory resources needed for the operational LDM service,
> > >   especially on unidata2 where there's currently no expandability and
> > >   currently no free memory.
> > >
> > >   We'd have to request another IP address from SSEC to make it go.
> > >
> > >   mike
> > >
> > >> Currently we have two independent systems ingesting both operational 
> > >> GVAR streams(gserve1 and
> > >> gserve2).   I only have one of the systems relaying the data via the ldm 
> > >> to unidata2.   It would be
> > >> nice if both systems were set up to relay the data, so if one system 
> > >> goes down, or is taken down for
> > >> maintenance, the other system will still be supplying the data.  Do you 
> > >> think it be possible for
> > >> both systems to relay the data without consuming too much bandwidth?
> > >
> > > I don't think that bandwidth is much of an issue for the Data Center.  
> > > That being said, LDM 6
> > > allows one to setup redundant feed requests and only one, the fastest 
> > > one, will result in files
> > > actually being transferred.  This is the approach that I would recommend 
> > > _after_ bringing up
> > > a Virtual interface and running a second LDM.
> > >
> > >> Also, I can think of a couple of problems with doing this ... while the 
> > >> data between the systems is
> > >> nearly identical .... a few bits of gvar data may be different on 
> > >> occasion between the two systems
> > >> .. not very many, but both ingestors are completely independent, and 
> > >> unless they are started at
> > >> precisely the same moment, there sometimes are a bit or two different on 
> > >> a couple images a day.
> > >> This would mean that the checksums for the same image will be different 
> > >> on the two servers.  The ldm
> > >> would treat these as different, and pull both over to unidata2 ... 
> > >> correct?
> > >
> > > Correct.  The duplicate product detection and rejection only works when 
> > > the MD5 checksums are identical.
> > >
> > >> And this would use
> > >> additional bandwidth, and allow for less products to remain in the queue?
> > >
> > > Yes and yes.
> > >
> > >> Any ideas, or should we just use one system for now?
> > >
> > > We would like to request a second IP address for unidata2.  We would then 
> > > like to bring up
> > > a separate LDM listening only to that interface and dedicated for the 
> > > GOES data transfers.
> > > After accomplishing that, we will be in a position to think about a 
> > > different approach to
> > > requesting the data redundantly.  One idea that comes to mind immediately 
> > > (unencumbered
> > > by the thought process ;-) is calculating the MD5 signature in a 
> > > different manner, like
> > > from meta data that is available.  By not assigning the signature the 
> > > value of the data
> > > product checksum, we could insure that the signatures from different 
> > > machines would be
> > > the same.
> >
> > The INDX files are almost always Identical ... so it may be a good 
> > candidate for calculating the
> > checksum.    Or creating a pseudo checksum from other meta data would be 
> > reasonable too.  Maybe the
> > INDX and or file name itself could be used, since these are unique for 
> > every Image.
> >
> > > This would allow the LDM to reject one of the redundantly requested 
> > > products.
> > > I like this approach (for now) because we do not know apriori which of 
> > > the images that
> > > are different is "better".
> > >
> > >> Thanks for any ideas.
> > >
> > > Please let us know what you think the issues above (i.e., second, virtual 
> > > interface;
> > > second LDM; different manner of calculating MD5 signature).
> > >
> > > Cheers,
> > >
> > > Tom
> >
> > Let me know what else I need to do on this end.
> >
> > Jerry
> >
> >
> > > ****************************************************************************
> > > Unidata User Support                                    UCAR Unidata 
> > > Program
> > > (303) 497-8642                                                 P.O. Box 
> > > 3000
> > > address@hidden                                   Boulder, CO 80307
> > > ----------------------------------------------------------------------------
> > > Unidata HomePage                       http://www.unidata.ucar.edu
> > > ****************************************************************************
> > >
> > >
> > > Ticket Details
> > > ===================
> > > Ticket ID: IJN-200350
> > > Department: Support Datastream
> > > Priority: Low
> > > Status: Closed
> > >
> >
> >
> > --
> > Jerrold Robaidek                       Email:  address@hidden
> > SSEC Data Center                       Phone: (608) 262-6025
> > University of Wisconsin                Fax: (608) 263-6738
> > Madison, Wisconsin
> >
> >
> 


Ticket Details
===================
Ticket ID: IJN-200350
Department: Support Datastream
Priority: Normal
Status: Closed